Keycloak - cannot start embedded server: WFLYEMB0022: Cannot invoke 'start' on embedded process: Parameter 'abstractPath' must not be empty

[margaretupshur]

When I clone this and then redeploy it via the container registry with the same config variables I am getting the following error. It's happening right after the change-database.sh script is run in the docker-entrypoint file. I have the same versions, same config and the button worked for me originally. I'm trying to do this because I want to add a custom theme to KC.

I'm using the heroku container:push web and heroku container:release web commands.

Jun 09 08:50:28 Deployed web (c19fbf3aed30) by user [EMAIL]
Jun 09 08:50:39 Starting process with command `-b 0.0.0.0`
Jun 09 08:50:40  Found database configuration in [CORRECT DETAILS WERE HERE]
Jun 09 08:50:43  Added [EMAIL] to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
Jun 09 08:50:43  -b 0.0.0.0
Jun 09 08:50:43  =========================================================================
Jun 09 08:50:43    Using PostgreSQL database
Jun 09 08:50:43  =========================================================================
Jun 09 08:50:44  15:50:44,394 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.2.Final
Jun 09 08:50:44  15:50:44,451 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.12.Final
Jun 09 08:50:44  15:50:44,459 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
Jun 09 08:50:44  15:50:44,576 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 12.0.0 (WildFly Core 13.0.3.Final) starting
Jun 09 08:50:44  15:50:44,611 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.as: org.jboss.msc.service.StartException in service jboss.as: Failed to start service
Jun 09 08:50:44     at org.jboss.msc@1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1731)
Jun 09 08:50:44     at org.jboss.msc@1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
Jun 09 08:50:44     at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
Jun 09 08:50:44     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
Jun 09 08:50:44     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
Jun 09 08:50:44     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
Jun 09 08:50:44     at java.base/java.lang.Thread.run(Thread.java:834)
Jun 09 08:50:44  Caused by: java.lang.IllegalArgumentException: COM00008: Parameter 'abstractPath' must not be empty
Jun 09 08:50:44     at org.wildfly.common@1.5.4.Final//org.wildfly.common.Assert.checkNotEmptyParam(Assert.java:104)
Jun 09 08:50:44     at org.jboss.as.controller@13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.convertPath(AbsolutePathService.java:70)
Jun 09 08:50:44     at org.jboss.as.controller@13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.<init>(AbsolutePathService.java:49)
Jun 09 08:50:44     at org.jboss.as.controller@13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.addService(AbsolutePathService.java:59)
Jun 09 08:50:44     at org.jboss.as.controller@13.0.3.Final//org.jboss.as.controller.services.path.AbsolutePathService.addService(AbsolutePathService.java:53)
Jun 09 08:50:44     at org.jboss.as.controller@13.0.3.Final//org.jboss.as.controller.services.path.PathManagerService.addAbsolutePathService(PathManagerService.java:259)
Jun 09 08:50:44     at org.jboss.as.controller@13.0.3.Final//org.jboss.as.controller.services.path.PathManagerService.addHardcodedAbsolutePath(PathManagerService.java:160)
Jun 09 08:50:44     at org.jboss.as.server@13.0.3.Final//org.jboss.as.server.ServerPathManagerService.addService(ServerPathManagerService.java:55)
Jun 09 08:50:44     at org.jboss.as.server@13.0.3.Final//org.jboss.as.server.ApplicationServerService.start(ApplicationServerService.java:179)
Jun 09 08:50:44     at org.jboss.msc@1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
Jun 09 08:50:44     at org.jboss.msc@1.4.12.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
Jun 09 08:50:44     ... 6 more
Jun 09 08:50:44  Cannot start embedded server: WFLYEMB0022: Cannot invoke 'start' on embedded process: WFLYSRV0141: Cannot start server: JBTHR00005: Operation failed: Failed to start service: COM00008: Parameter 'abstractPath' must not be empty```

[mieckert]

I've seen the same error appearing about a week ago on my existing deployments and I haven't really found out so far, what's causing it unfortunately :(. For some reason when I created a new app on Heroku and deployed, the error was not happening for that new app (incl. after pushes). I have a feeling that something going awry with old/cached docker images being pulled (but it's just a feeling not good knowledge):

Two things for you to maybe check:

• The repo is primarily meant to be built in Heroku (i.e., using git push https://devcenter.heroku.com/articles/build-docker-images-heroku-yml) instead of building the docker image locally and then uploading it to Heroku (which is what heroku container:pushdoes: https://devcenter.heroku.com/articles/container-registry-and-runtime).
• Your log indicates a version Keycloak 12.0.0 (WildFly Core 13.0.3.Final) -- this is a rather old version. Unless you modified the first line in the Dockerfile to specify that version, it seems the build process is for some reason not pulling the latest base image. You should currently be seing something like Keycloak 13.0.1 (WildFly Core 15.0.1.Final).

When did you deploy the image the first time to the current heroku app? (I saw the issues around Jun 2 I think). I'd be also interested to hear if you see the same effect that deploying to a new heroku app makes the problem disappear.

[PhalgunNKrishna]

Hello,

I have an almost identical error message as @margaretupshur discussed in her issue. I deployed this Keycloak instance to Heroku to work in tandem with my web application about 3 months ago but first noticed this error message in my Heroku logs on June 1,. Also, I noticed your comment regarding the line: Keycloak 12.0.0 (WildFly Core 13.0.3.Final). I am having a similar error message whereas mine says Keycloak 12.0.4 (WildFly Core 13.0.3.Final) instead.

[craigkerstiens]

This seems to happen now on a fresh heroku button deploy as well.

[brandur]

We ran into this problem yesterday on multiple Heroku instances. We weren't able to determine why it suddenly started happening — it wasn't coincidental with a deploy or upgrade — so we suspect some change the underlying Heroku platform. (Although it's quite strange that some of you ran into it months before we did.)

The error message Keycloak produces is absolutely godawful, but the problem it's running into is that Java's user.home property is suddenly coming up empty, originating from this line here:

https://github.com/wildfly/wildfly-core/blob/4fcdd7ee3d736de151cdabe6f00a5ca4fed28499/server/src/main/java/org/jboss/as/server/ServerPathManagerService.java#L55

We tried a number of approaches to fix the problem, and eventually settled on giving it an explicit value through JAVA_OPTS like this:

JAVA_OPTS="$JAVA_OPTS -Duser.home=/opt/jboss"

The entrypoint script starts a number of ancillary Java processes besides the core Keycloak server itself, so just injecting a user.home value to keycloak/bin/standalone.sh is not enough because those other services will still be missing one. Exporting an explicit value for HOME in the script didn't work either. This is why we settled on modifying JAVA_OPTS, but a Java wizard might know of a better way.

Here's the full patch we applied to fix our installations:

diff --git docker-entrypoint.sh docker-entrypoint.sh
index 6450b56..16aa30b 100755
--- docker-entrypoint.sh
+++ docker-entrypoint.sh
@@ -1,5 +1,34 @@
 #!/bin/bash
 
+#
+# IMPORTANT
+#
+# We manually override `user.home` here because we suddently have a
+# catastrophic problem where the Java runtime was no longer able to find a
+# value for `user.home`, which failed start up on Heroku. It's unclear as to
+# exactly why (it seems likely due to an underlying change on the Heroku
+# platform), but we're able to fix the problem by hard-injecting a value for
+# `user.home`.
+#
+# I tried a number of methods to fix this, but using `JAVA_OPTS` is the one
+# that ended up working. Java doesn't seem to respect `HOME` if we export it.
+# We can inject `-Duser.home=` to the program at the bottom of this file and it
+# fixes that one program, but the file also boots a cluster of ancillary
+# services from subscripts which then don't get a `user.home` value. Exporting
+# it to `JAVA_OPTS` lets the main program and all ancillary programs get it.
+#
+
+# This group of lines copied from `bin/standalone.conf` in Keycloak because
+# Keycloak won't set any of them if we override `JAVA_OPTS` at all.
+if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
+   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
+fi
+JAVA_OPTS="-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
+JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
+
+# Here's the added line.
+export JAVA_OPTS="$JAVA_OPTS -Duser.home=/opt/jboss"
+
 # Set database config from Heroku DATABASE_URL
 if [ "$DATABASE_URL" != "" ]; then
     echo "Found database configuration in DATABASE_URL=$DATABASE_URL"

[odelcroi]

Thanks I experimented the same issue since last Friday. @brandur proposal worked on my heroku keycloak instance.

[maersu]

Same issue here since last reboot. The fix from @brandur works. Thanks a lot!