EAP 7.3: java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF)...

we've got a web app (war) that deploys successfully to JBoss EAP 7.3 in domain mode (using the default profile), however deployment in standalone mode (config based on the default standalone.xml) it fails with:

14:47:40,342 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 88) MSC000001: Failed to start service jboss.deployment.unit."app.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."app.war".undertow-deployment: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:254)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
... 8 more
Caused by: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
at javax.el.FactoryFinder.newInstance(FactoryFinder.java:64)
at javax.el.FactoryFinder.find(FactoryFinder.java:103)
at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:140)
at javax.el.ELUtil.getExpressionFactory(ELUtil.java:185)
at javax.el.ELManager.getExpressionFactory(ELManager.java:38)
at org.apache.jasper.runtime.JspApplicationContextImpl.<init>(JspApplicationContextImpl.java:49)
at org.apache.jasper.runtime.JspApplicationContextImpl.getInstance(JspApplicationContextImpl.java:78)
at org.apache.jasper.runtime.JspFactoryImpl.getJspApplicationContext(JspFactoryImpl.java:218)
at org.wildfly.extension.undertow.deployment.JspInitializationListener.contextInitialized(JspInitializationListener.java:51)
at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:217)
at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:186)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:252)
... 10 more
Caused by: java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF) under one or more from [Module "deployment.app.war" from Service Module Loader]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:255)
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
at javax.el.FactoryFinder.newInstance(FactoryFinder.java:48)
... 29 more

14:47:40,351 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "app.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"app.war\".undertow-deployment" => "java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
Caused by: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
Caused by: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
Caused by: java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF) under one or more from [Module \"deployment.app.war\" from Service Module Loader]"}}


War and the application configuration are identical in both cases.

JBoss configuration is as close as possible in both cases. In domain mode it's based on the default profile with basically just a few resources added. In standalone mode it's based on the default standalone.xml with the same set of resources added. Therefore from this point of view I can't find any reason for the difference in behavior.

So far we've tracked this to tomcat-embed-el-9.0.39.jar which is included in our war due to some 3rd party dependency. This jar contains META-INF/services/javax.el.ExpressionFactory with the following content:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

org.apache.el.ExpressionFactoryImpl


Where the issue is obviously that the 1st line is read and processed as an actual entry (e.g. class name) even though it's just a comment. It's therefore similar to the following older issue that affected EAP 6: https://access.redhat.com/solutions/1270603 (Red Hat login required).

It's however strange that this occurs only in the standalone mode. Could this be considered a bug?

Note: we tested this at JBoss EAP 7.3.0 and 7.3.6 with the same results.

Continue reading...