You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2022/03/03 08:11:26 UTC
Re: Odd EL resolution issue - java.lang.NoClassDefFoundError: package/Class1 (wrong name: package/class1)
Robert,
Apologies for the delayed reply. Having found the root cause of the
issue I wanted to confirm whether or not the Docker desktop team viewed
it as a security vulnerability or not. I received confirmation yesterday
that they do not.
TL;DR
Running Tomcat in a container via Docker Desktop on a Windows host with
the web application served from a location on the host mounted/bound to
the container is insecure.
Longer version:
To ensure that things like security constraints are correctly applied,
Tomcat has to process URLs in a case sensitive manner. "/Test.jsp" is
not he same as "/test.jsp".
URLs are often mapped to resources on the file system. This mapping also
needs to be applied in a case sensitive manner. If the file system is
case sensitive, this is relatively straight forward. If the file system
is case insensitive, Tomcat performs some additional checks. These look
something like this:
- Request for "/Test.jsp"
- Find file "$appbase/Test.jsp"
- Get the canonical path
- Confirm the case matches the original request
On a case insensitive file system, looking up "$appbase/Test.jsp" will
match "$appbase/test.jsp" but the canonical path will return
"$appbase/test.jsp" which doesn't match so a 404 will be returned.
The issue with Docker Desktop is that paths on the Windows host
mounted/bound to the container behave like this:
- The path - as far as Windows is concerned - is "$appbase/Test.jsp"
- Code running in the container looks up "$appbase/Test.jsp"
- The file is found
- The canonical path is "$appbase/Test.jsp"
- So far, so good
- Code running in the container looks up "$appbase/test.jsp"
- The file is found
- The canonical path is "$appbase/test.jsp"
The issue is that the canonical path returned the second time matches
the path used to obtain the resource, not the canonical path on the
Windows file system. This means Tomcat cannot perform the case
sensitivity tests.
This creates the following security issues:
- security constraints can be bypasses
- JSP source code disclosure
It also creates class loading issues as Java class and package names are
also case sensitive.
The view of the Docket Desktop team is that this is, at best, a bug not
a security vulnerability because Docker Desktop is a developer tool, not
a tool for running production instances. Further, the expectation is
that the web application would be included in the container in production.
If you do continue to use this approach in development, keep in mind that:
- you may see issues like the original EL issue you reported
- security testing may report false positives
HTH,
Mark
On 08/02/2022 15:11, Robert Turner wrote:
> Okay. Yep, my most recent suspicion was correct -- it's related to the
> Docker bind to a local folder containing the webapps. As such, I believe
> it's a Docker issue of some sort and not Tomcat specific. However, you may
> want to understand it more completely in any case.
>
> Thanks for your help Mark, Rob S and Neil.
>
>
> Here are the full details of the reproduction scenario:
>
> Host system: MacOS 12.2; Docker Desktop v4.4.2 (73305), Engine 20.10.12
>
> 1. Using Maven Archetypes, do the following:
>
> mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes
> -DarchetypeArtifactId=maven-archetype-webapp -DarchetypeVersion=1.4
>
> When prompted for configuration values, use the following:
>
> groupId: com.example.rt
> artifactId: test-el-resolver
> version: 1.0-SNAPSHOT
> package: com.example.rt
>
> 2. Switch to the new folder created for the web app
>
> cd test-el-resolver
>
> 3. Modify the index.jsp file (src/main/webapp/index.jsp) to have the
> following contents:
>
> <%@page contentType="text/html" pageEncoding="UTF-8" isELIgnored="false" %>
> <%@page import="com.example.rt.Failing"%>
> <%
> final Failing failing = null;
> pageContext.setAttribute("failing", failing);
> %>
> <html>
> <body>
> <h2>Hello World!</h2>
>
> <p>field1=${failing.field1}</p>
> </body>
> </html>
>
> 4. Add the following contents to
> file src/main/java/com/example/rt/Failing.java:
>
> package com.example.rt;
>
> public class Failing {
>
> private final String field1 = "field1_value";
>
> public String getField1() {
> return field1;
> }
>
> }
>
>
> 5. Build the web app
>
> mvn package
>
>
> 6. Create a local folder to mount/bind into the Docker container
>
> mkdir webapps
>
>
> 7. Copy the WAR file to the new folder
>
> cp target/test-el-resolver.war webapps/
>
>
> 8. Start a docker container binding the local web apps folder
>
> docker run -d -p 8075:8080 --mount
> type=volume,src=`pwd`/webapps,dst=/usr/local/tomcat/webapps
> tomcat:9.0.58-jre11-openjdk
>
>
> 9. Using a browser (or equivalent), access the app:
>
> curl http://localhost:8075/test-el-resolver/
>
>
> 10. Observe the following exception in the error page:
>
> Stacktrace:
> org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:610)
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:489)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> </pre><p><b>Root Cause</b></p><pre>javax.servlet.ServletException:
> java.lang.NoClassDefFoundError: com/example/rt/Failing (wrong
> name: com/example/rt/failing)
> org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:657)
> org.apache.jsp.index_jsp._jspService(index_jsp.java:145)
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> </pre><p><b>Root Cause</b></p><pre>java.lang.NoClassDefFoundError:
> com/example/rt/Failing (wrong name:
> com/example/rt/failing)
> java.base/java.lang.ClassLoader.defineClass1(Native Method)
> java.base/java.lang.ClassLoader.defineClass(Unknown Source)
> java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
> org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2478)
> org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:870)
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1371)
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
> javax.el.ImportHandler.findClass(ImportHandler.java:477)
> javax.el.ImportHandler.resolveClass(ImportHandler.java:421)
> javax.servlet.jsp.el.ScopedAttributeELResolver.getValue(ScopedAttributeELResolver.java:85)
> org.apache.jasper.el.JasperELResolver.getValue(JasperELResolver.java:124)
> org.apache.el.parser.AstIdentifier.getValue(AstIdentifier.java:93)
> org.apache.el.parser.AstValue.getValue(AstValue.java:136)
> org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:189)
> org.apache.jasper.runtime.PageContextImpl.proprietaryEvaluate(PageContextImpl.java:692)
> org.apache.jsp.index_jsp._jspService(index_jsp.java:130)
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> </pre><p><b>Note</b> The full stack trace of the root cause is available in
> the server logs.</p><hr class="line" /><h3>Apache
> Tomcat/9.0.58</h3></body></html>
>
>
> And a "working" scenario, replace step 8 with the following:
>
> docker run -d -p 8075:8080 tomcat:9.0.58-jre11-openjdk
> docker cp target/test-e-resolver.war
> <container_id>:/usr/local/tomcat/webapps/
>
> Accessing the page with the curl command above will result in the expect
> page being displayed (and no exception).
>
>
> Robert
>
>
>
> On Tue, Feb 8, 2022 at 9:39 AM Robert Turner <rt...@e-djuster.ca> wrote:
>
>> Mark,
>>
>> Thanks for the quick follow up.
>>
>> Based on your comments I have some ideas on what it might be, and I should
>> be able to narrow that down further for you, and provide all the details
>> that you requested.
>>
>> A few notes on the environment I'm using:
>>
>> Docker (the host) is running on my Mac (12.2), not in AWS. AWS Linux is
>> only involved in the Docker image (as per the Dockerfile I shared earlier).
>> Moreover, it happens with Debian in the Docker image as well.
>> What I think is important is that I have "<tomcat>/work" and
>> "<tomcat>/webapps" mounted to file folders on the Mac, so I am suspecting
>> it's something to do with that "mounting" of the volume to the host OS, and
>> how Docker is mapping those and handling filename casing, etc. My MacOS
>> file system is APFS, Encrypted (and I thought I had case sensitivity
>> enabled, but I can no longer see that option -- maybe not an option for
>> APFS).
>>
>> I will try to confirm suspicions and provide details in a few hours
>> (hopefully -- got a few meetings today that will get in the way).
>>
>> Thanks again,
>>
>> Robert
>>
>>
>> On Tue, Feb 8, 2022 at 8:51 AM Mark Thomas <ma...@apache.org> wrote:
>>
>>> Robert,
>>>
>>> I agree this is something to do with the Docker environment.
>>>
>>> I think case insensitivity is involved somewhere as I can trigger the
>>> error if I copy my equivalent of Failure.class to failure.class and then
>>> call the JSP.
>>>
>>> I understand why it only occurs for * imports. In that instance, Tomcat
>>> has to check if the class can be loaded from each of the imported
>>> packages. Tomcat checks the file system first as that is faster than
>>> trying (and failing) to load a class. The file system checks are
>>> designed to be case sensitive - even on case insensitive file systems.
>>> Something seems to be going wrong here and I'm still not sure what.
>>>
>>> I have tried to recreate this on AWS without success. If you have time,
>>> I think we'd need the following to dig into this further:
>>>
>>> - Source for the simplest possible test WAR that demonstrates this
>>> issue. I think a single class and a single JSP page should be
>>> sufficient.
>>>
>>> - The WAR you created from the above (to rule out any build issues).
>>>
>>> - A minimal Dockerfile to create a Tomcat instance that demonstrates
>>> this issue. Should just copy the WAR to the container and start it
>>> with JPDA enabled.
>>>
>>> - Which AMI you used to create the AWS instance. I'm using the AWS free
>>> tier so I used a t2.micro instance with
>>> amzn2-ami-kernel-5.10-hvm-2.0.20220121.0-x86_64-gp2
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>>
>>> On 08/02/2022 13:24, Robert Turner wrote:
>>>> One thing to add is that my "conclusion" about OS variances I believe
>>> to be
>>>> incorrect. Our tests typically run on Linux, so I think it's still
>>>> something to do with a difference in the Docker-based environment.
>>>>
>>>> Let me know if you need any more details on anything...(but I suspect
>>> with
>>>> a debugger up on the Expression Resolvers, you will at least narrow it
>>> down
>>>> quickly...)
>>>>
>>>> On Tue, Feb 8, 2022 at 7:55 AM Robert Turner <rt...@e-djuster.ca>
>>> wrote:
>>>>
>>>>> Thanks Mark. Much appreciated.
>>>>>
>>>>> On Tue., Feb. 8, 2022, 04:06 Mark Thomas, <ma...@apache.org> wrote:
>>>>>
>>>>>> Robert,
>>>>>>
>>>>>> Thank you for putting the effort in to debugging this. Narrowing down
>>>>>> the issue to a simple test case is extremely helpful.
>>>>>>
>>>>>> The behaviour you describe looks odd to me. I'd expect consistent
>>>>>> behaviour across platforms irrespective of the case sensitivity of the
>>>>>> file system in use.
>>>>>>
>>>>>> I'm going to use your test case to investigate this further. I'll
>>> report
>>>>>> back here with my findings - hopefully later today.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>> On 08/02/2022 03:29, Robert Turner wrote:
>>>>>>> Okay, so I have finally narrowed it down the trivial failure case,
>>> and I
>>>>>>> think I have an explanation as a result:
>>>>>>>
>>>>>>> [1] works (in docker), and [2] fails (in docker) but works outside.
>>> The
>>>>>>> difference between the two is the import directive being a wildcard
>>>>>> (ugly,
>>>>>>> but historical in our app in some places we haven't yet cleaned up).
>>>>>>>
>>>>>>> I am therefore speculating based on the Expression Language
>>>>>> specification
>>>>>>> that because the class wasn't explicitly imported, it's not in the
>>> list
>>>>>> of
>>>>>>> available classes for static class resolution, and thus it fails.
>>>>>>> Combine this with MacOS and Windows not caring about filename cases,
>>> and
>>>>>>> Linux caring, then I suspect it's just matching differently in both
>>>>>> cases.
>>>>>>>
>>>>>>> Workaround/fix would be:
>>>>>>> - to ensure we explicitly import the class (instead or in addition to
>>>>>> the
>>>>>>> wildcard)
>>>>>>> OR
>>>>>>> - rename the attribute so it doesn't map directly to the class name.
>>>>>>>
>>>>>>>
>>>>>>> So I think I can bring my overly-detailed thread to an end...unless
>>> my
>>>>>>> guess at the reasoning is incorrect and someone has a better
>>>>>> explanation.
>>>>>>>
>>>>>>> Thanks Rob S and Neil for having a look and providing suggestions --
>>> in
>>>>>>> part, it was something related to what you both said, but I believe
>>> in
>>>>>>> different contexts than you expected.
>>>>>>>
>>>>>>> Robert
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>> $ cat src/main/webapp/index.jsp
>>>>>>> <%@page contentType="text/html" pageEncoding="UTF-8"
>>>>>> isELIgnored="false" %>
>>>>>>> <%@page import="com.example.rt.Failing" %>
>>>>>>> <%
>>>>>>> final Failing failing = null;
>>>>>>> pageContext.setAttribute("failing", failing);
>>>>>>> %>
>>>>>>> <html>
>>>>>>> <body>
>>>>>>> <h2>Hello World!</h2>
>>>>>>>
>>>>>>> <p>field1=${failing.field1}</p>
>>>>>>> </body>
>>>>>>>
>>>>>>> [2]
>>>>>>> $ cat src/main/webapp/index.jsp
>>>>>>> <%@page contentType="text/html" pageEncoding="UTF-8"
>>>>>> isELIgnored="false" %>
>>>>>>> <%@page import="com.example.rt.*" %>
>>>>>>> <%
>>>>>>> final Failing failing = null;
>>>>>>> pageContext.setAttribute("failing", failing);
>>>>>>> %>
>>>>>>> <html>
>>>>>>> <body>
>>>>>>> <h2>Hello World!</h2>
>>>>>>>
>>>>>>> <p>field1=${failing.field1}</p>
>>>>>>> </body>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 7, 2022 at 10:14 PM Robert Turner <rt...@e-djuster.ca>
>>>>>> wrote:
>>>>>>>
>>>>>>>> So back to a divide and conquer approach I think.
>>>>>>>>
>>>>>>>> I just created a trivial "example" [1] and it works as expected
>>> (i.e.
>>>>>> no
>>>>>>>> exception was generated) (on the same servers I was testing the
>>>>>> complex JAR
>>>>>>>> file) -- so possibly something else modifying the behaviour -- a JAR
>>>>>> on the
>>>>>>>> classpath, or something in the JSP file...
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>>>>> $ cat src/main/webapp/index.jsp
>>>>>>>> <%@page contentType="text/html" pageEncoding="UTF-8"
>>>>>> isELIgnored="false" %>
>>>>>>>> <%@page import="com.example.rt.Failing"%>
>>>>>>>> <%
>>>>>>>> final Failing failing = null;
>>>>>>>> pageContext.setAttribute("failing", failing);
>>>>>>>> %>
>>>>>>>> <html>
>>>>>>>> <body>
>>>>>>>> <h2>Hello World!</h2>
>>>>>>>>
>>>>>>>> <p>field1=${failing.field1}</p>
>>>>>>>> </body>
>>>>>>>> </html>
>>>>>>>>
>>>>>>>> $ cat src/main/java/com/example/rt/Failing.java
>>>>>>>> package com.example.rt;
>>>>>>>>
>>>>>>>> public class Failing {
>>>>>>>>
>>>>>>>> private final String field1 = "field1_value";
>>>>>>>>
>>>>>>>> public String getField1() {
>>>>>>>> return field1;
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 7, 2022 at 9:51 PM Robert Turner <rt...@e-djuster.ca>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'm just avoiding sharing product details or things I think only
>>>>>> serves
>>>>>>>>> to confuse the problem. Sorry if you felt I wasn't sharing. It
>>> wasn't
>>>>>> my
>>>>>>>>> intention to be obtuse. I didn't believe they added any value for
>>> the
>>>>>>>>> diagnostics (of course that assumes I know enough about the
>>> problem).
>>>>>>>>>
>>>>>>>>> However, since you think they might be useful, here they the name
>>>>>> mapping
>>>>>>>>> from the exception and the Java and JSP code excerpts:
>>>>>>>>>
>>>>>>>>> "package1" -> "model"
>>>>>>>>> "Class1" -> "Organization"
>>>>>>>>> "class1" -> "organization"
>>>>>>>>>
>>>>>>>>> The class is present in the package (see [1]) -- otherwise it
>>> wouldn't
>>>>>>>>> work in one environment and not the other -- I believe I have
>>>>>> confirmed
>>>>>>>>> that carefully. Full class paths (with path-names relativised to
>>>>>> compare
>>>>>>>>> between environments) listed below [2], and the code that emitted
>>> the
>>>>>>>>> listing is here [3]. I post-processed the log lines as follows [4].
>>>>>> Docker
>>>>>>>>> file for the Tomcat container provided [5]. JDK details listed in
>>> [6].
>>>>>>>>> Tomcat version is 9.0.58 in all cases.
>>>>>>>>>
>>>>>>>>> What seems to be different is:
>>>>>>>>> - the way the EL resolver is working
>>>>>>>>> OR
>>>>>>>>> - the behaviour of the class loader differs in the different
>>>>>>>>> environments.
>>>>>>>>> OR
>>>>>>>>> - something else I do not understand is relevant
>>>>>>>>>
>>>>>>>>> The working environments are MacOS and Windows, and the failing
>>>>>>>>> environment is either Debian or AWS Linux 2 running in a docker
>>>>>> container.
>>>>>>>>> If the class loaders behaviour differently, then that could explain
>>>>>> the
>>>>>>>>> issues, however, that would surprise me if they differed in any
>>>>>> material
>>>>>>>>> way on the different platforms.
>>>>>>>>>
>>>>>>>>> I hope that helps provide more detail that might be useful...
>>>>>>>>>
>>>>>>>>> Robert
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> $ unzip -l target/app.war | grep "model\/Organization\.class"
>>>>>>>>> 66246 02-07-2022 20:17
>>>>>> WEB-INF/classes/model/Organization.class
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [2]
>>>>>>>>> java.class.path=/bin/bootstrap.jar:/bin/tomcat-juli.jar
>>>>>>>>> Class loader URLs:
>>>>>>>>> Class loader
>>>>>> name=org.apache.catalina.loader.ParallelWebappClassLoader,
>>>>>>>>> URL count=146
>>>>>>>>> URL=file:/WEB-INF/classes/
>>>>>>>>> URL=file:/WEB-INF/lib/FastInfoset-1.2.18.jar
>>>>>>>>> URL=file:/WEB-INF/lib/SparseBitSet-1.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/VeracodeAnnotations-1.2.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/activation-1.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/animal-sniffer-annotations-1.20.jar
>>>>>>>>> URL=file:/WEB-INF/lib/annotations-4.1.1.4.jar
>>>>>>>>> URL=file:/WEB-INF/lib/api-common-2.0.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/auto-value-annotations-1.8.2.jar
>>>>>>>>>
>>> URL=file:/WEB-INF/lib/avatax-rest-v2-api-java_2.11-21.12.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-core-1.12.145.jar
>>>>>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-kms-1.12.145.jar
>>>>>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-s3-1.12.145.jar
>>>>>>>>> URL=file:/WEB-INF/lib/bcmail-jdk15on-1.70.jar
>>>>>>>>> URL=file:/WEB-INF/lib/bcpkix-jdk15on-1.70.jar
>>>>>>>>> URL=file:/WEB-INF/lib/bcprov-jdk15on-1.70.jar
>>>>>>>>> URL=file:/WEB-INF/lib/bcutil-jdk15on-1.70.jar
>>>>>>>>> URL=file:/WEB-INF/lib/castor-core-1.4.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/castor-xml-1.4.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/checker-qual-3.5.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-beanutils-1.9.4.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-codec-1.15.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-collections-3.2.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-collections4-4.4.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-compress-1.21.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-digester-2.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-fileupload-1.4.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-io-2.11.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-lang3-3.12.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-logging-1.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-math3-3.6.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/commons-text-1.9.jar
>>>>>>>>> URL=file:/WEB-INF/lib/conscrypt-openjdk-uber-2.5.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/curvesapi-1.06.jar
>>>>>>>>> URL=file:/WEB-INF/lib/ecj-3.21.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/encoder-1.2.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/encoder-jsp-1.2.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/error_prone_annotations-2.9.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/failureaccess-1.0.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/fontbox-2.0.25.jar
>>>>>>>>> URL=file:/WEB-INF/lib/gax-2.3.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/gax-grpc-2.3.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/gmbal-api-only-4.0.3.jar
>>>>>>>>>
>>> URL=file:/WEB-INF/lib/google-auth-library-credentials-1.1.0.jar
>>>>>>>>>
>>> URL=file:/WEB-INF/lib/google-auth-library-oauth2-http-1.1.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/google-cloud-speech-1.30.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/google-http-client-1.39.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/google-http-client-gson-1.39.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/googleauth-1.5.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-alts-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-api-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-auth-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-context-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-core-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-grpclb-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-netty-shaded-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-protobuf-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-protobuf-lite-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/grpc-stub-1.40.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/gson-2.8.9.jar
>>>>>>>>> URL=file:/WEB-INF/lib/guava-31.0.1-jre.jar
>>>>>>>>> URL=file:/WEB-INF/lib/ha-api-3.1.13.jar
>>>>>>>>> URL=file:/WEB-INF/lib/handlebars-4.3.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/httpclient-4.5.13.jar
>>>>>>>>> URL=file:/WEB-INF/lib/httpcore-4.4.15.jar
>>>>>>>>> URL=file:/WEB-INF/lib/ion-java-1.0.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/ipaddress-5.3.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/itext-2.1.7.js9.jar
>>>>>>>>> URL=file:/WEB-INF/lib/j2objc-annotations-1.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jackson-annotations-2.13.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jackson-core-2.13.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jackson-databind-2.13.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jackson-dataformat-cbor-2.13.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jai-imageio-core-1.4.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jakarta.activation-2.0.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jakarta.annotation-api-1.3.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jakarta.jws-api-2.1.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jakarta.mail-1.6.7.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jakarta.xml.bind-api-2.3.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jakarta.xml.soap-api-1.4.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jakarta.xml.ws-api-2.3.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jasperreports-6.18.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jasperreports-fonts-6.18.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/java-saml-2.8.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/java-saml-core-2.8.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/javax.activation-api-1.2.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/javax.annotation-api-1.3.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/javax.inject-1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/javax.mail-1.6.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/javax.xml.soap-api-1.4.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jaxb-api-2.3.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jaxb-core-2.3.0.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jaxb-impl-2.3.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jaxws-api-2.3.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jaxws-rt-2.3.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jbig2-imageio-3.0.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jcommon-1.0.24.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jempbox-1.8.16.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jettison-1.4.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jfreechart-1.0.19.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jmespath-java-1.12.145.jar
>>>>>>>>> URL=file:/WEB-INF/lib/joda-time-2.10.6.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jsch-0.1.55.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jsoup-1.14.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jsr181-api-1.0-MR1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/jsr305-3.0.2.jar
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>> URL=file:/WEB-INF/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
>>>>>>>>> URL=file:/WEB-INF/lib/log4j-api-2.17.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/log4j-core-2.17.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/log4j-slf4j-impl-2.17.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/log4j-web-2.17.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/management-api-3.2.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/metadata-extractor-2.16.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/mimepull-1.9.15.jar
>>>>>>>>> URL=file:/WEB-INF/lib/opencensus-api-0.28.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/opencensus-contrib-http-util-0.28.0.jar
>>>>>>>>>
>>> URL=file:/WEB-INF/lib/owasp-java-html-sanitizer-20211018.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/pdfbox-2.0.25.jar
>>>>>>>>> URL=file:/WEB-INF/lib/perfmark-api-0.23.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/poi-4.1.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/poi-ooxml-4.1.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/poi-ooxml-schemas-4.1.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/policy-2.7.10.jar
>>>>>>>>> URL=file:/WEB-INF/lib/postgresql-42.3.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1-1.30.5.jar
>>>>>>>>>
>>>>>> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1p1beta1-0.83.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/proto-google-common-protos-2.3.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/protobuf-java-3.17.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/protobuf-java-util-3.17.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/saaj-impl-1.5.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/scala-library-2.11.12.jar
>>>>>>>>> URL=file:/WEB-INF/lib/slf4j-api-1.7.33.jar
>>>>>>>>> URL=file:/WEB-INF/lib/stax-api-1.0.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/stax-ex-1.8.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/stax2-api-4.2.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/streambuffer-2.0.2.jar
>>>>>>>>> URL=file:/WEB-INF/lib/taglibs-standard-impl-1.2.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/taglibs-standard-spec-1.2.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/threetenbp-1.3.5.jar
>>>>>>>>> URL=file:/WEB-INF/lib/tika-core-2.2.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/tika-parser-image-module-2.2.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/tika-parser-xmp-commons-2.2.1.jar
>>>>>>>>> URL=file:/WEB-INF/lib/woodstox-core-6.2.6.jar
>>>>>>>>> URL=file:/WEB-INF/lib/xmlbeans-3.1.0.jar
>>>>>>>>> URL=file:/WEB-INF/lib/xmlsec-2.2.3.jar
>>>>>>>>> URL=file:/WEB-INF/lib/xmpcore-6.1.11.jar
>>>>>>>>> Class loader name=java.net.URLClassLoader, URL count=33
>>>>>>>>> URL=file:/lib/
>>>>>>>>> URL=file:/lib/annotations-api.jar
>>>>>>>>> URL=file:/lib/catalina-ant.jar
>>>>>>>>> URL=file:/lib/catalina-ha.jar
>>>>>>>>> URL=file:/lib/catalina-ssi.jar
>>>>>>>>> URL=file:/lib/catalina-storeconfig.jar
>>>>>>>>> URL=file:/lib/catalina-tribes.jar
>>>>>>>>> URL=file:/lib/catalina.jar
>>>>>>>>> URL=file:/lib/ecj-4.20.jar
>>>>>>>>> URL=file:/lib/el-api.jar
>>>>>>>>> URL=file:/lib/jasper-el.jar
>>>>>>>>> URL=file:/lib/jasper.jar
>>>>>>>>> URL=file:/lib/jaspic-api.jar
>>>>>>>>> URL=file:/lib/jsp-api.jar
>>>>>>>>> URL=file:/lib/servlet-api.jar
>>>>>>>>> URL=file:/lib/tomcat-api.jar
>>>>>>>>> URL=file:/lib/tomcat-coyote.jar
>>>>>>>>> URL=file:/lib/tomcat-dbcp.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-cs.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-de.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-es.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-fr.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-ja.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-ko.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-pt-BR.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-ru.jar
>>>>>>>>> URL=file:/lib/tomcat-i18n-zh-CN.jar
>>>>>>>>> URL=file:/lib/tomcat-jdbc.jar
>>>>>>>>> URL=file:/lib/tomcat-jni.jar
>>>>>>>>> URL=file:/lib/tomcat-util-scan.jar
>>>>>>>>> URL=file:/lib/tomcat-util.jar
>>>>>>>>> URL=file:/lib/tomcat-websocket.jar
>>>>>>>>> URL=file:/lib/websocket-api.jar
>>>>>>>>> Class loader
>>> name=jdk.internal.loader.ClassLoaders$AppClassLoader
>>>>>>>>> Class loader
>>>>>> name=jdk.internal.loader.ClassLoaders$PlatformClassLoader
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [3]
>>>>>>>>> private static void
>>>>>> logClassLoaderUrls(org.apache.logging.log4j.Level
>>>>>>>>> logLevel, ClassLoader cl) {
>>>>>>>>> if (null == cl) {
>>>>>>>>> return;
>>>>>>>>> }
>>>>>>>>> boolean loaderNameLogged = false;
>>>>>>>>> if (cl instanceof URLClassLoader) {
>>>>>>>>> final URLClassLoader urlCl = (URLClassLoader) cl;
>>>>>>>>> final URL[] urls = urlCl.getURLs();
>>>>>>>>> if (null != urls) {
>>>>>>>>> LOG.log(logLevel, " Class loader name={}, URL
>>>>>> count={}",
>>>>>>>>> cl.getClass().getName(), urls.length);
>>>>>>>>> loaderNameLogged = true;
>>>>>>>>> for (int idx = 0; idx < urls.length; idx++) {
>>>>>>>>> final URL url = urls[idx];
>>>>>>>>> LOG.log(logLevel, " URL[{}]={}", idx,
>>> url);
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> if (!loaderNameLogged) {
>>>>>>>>> LOG.log(logLevel, " Class loader name={}",
>>>>>>>>> cl.getClass().getName());
>>>>>>>>> }
>>>>>>>>> logClassLoaderUrls(logLevel, cl.getParent());
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> ..snip..
>>>>>>>>>
>>>>>>>>> LOG.info("java.class.path={}",
>>>>>>>>> System.getProperty("java.class.path"));
>>>>>>>>> LOG.debug("Class loader URLs:");
>>>>>>>>>
>>> logClassLoaderUrls(org.apache.logging.log4j.Level.DEBUG,
>>>>>>>>> ctx.getClassLoader());
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [4]
>>>>>>>>> cat log-extract.txt | cut -c137- | sed -e
>>>>>> "s+/usr/local/tomcat++g" -e
>>>>>>>>> "s+/usr/local/tomcat/webapps/exclaim++g" -e "s+\[[0-9]*\]++"
>>>>>>>>> and I manually sorted the "sections" using vim, with the "sort"
>>>>>> command.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [5]
>>>>>>>>> # Debian with OpenJDK
>>>>>>>>> #FROM tomcat:9.0.58-jre11-openjdk AS tomcat_package
>>>>>>>>> # AWS Linux 2 with Corretto
>>>>>>>>> FROM tomcat:9.0.58-jdk11-corretto AS tomcat_package
>>>>>>>>>
>>>>>>>>> # Set the timezone to Canada/Eastern
>>>>>>>>> # For Debian
>>>>>>>>> #RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime &&
>>> echo
>>>>>>>>> "Canada/Eastern" > /etc/timezone && dpkg-reconfigure -f
>>> noninteractive
>>>>>>>>> tzdata
>>>>>>>>> # For AWS Linux
>>>>>>>>> RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime &&
>>> sed -i
>>>>>>>>> 's/\"UTC\"/\"Canada\/Eastern\"/' /etc/sysconfig/clock # and reboot,
>>>>>> but
>>>>>>>>> will skip as container will be created and ran separately
>>>>>>>>>
>>>>>>>>> # Install a postgresql client for testing and the required font
>>>>>> packages
>>>>>>>>> needed by JasperReports
>>>>>>>>> # For Debian:
>>>>>>>>> #RUN echo "deb http://deb.debian.org/debian bullseye contrib
>>>>>> non-free"
>>>>>>>>>>> /etc/apt/sources.list && apt-get update && apt-get install -y
>>>>>>>>> postgresql-client ttf-mscorefonts-installer
>>>>>>>>> # For AWS Linux:
>>>>>>>>> RUN yum -q list installed epel-release.noarch &>/dev/null && echo
>>>>>>>>> 'epel-release already installed' || yum install -y
>>>>>>>>>
>>>>>>
>>> https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
>>>>>>>>> RUN yum install -y cabextract xorg-x11-font-utils fontconfig
>>>>>>>>> RUN rpm -q msttcore-fonts-installer-2.6-1 &>/dev/null && echo
>>>>>>>>> 'msttcore-fonts-installer-2.6-1 already installed' || rpm -i
>>>>>>>>>
>>>>>>
>>> https://iweb.dl.sourceforge.net/project/mscorefonts2/rpms/msttcore-fonts-installer-2.6-1.noarch.rpm
>>>>>>>>>
>>>>>>>>> # Add a health-check script to wait for exclaim to be running --
>>>>>> allow at
>>>>>>>>> least 10 minutes for it to start
>>>>>>>>> COPY health_check.sh /usr/local/bin/
>>>>>>>>> HEALTHCHECK --interval=10s --timeout=5s --start-period=10m
>>>>>> --retries=10
>>>>>>>>> CMD /usr/local/bin/health_check.sh
>>>>>>>>>
>>>>>>>>> # Set up the resource folders needed by application and make it a
>>>>>>>>> # volume so it persists across container restarts
>>>>>>>>> COPY resources/ /resources/
>>>>>>>>> VOLUME /resources
>>>>>>>>>
>>>>>>>>> # Map some of the Tomcat folders to volumes so they persist between
>>>>>>>>> restarts of the container
>>>>>>>>> VOLUME /usr/local/tomcat/logs
>>>>>>>>> VOLUME /usr/local/tomcat/webapps
>>>>>>>>> VOLUME /usr/local/tomcat/work
>>>>>>>>> VOLUME /usr/local/tomcat/conf
>>>>>>>>>
>>>>>>>>> # Enable connecting from anywhere for JPDA (req'd for Java 9+)
>>>>>>>>> ENV JPDA_ADDRESS=*:8000
>>>>>>>>>
>>>>>>>>> CMD ["catalina.sh","jpda","run"]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [6]
>>>>>>>>> JDK on MacOS is:
>>>>>>>>> openjdk version "11.0.13" 2021-10-19 LTS
>>>>>>>>> OpenJDK Runtime Environment Corretto-11.0.13.8.1 (build
>>> 11.0.13+8-LTS)
>>>>>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.13.8.1 (build 11.0.13+8-LTS,
>>>>>> mixed
>>>>>>>>> mode)
>>>>>>>>>
>>>>>>>>> JDK on Windows is:
>>>>>>>>> openjdk 11.0.14 2022-01-18 LTS
>>>>>>>>> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build
>>> 11.0.14+9-LTS)
>>>>>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS,
>>>>>> mixed
>>>>>>>>> mode)
>>>>>>>>>
>>>>>>>>> JDK on Docker with AWS Linux 2 is:
>>>>>>>>> openjdk 11.0.14 2022-01-18 LTS
>>>>>>>>> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build
>>> 11.0.14+9-LTS)
>>>>>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS,
>>>>>> mixed
>>>>>>>>> mode)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 7, 2022 at 9:24 PM Rob Sargent <rs...@xmission.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2/7/22 19:13, Robert Turner wrote:
>>>>>>>>>>> So, I've gone back and double-checked as much as I can (Tomcat
>>>>>>>>>> version, JDK
>>>>>>>>>>> version), and the classpath, and I have identical classpaths in
>>> both
>>>>>>>>>>> environments (except the sort order of the URLs per "level" of
>>>>>>>>>>> ClassLoader), and I've re-verified the behaviour:
>>>>>>>>>>> - fails in the docker environment
>>>>>>>>>>> - works locally
>>>>>>>>>>> (Same WAR file on both).
>>>>>>>>>>>
>>>>>>>>>>> I guess I'm in to one of the following approaches next:
>>>>>>>>>>> - build a debug version of Tomcat, and step through the code
>>> that
>>>>>>>>>> breaks
>>>>>>>>>>> and see if I can figure out why (not that I have time to do this
>>> of
>>>>>>>>>>> course...but might be necessary)
>>>>>>>>>>> - construct a trivial application reproduction, along with
>>> docker
>>>>>>>>>> layout,
>>>>>>>>>>> and see if anyone else can reproduce... (assuming anyone else has
>>>>>> time
>>>>>>>>>> to
>>>>>>>>>>> do that of course...)
>>>>>>>>>>>
>>>>>>>>>>> Anyone got any suggestions of what to look into next?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 7, 2022 at 5:05 PM Rob Sargent <
>>> rsargent@xmission.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> As you appear averse to sharing names of things, seems you will
>>> need
>>>>>> to
>>>>>>>>>> do a lot of extra clean-up before you can share a docker image.
>>> Make
>>>>>>>>>> sure you're clear on what NoClassDefFoundError indicates (present
>>> at
>>>>>>>>>> compile time, absent at runtime) and how that translates to
>>>>>> docker-ness
>>>>>>>>>> (of which I know naught).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Odd EL resolution issue - java.lang.NoClassDefFoundError: package/Class1 (wrong name: package/class1)
Posted by Robert Turner <rt...@e-djuster.ca>.
Mark,
Thanks for continuing to look into it, and producing a detailed record of
the issues and the cause (along with intermediate details). Hopefully it
will come in useful for others in the future.
Robert
On Thu, Mar 3, 2022 at 3:11 AM Mark Thomas <ma...@apache.org> wrote:
> Robert,
>
> Apologies for the delayed reply. Having found the root cause of the
> issue I wanted to confirm whether or not the Docker desktop team viewed
> it as a security vulnerability or not. I received confirmation yesterday
> that they do not.
>
> TL;DR
> Running Tomcat in a container via Docker Desktop on a Windows host with
> the web application served from a location on the host mounted/bound to
> the container is insecure.
>
> Longer version:
> To ensure that things like security constraints are correctly applied,
> Tomcat has to process URLs in a case sensitive manner. "/Test.jsp" is
> not he same as "/test.jsp".
>
> URLs are often mapped to resources on the file system. This mapping also
> needs to be applied in a case sensitive manner. If the file system is
> case sensitive, this is relatively straight forward. If the file system
> is case insensitive, Tomcat performs some additional checks. These look
> something like this:
> - Request for "/Test.jsp"
> - Find file "$appbase/Test.jsp"
> - Get the canonical path
> - Confirm the case matches the original request
>
> On a case insensitive file system, looking up "$appbase/Test.jsp" will
> match "$appbase/test.jsp" but the canonical path will return
> "$appbase/test.jsp" which doesn't match so a 404 will be returned.
>
> The issue with Docker Desktop is that paths on the Windows host
> mounted/bound to the container behave like this:
> - The path - as far as Windows is concerned - is "$appbase/Test.jsp"
> - Code running in the container looks up "$appbase/Test.jsp"
> - The file is found
> - The canonical path is "$appbase/Test.jsp"
> - So far, so good
> - Code running in the container looks up "$appbase/test.jsp"
> - The file is found
> - The canonical path is "$appbase/test.jsp"
>
> The issue is that the canonical path returned the second time matches
> the path used to obtain the resource, not the canonical path on the
> Windows file system. This means Tomcat cannot perform the case
> sensitivity tests.
> This creates the following security issues:
> - security constraints can be bypasses
> - JSP source code disclosure
> It also creates class loading issues as Java class and package names are
> also case sensitive.
>
> The view of the Docket Desktop team is that this is, at best, a bug not
> a security vulnerability because Docker Desktop is a developer tool, not
> a tool for running production instances. Further, the expectation is
> that the web application would be included in the container in production.
>
> If you do continue to use this approach in development, keep in mind that:
> - you may see issues like the original EL issue you reported
> - security testing may report false positives
>
> HTH,
>
> Mark
>
>
>
>
> On 08/02/2022 15:11, Robert Turner wrote:
> > Okay. Yep, my most recent suspicion was correct -- it's related to the
> > Docker bind to a local folder containing the webapps. As such, I believe
> > it's a Docker issue of some sort and not Tomcat specific. However, you
> may
> > want to understand it more completely in any case.
> >
> > Thanks for your help Mark, Rob S and Neil.
> >
> >
> > Here are the full details of the reproduction scenario:
> >
> > Host system: MacOS 12.2; Docker Desktop v4.4.2 (73305), Engine 20.10.12
> >
> > 1. Using Maven Archetypes, do the following:
> >
> > mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes
> > -DarchetypeArtifactId=maven-archetype-webapp -DarchetypeVersion=1.4
> >
> > When prompted for configuration values, use the following:
> >
> > groupId: com.example.rt
> > artifactId: test-el-resolver
> > version: 1.0-SNAPSHOT
> > package: com.example.rt
> >
> > 2. Switch to the new folder created for the web app
> >
> > cd test-el-resolver
> >
> > 3. Modify the index.jsp file (src/main/webapp/index.jsp) to have the
> > following contents:
> >
> > <%@page contentType="text/html" pageEncoding="UTF-8" isELIgnored="false"
> %>
> > <%@page import="com.example.rt.Failing"%>
> > <%
> > final Failing failing = null;
> > pageContext.setAttribute("failing", failing);
> > %>
> > <html>
> > <body>
> > <h2>Hello World!</h2>
> >
> > <p>field1=${failing.field1}</p>
> > </body>
> > </html>
> >
> > 4. Add the following contents to
> > file src/main/java/com/example/rt/Failing.java:
> >
> > package com.example.rt;
> >
> > public class Failing {
> >
> > private final String field1 = "field1_value";
> >
> > public String getField1() {
> > return field1;
> > }
> >
> > }
> >
> >
> > 5. Build the web app
> >
> > mvn package
> >
> >
> > 6. Create a local folder to mount/bind into the Docker container
> >
> > mkdir webapps
> >
> >
> > 7. Copy the WAR file to the new folder
> >
> > cp target/test-el-resolver.war webapps/
> >
> >
> > 8. Start a docker container binding the local web apps folder
> >
> > docker run -d -p 8075:8080 --mount
> > type=volume,src=`pwd`/webapps,dst=/usr/local/tomcat/webapps
> > tomcat:9.0.58-jre11-openjdk
> >
> >
> > 9. Using a browser (or equivalent), access the app:
> >
> > curl http://localhost:8075/test-el-resolver/
> >
> >
> > 10. Observe the following exception in the error page:
> >
> > Stacktrace:
> >
> org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:610)
> >
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:489)
> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> > </pre><p><b>Root Cause</b></p><pre>javax.servlet.ServletException:
> > java.lang.NoClassDefFoundError: com/example/rt/Failing (wrong
> > name: com/example/rt/failing)
> >
> org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:657)
> > org.apache.jsp.index_jsp._jspService(index_jsp.java:145)
> > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> >
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> > </pre><p><b>Root Cause</b></p><pre>java.lang.NoClassDefFoundError:
> > com/example/rt/Failing (wrong name:
> > com/example/rt/failing)
> > java.base/java.lang.ClassLoader.defineClass1(Native Method)
> > java.base/java.lang.ClassLoader.defineClass(Unknown Source)
> > java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
> >
> org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2478)
> >
> org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:870)
> >
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1371)
> >
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
> > javax.el.ImportHandler.findClass(ImportHandler.java:477)
> > javax.el.ImportHandler.resolveClass(ImportHandler.java:421)
> >
> javax.servlet.jsp.el.ScopedAttributeELResolver.getValue(ScopedAttributeELResolver.java:85)
> > org.apache.jasper.el.JasperELResolver.getValue(JasperELResolver.java:124)
> > org.apache.el.parser.AstIdentifier.getValue(AstIdentifier.java:93)
> > org.apache.el.parser.AstValue.getValue(AstValue.java:136)
> > org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:189)
> >
> org.apache.jasper.runtime.PageContextImpl.proprietaryEvaluate(PageContextImpl.java:692)
> > org.apache.jsp.index_jsp._jspService(index_jsp.java:130)
> > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> >
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> > </pre><p><b>Note</b> The full stack trace of the root cause is available
> in
> > the server logs.</p><hr class="line" /><h3>Apache
> > Tomcat/9.0.58</h3></body></html>
> >
> >
> > And a "working" scenario, replace step 8 with the following:
> >
> > docker run -d -p 8075:8080 tomcat:9.0.58-jre11-openjdk
> > docker cp target/test-e-resolver.war
> > <container_id>:/usr/local/tomcat/webapps/
> >
> > Accessing the page with the curl command above will result in the expect
> > page being displayed (and no exception).
> >
> >
> > Robert
> >
> >
> >
> > On Tue, Feb 8, 2022 at 9:39 AM Robert Turner <rt...@e-djuster.ca>
> wrote:
> >
> >> Mark,
> >>
> >> Thanks for the quick follow up.
> >>
> >> Based on your comments I have some ideas on what it might be, and I
> should
> >> be able to narrow that down further for you, and provide all the details
> >> that you requested.
> >>
> >> A few notes on the environment I'm using:
> >>
> >> Docker (the host) is running on my Mac (12.2), not in AWS. AWS Linux is
> >> only involved in the Docker image (as per the Dockerfile I shared
> earlier).
> >> Moreover, it happens with Debian in the Docker image as well.
> >> What I think is important is that I have "<tomcat>/work" and
> >> "<tomcat>/webapps" mounted to file folders on the Mac, so I am
> suspecting
> >> it's something to do with that "mounting" of the volume to the host OS,
> and
> >> how Docker is mapping those and handling filename casing, etc. My MacOS
> >> file system is APFS, Encrypted (and I thought I had case sensitivity
> >> enabled, but I can no longer see that option -- maybe not an option for
> >> APFS).
> >>
> >> I will try to confirm suspicions and provide details in a few hours
> >> (hopefully -- got a few meetings today that will get in the way).
> >>
> >> Thanks again,
> >>
> >> Robert
> >>
> >>
> >> On Tue, Feb 8, 2022 at 8:51 AM Mark Thomas <ma...@apache.org> wrote:
> >>
> >>> Robert,
> >>>
> >>> I agree this is something to do with the Docker environment.
> >>>
> >>> I think case insensitivity is involved somewhere as I can trigger the
> >>> error if I copy my equivalent of Failure.class to failure.class and
> then
> >>> call the JSP.
> >>>
> >>> I understand why it only occurs for * imports. In that instance, Tomcat
> >>> has to check if the class can be loaded from each of the imported
> >>> packages. Tomcat checks the file system first as that is faster than
> >>> trying (and failing) to load a class. The file system checks are
> >>> designed to be case sensitive - even on case insensitive file systems.
> >>> Something seems to be going wrong here and I'm still not sure what.
> >>>
> >>> I have tried to recreate this on AWS without success. If you have time,
> >>> I think we'd need the following to dig into this further:
> >>>
> >>> - Source for the simplest possible test WAR that demonstrates this
> >>> issue. I think a single class and a single JSP page should be
> >>> sufficient.
> >>>
> >>> - The WAR you created from the above (to rule out any build issues).
> >>>
> >>> - A minimal Dockerfile to create a Tomcat instance that demonstrates
> >>> this issue. Should just copy the WAR to the container and start it
> >>> with JPDA enabled.
> >>>
> >>> - Which AMI you used to create the AWS instance. I'm using the AWS free
> >>> tier so I used a t2.micro instance with
> >>> amzn2-ami-kernel-5.10-hvm-2.0.20220121.0-x86_64-gp2
> >>>
> >>> Thanks,
> >>>
> >>> Mark
> >>>
> >>>
> >>> On 08/02/2022 13:24, Robert Turner wrote:
> >>>> One thing to add is that my "conclusion" about OS variances I believe
> >>> to be
> >>>> incorrect. Our tests typically run on Linux, so I think it's still
> >>>> something to do with a difference in the Docker-based environment.
> >>>>
> >>>> Let me know if you need any more details on anything...(but I suspect
> >>> with
> >>>> a debugger up on the Expression Resolvers, you will at least narrow it
> >>> down
> >>>> quickly...)
> >>>>
> >>>> On Tue, Feb 8, 2022 at 7:55 AM Robert Turner <rt...@e-djuster.ca>
> >>> wrote:
> >>>>
> >>>>> Thanks Mark. Much appreciated.
> >>>>>
> >>>>> On Tue., Feb. 8, 2022, 04:06 Mark Thomas, <ma...@apache.org> wrote:
> >>>>>
> >>>>>> Robert,
> >>>>>>
> >>>>>> Thank you for putting the effort in to debugging this. Narrowing
> down
> >>>>>> the issue to a simple test case is extremely helpful.
> >>>>>>
> >>>>>> The behaviour you describe looks odd to me. I'd expect consistent
> >>>>>> behaviour across platforms irrespective of the case sensitivity of
> the
> >>>>>> file system in use.
> >>>>>>
> >>>>>> I'm going to use your test case to investigate this further. I'll
> >>> report
> >>>>>> back here with my findings - hopefully later today.
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> >>>>>> On 08/02/2022 03:29, Robert Turner wrote:
> >>>>>>> Okay, so I have finally narrowed it down the trivial failure case,
> >>> and I
> >>>>>>> think I have an explanation as a result:
> >>>>>>>
> >>>>>>> [1] works (in docker), and [2] fails (in docker) but works outside.
> >>> The
> >>>>>>> difference between the two is the import directive being a wildcard
> >>>>>> (ugly,
> >>>>>>> but historical in our app in some places we haven't yet cleaned
> up).
> >>>>>>>
> >>>>>>> I am therefore speculating based on the Expression Language
> >>>>>> specification
> >>>>>>> that because the class wasn't explicitly imported, it's not in the
> >>> list
> >>>>>> of
> >>>>>>> available classes for static class resolution, and thus it fails.
> >>>>>>> Combine this with MacOS and Windows not caring about filename
> cases,
> >>> and
> >>>>>>> Linux caring, then I suspect it's just matching differently in both
> >>>>>> cases.
> >>>>>>>
> >>>>>>> Workaround/fix would be:
> >>>>>>> - to ensure we explicitly import the class (instead or in addition
> to
> >>>>>> the
> >>>>>>> wildcard)
> >>>>>>> OR
> >>>>>>> - rename the attribute so it doesn't map directly to the class
> name.
> >>>>>>>
> >>>>>>>
> >>>>>>> So I think I can bring my overly-detailed thread to an end...unless
> >>> my
> >>>>>>> guess at the reasoning is incorrect and someone has a better
> >>>>>> explanation.
> >>>>>>>
> >>>>>>> Thanks Rob S and Neil for having a look and providing suggestions
> --
> >>> in
> >>>>>>> part, it was something related to what you both said, but I believe
> >>> in
> >>>>>>> different contexts than you expected.
> >>>>>>>
> >>>>>>> Robert
> >>>>>>>
> >>>>>>>
> >>>>>>> [1]
> >>>>>>> $ cat src/main/webapp/index.jsp
> >>>>>>> <%@page contentType="text/html" pageEncoding="UTF-8"
> >>>>>> isELIgnored="false" %>
> >>>>>>> <%@page import="com.example.rt.Failing" %>
> >>>>>>> <%
> >>>>>>> final Failing failing = null;
> >>>>>>> pageContext.setAttribute("failing", failing);
> >>>>>>> %>
> >>>>>>> <html>
> >>>>>>> <body>
> >>>>>>> <h2>Hello World!</h2>
> >>>>>>>
> >>>>>>> <p>field1=${failing.field1}</p>
> >>>>>>> </body>
> >>>>>>>
> >>>>>>> [2]
> >>>>>>> $ cat src/main/webapp/index.jsp
> >>>>>>> <%@page contentType="text/html" pageEncoding="UTF-8"
> >>>>>> isELIgnored="false" %>
> >>>>>>> <%@page import="com.example.rt.*" %>
> >>>>>>> <%
> >>>>>>> final Failing failing = null;
> >>>>>>> pageContext.setAttribute("failing", failing);
> >>>>>>> %>
> >>>>>>> <html>
> >>>>>>> <body>
> >>>>>>> <h2>Hello World!</h2>
> >>>>>>>
> >>>>>>> <p>field1=${failing.field1}</p>
> >>>>>>> </body>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Feb 7, 2022 at 10:14 PM Robert Turner <
> rturner@e-djuster.ca>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> So back to a divide and conquer approach I think.
> >>>>>>>>
> >>>>>>>> I just created a trivial "example" [1] and it works as expected
> >>> (i.e.
> >>>>>> no
> >>>>>>>> exception was generated) (on the same servers I was testing the
> >>>>>> complex JAR
> >>>>>>>> file) -- so possibly something else modifying the behaviour -- a
> JAR
> >>>>>> on the
> >>>>>>>> classpath, or something in the JSP file...
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>>>> $ cat src/main/webapp/index.jsp
> >>>>>>>> <%@page contentType="text/html" pageEncoding="UTF-8"
> >>>>>> isELIgnored="false" %>
> >>>>>>>> <%@page import="com.example.rt.Failing"%>
> >>>>>>>> <%
> >>>>>>>> final Failing failing = null;
> >>>>>>>> pageContext.setAttribute("failing", failing);
> >>>>>>>> %>
> >>>>>>>> <html>
> >>>>>>>> <body>
> >>>>>>>> <h2>Hello World!</h2>
> >>>>>>>>
> >>>>>>>> <p>field1=${failing.field1}</p>
> >>>>>>>> </body>
> >>>>>>>> </html>
> >>>>>>>>
> >>>>>>>> $ cat src/main/java/com/example/rt/Failing.java
> >>>>>>>> package com.example.rt;
> >>>>>>>>
> >>>>>>>> public class Failing {
> >>>>>>>>
> >>>>>>>> private final String field1 = "field1_value";
> >>>>>>>>
> >>>>>>>> public String getField1() {
> >>>>>>>> return field1;
> >>>>>>>> }
> >>>>>>>>
> >>>>>>>> }
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Feb 7, 2022 at 9:51 PM Robert Turner <
> rturner@e-djuster.ca>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I'm just avoiding sharing product details or things I think only
> >>>>>> serves
> >>>>>>>>> to confuse the problem. Sorry if you felt I wasn't sharing. It
> >>> wasn't
> >>>>>> my
> >>>>>>>>> intention to be obtuse. I didn't believe they added any value for
> >>> the
> >>>>>>>>> diagnostics (of course that assumes I know enough about the
> >>> problem).
> >>>>>>>>>
> >>>>>>>>> However, since you think they might be useful, here they the name
> >>>>>> mapping
> >>>>>>>>> from the exception and the Java and JSP code excerpts:
> >>>>>>>>>
> >>>>>>>>> "package1" -> "model"
> >>>>>>>>> "Class1" -> "Organization"
> >>>>>>>>> "class1" -> "organization"
> >>>>>>>>>
> >>>>>>>>> The class is present in the package (see [1]) -- otherwise it
> >>> wouldn't
> >>>>>>>>> work in one environment and not the other -- I believe I have
> >>>>>> confirmed
> >>>>>>>>> that carefully. Full class paths (with path-names relativised to
> >>>>>> compare
> >>>>>>>>> between environments) listed below [2], and the code that emitted
> >>> the
> >>>>>>>>> listing is here [3]. I post-processed the log lines as follows
> [4].
> >>>>>> Docker
> >>>>>>>>> file for the Tomcat container provided [5]. JDK details listed in
> >>> [6].
> >>>>>>>>> Tomcat version is 9.0.58 in all cases.
> >>>>>>>>>
> >>>>>>>>> What seems to be different is:
> >>>>>>>>> - the way the EL resolver is working
> >>>>>>>>> OR
> >>>>>>>>> - the behaviour of the class loader differs in the different
> >>>>>>>>> environments.
> >>>>>>>>> OR
> >>>>>>>>> - something else I do not understand is relevant
> >>>>>>>>>
> >>>>>>>>> The working environments are MacOS and Windows, and the failing
> >>>>>>>>> environment is either Debian or AWS Linux 2 running in a docker
> >>>>>> container.
> >>>>>>>>> If the class loaders behaviour differently, then that could
> explain
> >>>>>> the
> >>>>>>>>> issues, however, that would surprise me if they differed in any
> >>>>>> material
> >>>>>>>>> way on the different platforms.
> >>>>>>>>>
> >>>>>>>>> I hope that helps provide more detail that might be useful...
> >>>>>>>>>
> >>>>>>>>> Robert
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>> $ unzip -l target/app.war | grep "model\/Organization\.class"
> >>>>>>>>> 66246 02-07-2022 20:17
> >>>>>> WEB-INF/classes/model/Organization.class
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [2]
> >>>>>>>>> java.class.path=/bin/bootstrap.jar:/bin/tomcat-juli.jar
> >>>>>>>>> Class loader URLs:
> >>>>>>>>> Class loader
> >>>>>> name=org.apache.catalina.loader.ParallelWebappClassLoader,
> >>>>>>>>> URL count=146
> >>>>>>>>> URL=file:/WEB-INF/classes/
> >>>>>>>>> URL=file:/WEB-INF/lib/FastInfoset-1.2.18.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/SparseBitSet-1.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/VeracodeAnnotations-1.2.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/activation-1.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/animal-sniffer-annotations-1.20.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/annotations-4.1.1.4.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/api-common-2.0.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/auto-value-annotations-1.8.2.jar
> >>>>>>>>>
> >>> URL=file:/WEB-INF/lib/avatax-rest-v2-api-java_2.11-21.12.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-core-1.12.145.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-kms-1.12.145.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-s3-1.12.145.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/bcmail-jdk15on-1.70.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/bcpkix-jdk15on-1.70.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/bcprov-jdk15on-1.70.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/bcutil-jdk15on-1.70.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/castor-core-1.4.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/castor-xml-1.4.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/checker-qual-3.5.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-beanutils-1.9.4.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-codec-1.15.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-collections-3.2.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-collections4-4.4.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-compress-1.21.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-digester-2.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-fileupload-1.4.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-io-2.11.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-lang3-3.12.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-logging-1.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-math3-3.6.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/commons-text-1.9.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/conscrypt-openjdk-uber-2.5.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/curvesapi-1.06.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/ecj-3.21.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/encoder-1.2.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/encoder-jsp-1.2.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/error_prone_annotations-2.9.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/failureaccess-1.0.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/fontbox-2.0.25.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/gax-2.3.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/gax-grpc-2.3.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/gmbal-api-only-4.0.3.jar
> >>>>>>>>>
> >>> URL=file:/WEB-INF/lib/google-auth-library-credentials-1.1.0.jar
> >>>>>>>>>
> >>> URL=file:/WEB-INF/lib/google-auth-library-oauth2-http-1.1.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/google-cloud-speech-1.30.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/google-http-client-1.39.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/google-http-client-gson-1.39.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/googleauth-1.5.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-alts-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-api-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-auth-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-context-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-core-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-grpclb-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-netty-shaded-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-protobuf-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-protobuf-lite-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/grpc-stub-1.40.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/gson-2.8.9.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/guava-31.0.1-jre.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/ha-api-3.1.13.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/handlebars-4.3.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/httpclient-4.5.13.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/httpcore-4.4.15.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/ion-java-1.0.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/ipaddress-5.3.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/itext-2.1.7.js9.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/j2objc-annotations-1.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jackson-annotations-2.13.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jackson-core-2.13.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jackson-databind-2.13.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jackson-dataformat-cbor-2.13.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jai-imageio-core-1.4.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jakarta.activation-2.0.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jakarta.annotation-api-1.3.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jakarta.jws-api-2.1.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jakarta.mail-1.6.7.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jakarta.xml.bind-api-2.3.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jakarta.xml.soap-api-1.4.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jakarta.xml.ws-api-2.3.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jasperreports-6.18.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jasperreports-fonts-6.18.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/java-saml-2.8.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/java-saml-core-2.8.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/javax.activation-api-1.2.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/javax.annotation-api-1.3.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/javax.inject-1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/javax.mail-1.6.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/javax.xml.soap-api-1.4.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jaxb-api-2.3.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jaxb-core-2.3.0.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jaxb-impl-2.3.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jaxws-api-2.3.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jaxws-rt-2.3.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jbig2-imageio-3.0.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jcommon-1.0.24.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jempbox-1.8.16.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jettison-1.4.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jfreechart-1.0.19.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jmespath-java-1.12.145.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/joda-time-2.10.6.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jsch-0.1.55.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jsoup-1.14.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jsr181-api-1.0-MR1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/jsr305-3.0.2.jar
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>
> URL=file:/WEB-INF/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/log4j-api-2.17.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/log4j-core-2.17.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/log4j-slf4j-impl-2.17.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/log4j-web-2.17.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/management-api-3.2.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/metadata-extractor-2.16.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/mimepull-1.9.15.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/opencensus-api-0.28.0.jar
> >>>>>>>>>
> URL=file:/WEB-INF/lib/opencensus-contrib-http-util-0.28.0.jar
> >>>>>>>>>
> >>> URL=file:/WEB-INF/lib/owasp-java-html-sanitizer-20211018.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/pdfbox-2.0.25.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/perfmark-api-0.23.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/poi-4.1.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/poi-ooxml-4.1.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/poi-ooxml-schemas-4.1.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/policy-2.7.10.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/postgresql-42.3.1.jar
> >>>>>>>>>
> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1-1.30.5.jar
> >>>>>>>>>
> >>>>>> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1p1beta1-0.83.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/proto-google-common-protos-2.3.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/protobuf-java-3.17.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/protobuf-java-util-3.17.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/saaj-impl-1.5.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/scala-library-2.11.12.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/slf4j-api-1.7.33.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/stax-api-1.0.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/stax-ex-1.8.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/stax2-api-4.2.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/streambuffer-2.0.2.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/taglibs-standard-impl-1.2.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/taglibs-standard-spec-1.2.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/threetenbp-1.3.5.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/tika-core-2.2.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/tika-parser-image-module-2.2.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/tika-parser-xmp-commons-2.2.1.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/woodstox-core-6.2.6.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/xmlbeans-3.1.0.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/xmlsec-2.2.3.jar
> >>>>>>>>> URL=file:/WEB-INF/lib/xmpcore-6.1.11.jar
> >>>>>>>>> Class loader name=java.net.URLClassLoader, URL count=33
> >>>>>>>>> URL=file:/lib/
> >>>>>>>>> URL=file:/lib/annotations-api.jar
> >>>>>>>>> URL=file:/lib/catalina-ant.jar
> >>>>>>>>> URL=file:/lib/catalina-ha.jar
> >>>>>>>>> URL=file:/lib/catalina-ssi.jar
> >>>>>>>>> URL=file:/lib/catalina-storeconfig.jar
> >>>>>>>>> URL=file:/lib/catalina-tribes.jar
> >>>>>>>>> URL=file:/lib/catalina.jar
> >>>>>>>>> URL=file:/lib/ecj-4.20.jar
> >>>>>>>>> URL=file:/lib/el-api.jar
> >>>>>>>>> URL=file:/lib/jasper-el.jar
> >>>>>>>>> URL=file:/lib/jasper.jar
> >>>>>>>>> URL=file:/lib/jaspic-api.jar
> >>>>>>>>> URL=file:/lib/jsp-api.jar
> >>>>>>>>> URL=file:/lib/servlet-api.jar
> >>>>>>>>> URL=file:/lib/tomcat-api.jar
> >>>>>>>>> URL=file:/lib/tomcat-coyote.jar
> >>>>>>>>> URL=file:/lib/tomcat-dbcp.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-cs.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-de.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-es.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-fr.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-ja.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-ko.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-pt-BR.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-ru.jar
> >>>>>>>>> URL=file:/lib/tomcat-i18n-zh-CN.jar
> >>>>>>>>> URL=file:/lib/tomcat-jdbc.jar
> >>>>>>>>> URL=file:/lib/tomcat-jni.jar
> >>>>>>>>> URL=file:/lib/tomcat-util-scan.jar
> >>>>>>>>> URL=file:/lib/tomcat-util.jar
> >>>>>>>>> URL=file:/lib/tomcat-websocket.jar
> >>>>>>>>> URL=file:/lib/websocket-api.jar
> >>>>>>>>> Class loader
> >>> name=jdk.internal.loader.ClassLoaders$AppClassLoader
> >>>>>>>>> Class loader
> >>>>>> name=jdk.internal.loader.ClassLoaders$PlatformClassLoader
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [3]
> >>>>>>>>> private static void
> >>>>>> logClassLoaderUrls(org.apache.logging.log4j.Level
> >>>>>>>>> logLevel, ClassLoader cl) {
> >>>>>>>>> if (null == cl) {
> >>>>>>>>> return;
> >>>>>>>>> }
> >>>>>>>>> boolean loaderNameLogged = false;
> >>>>>>>>> if (cl instanceof URLClassLoader) {
> >>>>>>>>> final URLClassLoader urlCl = (URLClassLoader) cl;
> >>>>>>>>> final URL[] urls = urlCl.getURLs();
> >>>>>>>>> if (null != urls) {
> >>>>>>>>> LOG.log(logLevel, " Class loader name={}, URL
> >>>>>> count={}",
> >>>>>>>>> cl.getClass().getName(), urls.length);
> >>>>>>>>> loaderNameLogged = true;
> >>>>>>>>> for (int idx = 0; idx < urls.length; idx++) {
> >>>>>>>>> final URL url = urls[idx];
> >>>>>>>>> LOG.log(logLevel, " URL[{}]={}", idx,
> >>> url);
> >>>>>>>>> }
> >>>>>>>>> }
> >>>>>>>>> }
> >>>>>>>>> if (!loaderNameLogged) {
> >>>>>>>>> LOG.log(logLevel, " Class loader name={}",
> >>>>>>>>> cl.getClass().getName());
> >>>>>>>>> }
> >>>>>>>>> logClassLoaderUrls(logLevel, cl.getParent());
> >>>>>>>>> }
> >>>>>>>>>
> >>>>>>>>> ..snip..
> >>>>>>>>>
> >>>>>>>>> LOG.info("java.class.path={}",
> >>>>>>>>> System.getProperty("java.class.path"));
> >>>>>>>>> LOG.debug("Class loader URLs:");
> >>>>>>>>>
> >>> logClassLoaderUrls(org.apache.logging.log4j.Level.DEBUG,
> >>>>>>>>> ctx.getClassLoader());
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [4]
> >>>>>>>>> cat log-extract.txt | cut -c137- | sed -e
> >>>>>> "s+/usr/local/tomcat++g" -e
> >>>>>>>>> "s+/usr/local/tomcat/webapps/exclaim++g" -e "s+\[[0-9]*\]++"
> >>>>>>>>> and I manually sorted the "sections" using vim, with the "sort"
> >>>>>> command.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [5]
> >>>>>>>>> # Debian with OpenJDK
> >>>>>>>>> #FROM tomcat:9.0.58-jre11-openjdk AS tomcat_package
> >>>>>>>>> # AWS Linux 2 with Corretto
> >>>>>>>>> FROM tomcat:9.0.58-jdk11-corretto AS tomcat_package
> >>>>>>>>>
> >>>>>>>>> # Set the timezone to Canada/Eastern
> >>>>>>>>> # For Debian
> >>>>>>>>> #RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime &&
> >>> echo
> >>>>>>>>> "Canada/Eastern" > /etc/timezone && dpkg-reconfigure -f
> >>> noninteractive
> >>>>>>>>> tzdata
> >>>>>>>>> # For AWS Linux
> >>>>>>>>> RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime &&
> >>> sed -i
> >>>>>>>>> 's/\"UTC\"/\"Canada\/Eastern\"/' /etc/sysconfig/clock # and
> reboot,
> >>>>>> but
> >>>>>>>>> will skip as container will be created and ran separately
> >>>>>>>>>
> >>>>>>>>> # Install a postgresql client for testing and the required font
> >>>>>> packages
> >>>>>>>>> needed by JasperReports
> >>>>>>>>> # For Debian:
> >>>>>>>>> #RUN echo "deb http://deb.debian.org/debian bullseye contrib
> >>>>>> non-free"
> >>>>>>>>>>> /etc/apt/sources.list && apt-get update && apt-get install -y
> >>>>>>>>> postgresql-client ttf-mscorefonts-installer
> >>>>>>>>> # For AWS Linux:
> >>>>>>>>> RUN yum -q list installed epel-release.noarch &>/dev/null && echo
> >>>>>>>>> 'epel-release already installed' || yum install -y
> >>>>>>>>>
> >>>>>>
> >>> https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
> >>>>>>>>> RUN yum install -y cabextract xorg-x11-font-utils fontconfig
> >>>>>>>>> RUN rpm -q msttcore-fonts-installer-2.6-1 &>/dev/null && echo
> >>>>>>>>> 'msttcore-fonts-installer-2.6-1 already installed' || rpm -i
> >>>>>>>>>
> >>>>>>
> >>>
> https://iweb.dl.sourceforge.net/project/mscorefonts2/rpms/msttcore-fonts-installer-2.6-1.noarch.rpm
> >>>>>>>>>
> >>>>>>>>> # Add a health-check script to wait for exclaim to be running --
> >>>>>> allow at
> >>>>>>>>> least 10 minutes for it to start
> >>>>>>>>> COPY health_check.sh /usr/local/bin/
> >>>>>>>>> HEALTHCHECK --interval=10s --timeout=5s --start-period=10m
> >>>>>> --retries=10
> >>>>>>>>> CMD /usr/local/bin/health_check.sh
> >>>>>>>>>
> >>>>>>>>> # Set up the resource folders needed by application and make it a
> >>>>>>>>> # volume so it persists across container restarts
> >>>>>>>>> COPY resources/ /resources/
> >>>>>>>>> VOLUME /resources
> >>>>>>>>>
> >>>>>>>>> # Map some of the Tomcat folders to volumes so they persist
> between
> >>>>>>>>> restarts of the container
> >>>>>>>>> VOLUME /usr/local/tomcat/logs
> >>>>>>>>> VOLUME /usr/local/tomcat/webapps
> >>>>>>>>> VOLUME /usr/local/tomcat/work
> >>>>>>>>> VOLUME /usr/local/tomcat/conf
> >>>>>>>>>
> >>>>>>>>> # Enable connecting from anywhere for JPDA (req'd for Java 9+)
> >>>>>>>>> ENV JPDA_ADDRESS=*:8000
> >>>>>>>>>
> >>>>>>>>> CMD ["catalina.sh","jpda","run"]
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> [6]
> >>>>>>>>> JDK on MacOS is:
> >>>>>>>>> openjdk version "11.0.13" 2021-10-19 LTS
> >>>>>>>>> OpenJDK Runtime Environment Corretto-11.0.13.8.1 (build
> >>> 11.0.13+8-LTS)
> >>>>>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.13.8.1 (build
> 11.0.13+8-LTS,
> >>>>>> mixed
> >>>>>>>>> mode)
> >>>>>>>>>
> >>>>>>>>> JDK on Windows is:
> >>>>>>>>> openjdk 11.0.14 2022-01-18 LTS
> >>>>>>>>> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build
> >>> 11.0.14+9-LTS)
> >>>>>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build
> 11.0.14+9-LTS,
> >>>>>> mixed
> >>>>>>>>> mode)
> >>>>>>>>>
> >>>>>>>>> JDK on Docker with AWS Linux 2 is:
> >>>>>>>>> openjdk 11.0.14 2022-01-18 LTS
> >>>>>>>>> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build
> >>> 11.0.14+9-LTS)
> >>>>>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build
> 11.0.14+9-LTS,
> >>>>>> mixed
> >>>>>>>>> mode)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Feb 7, 2022 at 9:24 PM Rob Sargent <
> rsargent@xmission.com>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 2/7/22 19:13, Robert Turner wrote:
> >>>>>>>>>>> So, I've gone back and double-checked as much as I can (Tomcat
> >>>>>>>>>> version, JDK
> >>>>>>>>>>> version), and the classpath, and I have identical classpaths in
> >>> both
> >>>>>>>>>>> environments (except the sort order of the URLs per "level" of
> >>>>>>>>>>> ClassLoader), and I've re-verified the behaviour:
> >>>>>>>>>>> - fails in the docker environment
> >>>>>>>>>>> - works locally
> >>>>>>>>>>> (Same WAR file on both).
> >>>>>>>>>>>
> >>>>>>>>>>> I guess I'm in to one of the following approaches next:
> >>>>>>>>>>> - build a debug version of Tomcat, and step through the
> code
> >>> that
> >>>>>>>>>> breaks
> >>>>>>>>>>> and see if I can figure out why (not that I have time to do
> this
> >>> of
> >>>>>>>>>>> course...but might be necessary)
> >>>>>>>>>>> - construct a trivial application reproduction, along with
> >>> docker
> >>>>>>>>>> layout,
> >>>>>>>>>>> and see if anyone else can reproduce... (assuming anyone else
> has
> >>>>>> time
> >>>>>>>>>> to
> >>>>>>>>>>> do that of course...)
> >>>>>>>>>>>
> >>>>>>>>>>> Anyone got any suggestions of what to look into next?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Feb 7, 2022 at 5:05 PM Rob Sargent <
> >>> rsargent@xmission.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> As you appear averse to sharing names of things, seems you will
> >>> need
> >>>>>> to
> >>>>>>>>>> do a lot of extra clean-up before you can share a docker image.
> >>> Make
> >>>>>>>>>> sure you're clear on what NoClassDefFoundError indicates
> (present
> >>> at
> >>>>>>>>>> compile time, absent at runtime) and how that translates to
> >>>>>> docker-ness
> >>>>>>>>>> (of which I know naught).
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>
> >>>
> >
>