Inconsistent Class-loading issue with recent build

[Jason Altekruse <al...@gmail.com>]

Drillers,

Sudheesh was trying to run some queries against the parquet files we
package with the build and was seeing this error message [1] in embedded
mode. I tried doing a build on my machine and ran the same query and did
not see the problem, we also exchanged builds and he saw the issue on his
machine with my build and I again did not see it with his.

There are two issues here. Firstly he only saw this error because he was in
embedded mode, it was not being propagated back to the server, it was just
being sent to stdout or stderr. This seems likely to be caused by the fact
that this is actually an Error, not an exception. Do we have anywhere where
we attempt catch Errors? I understand that these are not supposed to happen
in execution, so we would likely not try to keep the Drillbit going if we
see an Error, I am curious if we try to catch them and print something like
a system failure message before exiting.

The second issue it the particular type of error. Obviously this type of
issue should be caught in normal compilation, or be caused by a mismatch
between classes compiled against different versions of a dependency where
access changed to the superclass method. As we haven't updated the parquet
jar in several weeks, this is not seeming to be the problem. Additionally,
I was able to run the same query on my machine without problems, so it
seems unlikely to be an issue with the compiled classes and dependencies.

I tried searching for the error message and I found this discussion [0]
that claims if two classes are loaded with different classloaders that they
will not be considered the same and can cause issues like this. I do not
fully understand if Drill is managing multiple classloaders, but it seems
like a likely candidate for explaining the issue with all of the pluggable
components we load through reflection and classpath scanning. Any thoughts
for debugging or fixing this issue?

[0] - https://github.com/raphw/byte-buddy/issues/1#issuecomment-44696722
[1]
Exception in thread "WorkManager-2" java.lang.IllegalAccessError: class
org.apache.drill.exec.store.parquet.DirectCodecFactory cannot access its
superclass parquet.hadoop.CodecFactory
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.apache.drill.exec.store.parquet.ParquetScanBatchCreator.getBatch(ParquetScanBatchCreator.java:129)
at
org.apache.drill.exec.store.parquet.ParquetScanBatchCreator.getBatch(ParquetScanBatchCreator.java:55)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:140)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:163)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:121)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:163)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:121)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:163)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:121)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:163)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:121)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:163)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:121)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:163)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:96)
at
org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:77)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:199)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)