You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/07/29 15:39:33 UTC
[GitHub] [accumulo] milleruntime opened a new issue, #2835: Hadoop LzoCodec ClassNotFoundException
milleruntime opened a new issue, #2835:
URL: https://github.com/apache/accumulo/issues/2835
I saw a ClassNotFoundException for one of the Hadoop codecs while doing some testing. I was using Uno to test my per-table crypto changes in #2197 and the error was thrown while the tserver was starting up.
<pre>
export HADOOP_VERSION=${HADOOP_VERSION:-3.3.3}
export ZOOKEEPER_VERSION=${ZOOKEEPER_VERSION:-3.7.1}
export ACCUMULO_HOME="~/workspace/uno/install/accumulo-2.1.0-SNAPSHOT"
</pre>
It is possible this is related to changes in my branch but I wanted to capture the exception for now.
<pre>
2022-07-29T11:15:58,745 [bcfile.CompressionAlgorithm] DEBUG: Unable to load codec class org.apache.hadoop.io.compress.LzoCodec for io.compression.codec.lzo.class
java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.LzoCodec
at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:315) ~[?:?]
at org.apache.accumulo.core.file.rfile.bcfile.CompressionAlgorithm.createNewCodec(CompressionAlgorithm.java:298) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.bcfile.CompressionAlgorithm.createNewCodec(CompressionAlgorithm.java:147) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.bcfile.CompressionAlgorithm.initCodec(CompressionAlgorithm.java:277) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.bcfile.CompressionAlgorithm.<init>(CompressionAlgorithm.java:139) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.bcfile.Compression.lambda$static$0(Compression.java:65) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.apache.accumulo.core.file.rfile.bcfile.Compression.<clinit>(Compression.java:66) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.bcfile.BCFile$MetaIndexEntry.<init>(BCFile.java:814) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.bcfile.BCFile$MetaIndex.<init>(BCFile.java:774) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.bcfile.BCFile$Reader.<init>(BCFile.java:631) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBCFile(CachableBlockFile.java:171) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader$BCFileLoader.load(CachableBlockFile.java:221) ~[accumulo-core-2.1.0-SNAPSHOT.jar:
at org.apache.accumulo.core.file.blockfile.cache.lru.SynchronousLoadingBlockCache.getBlock(SynchronousLoadingBlockCache.java:127) ~[accumulo-core-2.1.0-SNAPSHOT.jar:
at org.apache.accumulo.core.file.blockfile.cache.lru.SynchronousLoadingBlockCache.resolveDependencies(SynchronousLoadingBlockCache.java:64)
at org.apache.accumulo.core.file.blockfile.cache.lru.SynchronousLoadingBlockCache.getBlock(SynchronousLoadingBlockCache.java:109) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:376) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.RFile$Reader.<init>(RFile.java:1164) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.RFile$Reader.<init>(RFile.java:1256) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.RFileOperations.getReader(RFileOperations.java:55) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.rfile.RFileOperations.openReader(RFileOperations.java:70) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.DispatchingFileFactory.openReader(DispatchingFileFactory.java:85) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.file.FileOperations$ReaderBuilder.build(FileOperations.java:447) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.fs.FileManager.reserveReaders(FileManager.java:310) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.fs.FileManager$ScanFileManager.openFiles(FileManager.java:493) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.fs.FileManager$ScanFileManager.openFiles(FileManager.java:502) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.ScanDataSource.createIterator(ScanDataSource.java:163) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.ScanDataSource.iterator(ScanDataSource.java:120) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.iteratorsImpl.system.SourceSwitchingIterator.seek(SourceSwitchingIterator.java:230) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.TabletBase.nextBatch(TabletBase.java:258) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.tablet.Scanner.read(Scanner.java:108) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.scan.NextBatchTask.run(NextBatchTask.java:74) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer.run(ScanSession.java:54) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.TabletServerResourceManager.executeReadAhead(TabletServerResourceManager.java:778) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.ThriftScanClientHandler.continueScan(ThriftScanClientHandler.java:227) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.tserver.ThriftScanClientHandler.startScan(ThriftScanClientHandler.java:192) ~[accumulo-tserver-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2835: Hadoop LzoCodec ClassNotFoundException
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2835:
URL: https://github.com/apache/accumulo/issues/2835#issuecomment-1199652686
Looks like since the lzo library is GPL, it's not distributed with Hadoop and you have to configure it manually. Here are some old instructions https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.4.2-Win/bk_HDP_Install_Win/content/LZOCompression.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion closed issue #2835: Hadoop LzoCodec ClassNotFoundException
Posted by GitBox <gi...@apache.org>.
dlmarion closed issue #2835: Hadoop LzoCodec ClassNotFoundException
URL: https://github.com/apache/accumulo/issues/2835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2835: Hadoop LzoCodec ClassNotFoundException
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2835:
URL: https://github.com/apache/accumulo/issues/2835#issuecomment-1199754330
> Looks like the LzoCodec was removed a long time ago due to license issues
Interesting. I wonder if we should not try to load it by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2835: Hadoop LzoCodec ClassNotFoundException
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2835:
URL: https://github.com/apache/accumulo/issues/2835#issuecomment-1199606916
Looks like the LzoCodec was removed in 3.3.0 due to license issues. Comments indicate that there might be a replacement in the future. https://issues.apache.org/jira/browse/HADOOP-17124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2835: Hadoop LzoCodec ClassNotFoundException
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2835:
URL: https://github.com/apache/accumulo/issues/2835#issuecomment-1199810936
I don't see a problem trying to load it by default. But, the verbose stack trace isn't particularly useful for ClassNotFoundException. It could be a shorter debug message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org