You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/09/03 17:46:35 UTC
[GitHub] [pinot] kkrugler opened a new issue #7393: Plugin ClassNotFoundExceptions when running segment building jobs
kkrugler opened a new issue #7393:
URL: https://github.com/apache/pinot/issues/7393
This is with Pinot 0.8.0, Hadoop 2.7.1, Java 8. When starting the job, we get the error below. This doesn't happen with Pinot 0.7.1
```
21/09/03 17:05:49 INFO batch.IngestionJobLauncher: Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentGenerationJobRunner
21/09/03 17:05:49 ERROR command.LaunchDataIngestionJobCommand: Got exception to kick off standalone data ingestion job -
java.lang.RuntimeException: Failed to create IngestionJobRunner instance for class - org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentGenerationJobRunner
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:139)
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:103)
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:143)
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main(LaunchDataIngestionJobCommand.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: java.lang.ClassNotFoundException: org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentGenerationJobRunner
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at org.apache.pinot.spi.plugin.PluginClassLoader.loadClass(PluginClassLoader.java:77)
at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:302)
at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:273)
at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:254)
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:137)
... 9 more
```
Looking earlier in the logs, I see:
```
21/09/03 17:05:49 INFO plugin.PluginManager: Trying to load plugin [pinot-batch-ingestion-hadoop] from location [/ebs/workflow/apache-pinot-0.8.0-SNAPSHOT-bin/plugins/pinot-batch-ing
estion/pinot-batch-ingestion-hadoop]
21/09/03 17:05:49 INFO plugin.PluginManager: Successfully loaded plugin [pinot-batch-ingestion-hadoop] from jar files: [file:/ebs/workflow/apache-pinot-0.8.0-SNAPSHOT-bin/plugins/pin
ot-batch-ingestion/pinot-batch-ingestion-hadoop/pinot-batch-ingestion-hadoop-0.8.0-SNAPSHOT-shaded.jar]
21/09/03 17:05:49 INFO plugin.PluginManager: Successfully Loaded plugin [pinot-batch-ingestion-hadoop] from dir [/ebs/workflow/apache-pinot-0.8.0-SNAPSHOT-bin/plugins/pinot-batch-ing
estion/pinot-batch-ingestion-hadoop]
```
So it seems like the plugin was loaded successfully.
If we change the run script to set `HADOOP_CLASSPATH` to explicitly include the `pinot-batch-ingestion-hadoop-0.8.0-SNAPSHOT-shaded.jar`, then we get past this first ClassNotFoundException, but then it fails in the Hadoop map task with a similar issue, even though we see the plugin tarball in the Hadoop distributed cache, and it has the expected set of plugins, and we see the plugin getting loaded (added extra logging, which is why the above build version is 0.8.0-SNAPSHOT).
Kulbir Nijjer had the same issue with his Spark-based segment building job, which he worked around via:
> @Ken Krugler yes there are some oddities around how things are working in certain Spark versions vs. others with same exact steps and we are looking into it. On Spark side in between using dependencyJarDir in ingestion yaml (need to manually upload all plugin jars to HDFS/S3/GCS path) and using --jars has worked well in my testing. Just by using -Dplugins in the command should cause jars to be packaged and added to YARN distributed cache but somehow that's not always happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #7393: Plugin ClassNotFoundExceptions when running segment building jobs
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7393:
URL: https://github.com/apache/pinot/issues/7393#issuecomment-924158376
See useful documentation on [Flink's classloading](https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/). We don't yet support building segments with Flink, but we will eventually want to do so. And this page has a pretty good explanation of common classloading fun with a distributed system that runs user code (Hadoop is similar, I assume the same is true for Spark).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #7393: Plugin ClassNotFoundExceptions when running segment building jobs
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7393:
URL: https://github.com/apache/pinot/issues/7393#issuecomment-913187119
@elonazoulay - when I build a version of Pinot from the release-0.8.0 tag, but revert your change in `PluginClassLoader.java`, then the Hadoop job runs successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #7393: Plugin ClassNotFoundExceptions when running segment building jobs
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7393:
URL: https://github.com/apache/pinot/issues/7393#issuecomment-924159735
@richardstartin - I don't know if you want to keep using this issue for fixes/improvements to plugin classloading, so I haven't closed it. I would favor a new issue that describes the more general problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #7393: Plugin ClassNotFoundExceptions when running segment building jobs
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #7393:
URL: https://github.com/apache/pinot/issues/7393#issuecomment-926923172
I'm going to close this issue, as PR #7400 resolves the specific problem mentioned here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler closed issue #7393: Plugin ClassNotFoundExceptions when running segment building jobs
Posted by GitBox <gi...@apache.org>.
kkrugler closed issue #7393:
URL: https://github.com/apache/pinot/issues/7393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org