You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/11/12 02:21:33 UTC
[GitHub] [hudi] jon-qj opened a new issue #2247: [SUPPORT] org.eclipse.jetty.util.thread.ThreadPool$SizedThreadPool.getThreadPoolBudget()
jon-qj opened a new issue #2247:
URL: https://github.com/apache/hudi/issues/2247
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
spark-shell :
java.lang.NoSuchMethodError: org.eclipse.jetty.util.thread.ThreadPool$SizedThreadPool.getThreadPoolBudget()Lorg/eclipse/jetty/util/thread/ThreadPoolBudget;
at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:179)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:281)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:394)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at io.javalin.core.util.JettyServerUtil.initialize(JettyServerUtil.kt:110)
at io.javalin.Javalin.start(Javalin.java:140)
at io.javalin.Javalin.start(Javalin.java:119)
at org.apache.hudi.timeline.service.TimelineService.startServiceOnPort(TimelineService.java:111)
at org.apache.hudi.timeline.service.TimelineService.startService(TimelineService.java:134)
at org.apache.hudi.client.embedded.EmbeddedTimelineService.startServer(EmbeddedTimelineService.java:72)
at org.apache.hudi.client.embedded.EmbeddedTimelineServerHelper.createEmbeddedTimelineService(EmbeddedTimelineServerHelper.java:54)
at org.apache.hudi.client.AbstractHoodieClient.startEmbeddedServerView(AbstractHoodieClient.java:103)
at org.apache.hudi.client.AbstractHoodieClient.<init>(AbstractHoodieClient.java:73)
at org.apache.hudi.client.AbstractHoodieWriteClient.<init>(AbstractHoodieWriteClient.java:128)
at org.apache.hudi.client.AbstractHoodieWriteClient.<init>(AbstractHoodieWriteClient.java:115)
at org.apache.hudi.client.SparkRDDWriteClient.<init>(SparkRDDWriteClient.java:70)
at org.apache.hudi.DataSourceUtils.createHoodieClient(DataSourceUtils.java:190)
at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$3.apply(HoodieSparkSqlWriter.scala:159)
at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$3.apply(HoodieSparkSqlWriter.scala:159)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:159)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:125)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228)
**To Reproduce**
Steps to reproduce the behavior:
1. running spark-shell with Hudi
sudo -u hive spark-shell --jars 'hdfs:///user/hudi/*'
2.import jars with dependences
3.load the data from hive table whose format is textfile into a df
4. df.write.format("org.apache.hudi").
| options(getQuickstartWriteConfigs).
| option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY,"pk_company").
| option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY,"update_time").
| option("hoodie.insert.shuffle.parallelism","10").
| option("hoodie.upsert.shuffle.parallelism","10").
| option(HoodieWriteConfig.TABLE_NAME,"test.test_companies").
| mode(SaveMode.Append).save("hdfs:///user/hive/external/test/test_companies")
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : Hudi-Master(mvn clean package -DskipTests -DskipITs -Dhadoop.version=3.0.0-cdh6.3.2)
* Spark version :2.4.0-cdh6.3.2
* Hive version :2.1.1-cdh6.3.2
* Hadoop version :3.0.0-cdh6.3.2
* Storage (HDFS/S3/GCS..) :HDFS
* Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jon-qj commented on issue #2247: [SUPPORT] org.eclipse.jetty.util.thread.ThreadPool$SizedThreadPool.getThreadPoolBudget()
Posted by GitBox <gi...@apache.org>.
jon-qj commented on issue #2247:
URL: https://github.com/apache/hudi/issues/2247#issuecomment-729485960
> You just need spark-bundle to run in spark-shell. I think Hadoop 3.0.0 is bringing a different version of eclipse which is causing this issue. Can you try with Hadoop 2.9.x/2.8.x to see if the problem still exists ?
I've made it, thanks a lot 💯
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jon-qj closed issue #2247: [SUPPORT] org.eclipse.jetty.util.thread.ThreadPool$SizedThreadPool.getThreadPoolBudget()
Posted by GitBox <gi...@apache.org>.
jon-qj closed issue #2247:
URL: https://github.com/apache/hudi/issues/2247
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2247: [SUPPORT] org.eclipse.jetty.util.thread.ThreadPool$SizedThreadPool.getThreadPoolBudget()
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2247:
URL: https://github.com/apache/hudi/issues/2247#issuecomment-725896914
You just need spark-bundle to run in spark-shell. I think Hadoop 3.0.0 is bringing a different version of eclipse which is causing this issue. Can you try with Hadoop 2.9.x/2.8.x to see if the problem still exists ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jon-qj commented on issue #2247: [SUPPORT] org.eclipse.jetty.util.thread.ThreadPool$SizedThreadPool.getThreadPoolBudget()
Posted by GitBox <gi...@apache.org>.
jon-qj commented on issue #2247:
URL: https://github.com/apache/hudi/issues/2247#issuecomment-729313447
> You just need spark-bundle to run in spark-shell. I think Hadoop 3.0.0 is bringing a different version of eclipse which is causing this issue. Can you try with Hadoop 2.9.x/2.8.x to see if the problem still exists ?
I didn't try with Hadoop 2.9.x/2.8.x and just used another release(0.5.3),but I got another error.
I test it using test-cluster with the same configs,it shows no error.
Perhaps I missed something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org