You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/01/22 02:37:54 UTC

[GitHub] [incubator-hudi] lamber-ken opened a new pull request #1268: [HUDI-569] Fix java.lang.ClassNotFoundException in quickstart example

lamber-ken opened a new pull request #1268: [HUDI-569] Fix java.lang.ClassNotFoundException in quickstart example
URL: https://github.com/apache/incubator-hudi/pull/1268
 
 
   ## What is the purpose of the pull request
   
   Spark-2.4.4 avro data source guide says "The spark-avro module is external", and met exception 
   
   ```
   java.lang.NoClassDefFoundError: org/apache/spark/sql/avro/SchemaConverters$
     at org.apache.hudi.AvroConversionUtils$.convertStructTypeToAvroSchema(AvroConversionUtils.scala:87)
     at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:92)
     at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:91)
     at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
     at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
     at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
     at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
     at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
     at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
     at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
     at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
     at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
     at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
     at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
     at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:656)
     at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:656)
     at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
     at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:656)
   
   ```
   
   ## Brief change log
   
     - Add *--packages org.apache.spark:spark-avro_2.11:2.4.4*
   
   ## Verify this pull request
   
   Test quickstart guide.
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example
URL: https://github.com/apache/incubator-hudi/pull/1268#issuecomment-577014164
 
 
   > ```
   >  --packages org.apache.hudi:hudi-spark-bundle:0.5.0-incubating \
   >      --packages org.apache.spark:spark-avro_2.11:2.4.4 \
   > ```
   > 
   > why does 0.5.0 need `org.apache.spark:spark-avro` .. I think you mean 0.5.1 upcoming release? The idea is to get the *.md files ready for the release and then publish the site when the artifacts are actually ready..
   > 
   > So, I think we can close this PR and track it via the existing issue..?
   
   RIght, this pr is for upcoming release. Got it, close this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example

Posted by GitBox <gi...@apache.org>.
lamber-ken commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example
URL: https://github.com/apache/incubator-hudi/pull/1268#issuecomment-577018734
 
 
   > @lamber-ken this is already tracked as a part of work @bhasudha is doing ..
   
   Thanks for reminding me.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example
URL: https://github.com/apache/incubator-hudi/pull/1268#issuecomment-577008058
 
 
   ```
    --packages org.apache.hudi:hudi-spark-bundle:0.5.0-incubating \
        --packages org.apache.spark:spark-avro_2.11:2.4.4 \
   ```
   
   why does 0.5.0 need `org.apache.spark:spark-avro` .. I think you mean 0.5.1 upcoming release? The idea is to get the *.md files ready for the release and then publish the site when the artifacts are actually ready.. 
   
   So, I think we can close this PR and track it via the existing issue..? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] lamber-ken closed pull request #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example

Posted by GitBox <gi...@apache.org>.
lamber-ken closed pull request #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example
URL: https://github.com/apache/incubator-hudi/pull/1268
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1268: [HUDI-569] Fix java.lang. NoClassDefFoundError in quickstart example
URL: https://github.com/apache/incubator-hudi/pull/1268#issuecomment-577007640
 
 
   @lamber-ken this is already tracked as a part of work @bhasudha is doing .. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services