You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/17 22:20:03 UTC

[GitHub] [hudi] rubenssoto opened a new issue #3814: [SUPPORT] Error o Trying to create a table using Spark SQL

rubenssoto opened a new issue #3814:
URL: https://github.com/apache/hudi/issues/3814


   Hello,
   
   I have spark server using EMR 6.2 and I started the thrift server using the following command
   ```
   sudo /usr/lib/spark/sbin/start-thriftserver.sh --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.9.0,org.apache.spark:spark-avro_2.12:3.0.1 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
   ```
   
   I made a JDBC connection using a SQL Client, so I tried to execute:
   ```
   create table if not exists dbt_test.hudi_table0 (
     id int, 
     name string, 
     price double
   ) using hudi
   options (
     type = 'cow',
     primaryKey = 'id'
   );
   ```
   
   And I had the error
   ```
   SQL Error: Error running query: java.lang.ClassNotFoundException: hudi.DefaultSource
   ```
   
   Could you help me with this?
   
   Thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan closed issue #3814: [SUPPORT] Error o Trying to create a table using Spark SQL

Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3814:
URL: https://github.com/apache/hudi/issues/3814


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on issue #3814: [SUPPORT] Error o Trying to create a table using Spark SQL

Posted by GitBox <gi...@apache.org>.
codope commented on issue #3814:
URL: https://github.com/apache/hudi/issues/3814#issuecomment-945572675


   Does the same `create table` statement work with spark-sql? I believe both spark thrift server and sql share the same spark context so it may error out again. Just to conform, can you run with spark-sql?
   
   If it works, then maybe you could check from jdbc client whether `show tables` is listing the table that was created earlier using spark-sql? 
   
   FYI, [DefaultSource](https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala) is an entrypoint in Hudi, which creates the relation for reading/writing to Hudi tables. Most likely, Hudi jar is missing from the classpath.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on issue #3814: [SUPPORT] Error o Trying to create a table using Spark SQL

Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3814:
URL: https://github.com/apache/hudi/issues/3814#issuecomment-946390788


   @rubenssoto since you're on EMR, please use EMR pre-installed hudi jars instead of open source ones
   
   ```
   --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.9.0,org.apache.spark:spark-avro_2.12:3.0.1
   ```
   
   change to
   
   ```
   --jars /usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/spark/external/lib/spark-avro.jar
   ```
   
   See more from
   https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-installation-and-configuration.html
   https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-work-with-dataset.html
   
   
   And please engage with AWS support for EMR specific setup problems.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org