You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/30 09:10:13 UTC
[GitHub] [hudi] nleena123 opened a new issue, #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
nleena123 opened a new issue, #5714:
URL: https://github.com/apache/hudi/issues/5714
Hi Team ,
Getting the below issue in hudi job while executing through databricks.
ERROR details:-
=============
NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
We are running the hudi code through databricks .
**Environment Description**
* Hudi version : 0.11.0
* Spark version : 3.1.2
* Scala Version :2.12
* Storage (HDFS/S3/GCS..) : AZURE Blob Storage
* AVRO SOURCE we are using to read the kafka data .
* Used 9.1 LTS Databricks RUNTIME version
With same configurations job succeeded in non prod environment getting error in prod environment
Please find the attached details and log file ,Let me know the resolution steps
Full log -
=======
[log4j-active (43).txt](https://github.com/apache/hudi/files/8797106/log4j-active.43.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5714:
URL: https://github.com/apache/hudi/issues/5714#issuecomment-1142180448
The new exception is application-level error, specific to your data and scenario, not related to dependency. Please examine your executor logs and data for details. Closing this as the original problem about the dependency was resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5714:
URL: https://github.com/apache/hudi/issues/5714#issuecomment-1141046717
can you share the complete spark-submit command with args? need to see what jars are used.
> With same configurations job succeeded in non prod environment getting error in prod environment
so this is related to your environment discrepancy. are you able to check from your side what jars not present in your prod env?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5714:
URL: https://github.com/apache/hudi/issues/5714#issuecomment-1141106226
spark-avro package you used is mismatch; it's for spark 3.2.1 but your main spark is 3.1. can you refer to 0.11 release note migration guide? spark-avro is not required. And also you don't need spark3-bundle, as you already used utilities-bundle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nleena123 commented on issue #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
Posted by GitBox <gi...@apache.org>.
nleena123 commented on issue #5714:
URL: https://github.com/apache/hudi/issues/5714#issuecomment-1141211452
HI @xushiyan ,
I have made the changes suggested by you, removed spark3-bundle and spark_avro and spark3 bundle dependencies are removed and submitted the job it's giving the another error .
**ERROR details:-**
============
**HoodieException: Commit 20220530133502327 failed and rolled-back !**
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:649)
Please find the attached screen shot and log file .
[log4j-active (44).txt](https://github.com/apache/hudi/files/8799012/log4j-active.44.txt)
<img width="715" alt="Screenshot 2022-05-30 at 7 39 59 PM" src="https://user-images.githubusercontent.com/101168016/171010255-d42ab198-cf61-417b-8bab-28fa95fc1a56.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nleena123 commented on issue #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
Posted by GitBox <gi...@apache.org>.
nleena123 commented on issue #5714:
URL: https://github.com/apache/hudi/issues/5714#issuecomment-1141099747
Hi @xushiyan ,
i verified the both non prod and prod jars , it is same in both environments .
Please find the attached screenshot of Databrick job details.
<img width="1680" alt="Screenshot 2022-05-30 at 5 55 43 PM" src="https://user-images.githubusercontent.com/101168016/170992046-52de53db-8da1-4b93-afc9-b8efaa53e9b9.png">
Parameters that we used for the job :-
["--table-type","COPY_ON_WRITE","--source-ordering-field","CDC_TS","--source-limit","1000","--source-class","com.optum.df.hudi.sources.DFAvroKafkaSource","--target-base-path","/mnt/ulp/dataassets-lake/metrics/","--target-table","metrics","--schemaprovider-class","org.apache.hudi.utilities.schema.SchemaRegistryProvider","--props","/mnt/ulp/artifacts/properties/metrics.properties"]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #5714: NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
URL: https://github.com/apache/hudi/issues/5714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org