[GitHub] [hudi] Kavin88 edited a comment on issue #2406: [SUPPORT] HoodieMultiTableDeltastreamer - Bypassing SchemaProvider-Class requirement for ParquetDFS

[GitBox <gi...@apache.org>]

Kavin88 edited a comment on issue #2406:
URL: https://github.com/apache/hudi/issues/2406#issuecomment-953064052


   @nsivabalan @vinothchandar  Facing below Null Pointer Exception in MultitableDeltastreamer. As per above comments, this issue had been fixed. Still the error is occuring. 
   
   Hudi - 0.8.0
   
   spark-submit --jars "/usr/lib/hudi/hudi-utilities-bundle_2.12-0.8.0-amzn-0.jar,/usr/lib/spark/external/lib/spark-avro_2.12-3.1.2-amzn-0.jar" \
   --master yarn --deploy-mode cluster \
   --class org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer /usr/lib/hudi/hudi-utilities-bundle_2.12-0.8.0-amzn-0.jar \
   --props s3://test/file.properties \
   --config-folder s3://test/config/ \
   --source-class org.apache.hudi.utilities.sources.ParquetDFSSource  \
   --target-table dummy_table \
   --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
   --table-type COPY_ON_WRITE
   
   Property file :
   hoodie.deltastreamer.ingestion.tablesToBeIngested=default.table1,default.table2
   hoodie.deltastreamer.ingestion.default.table1.configFile=s3://test/config/config_table1.properties
   hoodie.deltastreamer.ingestion.default.table2.configFile=s3://test/config/config_table2.properties
   
   Config file for table1  :
   hoodie.deltastreamer.source.dfs.root=s3://input/
   hoodie.datasource.write.partitionpath.field=
   hoodie.deltastreamer.ingestion.targetBasePath=/tmp/hudi/table1/
   hoodie.deltastreamer.schemaprovider.source.schema.file=s3://test/source.avsc
   hoodie.deltastreamer.schemaprovider.source.schema.file=s3://test/target.avsc
   hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator
   
   INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.NullPointerException
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.resetTarget(HoodieMultiTableDeltaStreamer.java:345)
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.populateTableExecutionContextList(HoodieMultiTableDeltaStreamer.java:124)
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.<init>(HoodieMultiTableDeltaStreamer.java:81)
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.main(HoodieMultiTableDeltaStreamer.java:204)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org