[GitHub] [hudi] gopi-t2s commented on issue #2406: [SUPPORT] HoodieMultiTableDeltastreamer - Bypassing SchemaProvider-Class requirement for ParquetDFS

[GitBox <gi...@apache.org>]

gopi-t2s commented on issue #2406:
URL: https://github.com/apache/hudi/issues/2406#issuecomment-804020555


   Hi @nsivabalan, 
   @SureshK-T2S and me working together to setup multi table delta streamer
   
   I downloaded the latest version(0.7.0) of hudi-utilities-bundle.jar from MAVEN reporisitory(https://mvnrepository.com/artifact/org.apache.hudi/hudi-utilities-bundle_2.11/0.7.0) and tried to run the spark-submit multi table delta streamer command without providing the schema provider class(hope this is not mandatory now after this fix #2577).
   
   But still receiving the same error mentioned above by Suresh.
   **ERROR LOG:**
   `Exception in thread "main" java.lang.NullPointerException
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.populateSchemaProviderProps(HoodieMultiTableDeltaStreamer.java:150)
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.populateTableExecutionContextList(HoodieMultiTableDeltaStreamer.java:130)
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.<init>(HoodieMultiTableDeltaStreamer.java:80)
   	at org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer.main(HoodieMultiTableDeltaStreamer.java:203)`
   
   **SPARK SUBMIT COMMAND**
   `spark-submit --class org.apache.hudi.utilities.deltastreamer.HoodieMultiTableDeltaStreamer ` `ls ~/hudi/hudi-utilities-bundle_2.11-0.7.0.jar``\
     --table-type COPY_ON_WRITE \
    --props s3://path/s3_source.properties \
     --config-folder s3://folder-path \
     --source-class org.apache.hudi.utilities.sources.ParquetDFSSource \
    --source-ordering-field updated_at \
     --base-path-prefix s3://object --target-table dummy_table --op UPSERT`
   
   Do I miss anything here or the above PR is not merged in 0.7.0 version maven jar.
   Could you share your valuable thoughts here.
   
   Thank you..
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org