You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/03 08:05:35 UTC

[GitHub] [hudi] Kavin88 opened a new issue #3913: [SUPPORT] Hudi deltastreamer deployment Model

Kavin88 opened a new issue #3913:
URL: https://github.com/apache/hudi/issues/3913


   what is the deployment model to be followed for Hudi deltastreamer ? Could not find any references other than direct spark-submit. Can it be called through .py python file as spark submit ? property and config file would be different for each table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #3913: [SUPPORT] Hudi deltastreamer deployment Model

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3913:
URL: https://github.com/apache/hudi/issues/3913#issuecomment-961834734






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan closed issue #3913: [SUPPORT] Hudi deltastreamer deployment Model

Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3913:
URL: https://github.com/apache/hudi/issues/3913


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on issue #3913: [SUPPORT] Hudi deltastreamer deployment Model

Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3913:
URL: https://github.com/apache/hudi/issues/3913#issuecomment-966151436


   @Kavin88 delta streamer is a spark application so you need to do spark-submit to get it running on a cluster. It's not meant to be called thru Python code as it is a bundled application. If you want to do pyspark with Hudi, then you should follow guide in https://hudi.apache.org/docs/quick-start-guide which is about writing your own pyspark application. Again you'll need to spark-submit that too. As @nsivabalan mentioned you need one set of config per table. Hope these helps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #3913: [SUPPORT] Hudi deltastreamer deployment Model

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3913:
URL: https://github.com/apache/hudi/issues/3913#issuecomment-961834734


   My understanding is that, you run pyspark job in spark-submt?
   https://www.tutorialkart.com/apache-spark/submit-spark-application-python-example/
   or am I getting your requirement wrong.
   If you have multiple tables, yeah, each one has to be different. But hudi does provide a [MultiTableDetlastreamer](https://hudi.apache.org/blog/2020/08/22/ingest-multiple-tables-using-hudi).  Please check it out if it will meet your needs. 
   
   If you can go over your requirements, we can chat through how to go about it. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] harishraju-govindaraju commented on issue #3913: [SUPPORT] Hudi deltastreamer deployment Model

Posted by GitBox <gi...@apache.org>.
harishraju-govindaraju commented on issue #3913:
URL: https://github.com/apache/hudi/issues/3913#issuecomment-1013983252


   Can we use aws glue job to run deltastreamer  instead of spark submit? We dont use EMR. However, we wanted to see if aws glue can run deltastreamer ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org