You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/04/12 15:12:39 UTC

[GitHub] [hudi] rubenssoto opened a new issue #2810: [SUPPORT] How Hudi with Spark Streaming works?

rubenssoto opened a new issue #2810:
URL: https://github.com/apache/hudi/issues/2810


   Hello Guys,
   
   How hudi with spark structured streaming works?
   
   I'm trying to use no error, but no files are written.
   
   ---Function to read Hudi Table
   def read_hudi_stream(spark_session, read_folder_path, max_files_per_trigger):
       spark = spark_session
       df = (
           spark.readStream.format("hudi")
           .load(read_folder_path)
       )
       return df
   
   
   --Function to write the stream
   def write_parquet_stream_trigger_once(
       spark_data_frame, checkpoint_location_folder, foreach_batch_function
   ):
       df_write_query = (
           spark_data_frame.writeStream.trigger(once=True)
           .foreachBatch(foreach_batch_function)
           .outputMode("append")
           .option("checkpointLocation", checkpoint_location_folder)
           .start()
       )
       df_write_query.awaitTermination()
   
   
   Im using a foreach batch function, and my hudi write with hudi options are inside my foreach batch.
   
   It is a new feature, so I dont know how it works.
   
   @pengzhiwei2018 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on issue #2810: [SUPPORT] How Hudi with Spark Streaming works?

Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2810:
URL: https://github.com/apache/hudi/issues/2810#issuecomment-826568664


   @rubenssoto Does @pengzhiwei2018 recommendation solve your problem ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pengzhiwei2018 commented on issue #2810: [SUPPORT] How Hudi with Spark Streaming works?

Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on issue #2810:
URL: https://github.com/apache/hudi/issues/2810#issuecomment-818387671


   Hi @rubenssoto , first you should write some data to the hudi table, then you can consume the data from that hudi table.
   You can refer to  the test case in [TestStreamingSource](https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSource.scala).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash closed issue #2810: [SUPPORT] How Hudi with Spark Streaming works?

Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2810:
URL: https://github.com/apache/hudi/issues/2810


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on issue #2810: [SUPPORT] How Hudi with Spark Streaming works?

Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2810:
URL: https://github.com/apache/hudi/issues/2810#issuecomment-854403589


   @rubenssoto Closing this ticket due to inactivity - feel free to re-open if you need further assistance. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash closed issue #2810: [SUPPORT] How Hudi with Spark Streaming works?

Posted by GitBox <gi...@apache.org>.
n3nash closed issue #2810:
URL: https://github.com/apache/hudi/issues/2810


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on issue #2810: [SUPPORT] How Hudi with Spark Streaming works?

Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2810:
URL: https://github.com/apache/hudi/issues/2810#issuecomment-854403589


   @rubenssoto Closing this ticket due to inactivity - feel free to re-open if you need further assistance. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org