You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2024/03/20 21:58:57 UTC
[I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
soumilshah1995 opened a new issue, #10896:
URL: https://github.com/apache/hudi/issues/10896
Hello,
I recently created a video tutorial on backfilling with Hudi, and during my experimentation, I encountered a challenge regarding the insert_overwrite method while using the Delta Streamer. I've been primarily working with PySpark for these tasks.
Video Link: [Video on Backfilling with Hudi](https://www.example.com/)
Code Base: [GitHub Repository](https://github.com/soumilshah1995?tab=repositories)
In my workflow, I intended to perform an insert_overwrite on an entire partition. However, when attempting to execute the insert_overwrite job, I encountered an error indicating that the method was not found.
I understand that I have successfully executed similar tasks using PySpark. My question is: does Delta Streamer support the insert_overwrite operation? If it does not currently support this operation, I would like to request adding this feature to enhance the functionality of Delta Streamer.
I believe that supporting insert_overwrite would greatly benefit users who rely on Delta Streamer for data backfilling and other data management tasks.
Thank you for your attention to this matter. I look forward to hearing from the community regarding the feasibility of adding this feature.
Slack Thread https://apache-hudi.slack.com/archives/C4D716NPQ/p1710788154289249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2016802840
I tried --op insert_overwrite I guess its not natively supported on docs as well it says support for insert | bulk_insert| upsert
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2032790976
ill send email [dev@hudi.apache.org](mailto:dev@hudi.apache.org)
ill close this thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2032779435
Roger that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2018506237
The only use case I can think of is for table maintenance activity, may be run it with mode run_once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2029348950
As, Sudha suggested, can you also send a mail to dev list thread and point the conversation here. Good to hear thought on this from others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2032780049
you want me to close this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2015740027
its here https://github.com/soumilshah1995/DeltaHudiTransformations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2020198016
adding insert_overwrite can also help to build gold zone I can read data from multiple hudi tables and insert_overwrite into gold aggregated tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2020148955
Could you confirm if DeltaStreamer supports "insert_overwrite"? If not, I'm interested in understanding why. The reason for this inquiry is that in scenarios where I'm utilizing SQLSource and need to rectify an entire partition from which I'm reading, I would prefer to use "insert_overwrite" as it facilitates index lookup, akin to what "upsert" would accomplish. Ideally, having support for "insert_overwrite" in DeltaStreamer would prove immensely beneficial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2018504236
@soumilshah1995 I am just thinking does insert_overwrite is even make sense for streaming workloads. Do you have any such use case?
For sources like Kafka, for sure doesn't makes sense at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2015088337
Thanks @soumilshah1995 for the details. can you share the full Hudi Streamer command/code which you were using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10896:
URL: https://github.com/apache/hudi/issues/10896#issuecomment-2029348293
@soumilshah1995 This makes sense. Create a JIRA also to track - https://issues.apache.org/jira/browse/HUDI-7558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 closed issue #10896: [SUPPORT] Requesting Support for insert_overwrite in Delta Streamer
URL: https://github.com/apache/hudi/issues/10896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org