You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Limess (via GitHub)" <gi...@apache.org> on 2023/03/07 16:35:59 UTC
[GitHub] [hudi] Limess opened a new issue, #8114: [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation
Limess opened a new issue, #8114:
URL: https://github.com/apache/hudi/issues/8114
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
After running an insert using `insert_overwrite_table`, partitions which no longer exist in the new input data are not removed by the Hive Sync
This is on Hudi 0.12.1, but I'm fairly sure this issue still exists on 0.13.0 - this change: https://github.com/apache/hudi/pull/6662 fixes this behaviour for `delete_partition` operations, but doesn't add any handling for `insert_overwrite_table`. I'd be happy to be proven otherwise.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a new Hudi table using input data with two partitions
2. Insert into the table using `INSERT_OVERWRITE_TABLE` with input data containing 1/2 of the original partitions
3. Run HiveSyncTool
4. Check the Hive partitions. Both partitions still exist
**Expected behavior**
I'd expect the partition which was not inserted to be removed
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.3.1
* Hive version : AWS Glue
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Running on EMR 0.6.9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2081940660
cc @codope guess this should have been fixed? https://github.com/apache/hudi/pull/6662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2185495368
@zhaobangcai The full context is that the issue was fixed but in order to fix, the archived timeline was also being read. This caused too high sync latency. Hence, the fix was reverted. Generally, reading archived timeline is an anti-pattern in Hudi, and we are optimizing this by implemeting LSM timeline in 1.0.0. That said, I think we did fix the timeline loading in https://github.com/apache/hudi/commit/ab61f61df9686793406300c0018924a119b02855 which I believe is in 0.14. Can you please share a script/test case to reproduce the issue with all configs that you used in your env? I am going to reopen the issue based on your comment and debug further once you provide the script/test case. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "zhaobangcai (via GitHub)" <gi...@apache.org>.
zhaobangcai commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2081816664
Has this problem been solved? @Limess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8114: [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-1459680237
Thanks for the feedback, guess you are right, this should be supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "zhaobangcai (via GitHub)" <gi...@apache.org>.
zhaobangcai commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2185411734
@codope ：hello,this issue still exists in version 0.14, why was it closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "zhaobangcai (via GitHub)" <gi...@apache.org>.
zhaobangcai commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2185478617
@codope :As stated by the Issue, the problem is a necessary occurrence. The version we are currently using is 0.14.
@Limess :Have you not encountered this problem again? May I ask how was it avoided?Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #8114: [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation
URL: https://github.com/apache/hudi/issues/8114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "zhaobangcai (via GitHub)" <gi...@apache.org>.
zhaobangcai commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2081815485
Is there no further text on this question? Do you have any plans to fix it in which version? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2081997720
Yes this was fixed in 0.13.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi partitions not dropped by Hive sync after `insert_overwrite_table` operation [hudi]
Posted by "Limess (via GitHub)" <gi...@apache.org>.
Limess commented on issue #8114:
URL: https://github.com/apache/hudi/issues/8114#issuecomment-2185835901
> @codope :As stated by the Issue, the problem is a necessary occurrence. The version we are currently using is 0.14. @Limess :Have you not encountered this problem again? May I ask how was it avoided?Thanks!
We never pursued this and are still on 0.13.0 for now, so I can't verify either way, sorry!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org