You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/29 10:05:12 UTC
[GitHub] [hudi] txl2017 opened a new issue, #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
txl2017 opened a new issue, #6527:
URL: https://github.com/apache/hudi/issues/6527
i use cow tableļ¼and 'clean-policy' = 'KEEP_LATEST_FILE_VERSIONS'
with table's parition increased, the flink task became instability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6527:
URL: https://github.com/apache/hudi/issues/6527#issuecomment-1263429401
Closing it as we have a fix under review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] txl2017 commented on issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
txl2017 commented on issue #6527:
URL: https://github.com/apache/hudi/issues/6527#issuecomment-1232895212
the same case for compact too
2022-08-31 12:35:33,278 INFO org.apache.hudi.table.action.compact.HoodieCompactor [] - Compacting s3:// with commit 20220831123532928
2022-08-31 12:35:34,018 INFO org.apache.hudi.table.action.compact.HoodieCompactor [] - Compaction looking for files to compact in [19700101, 20220511, 20220512, 20220513, 20220514, 20220515, 20220516, 20220517, 20220518, 20220519, 20220520, 20220521, 20220522, 20220523, 20220524, 20220525, 20220526, 20220527, 20220528, 20220529, 20220530, 20220531, 20220601, 20220602, 20220603, 20220604, 20220605, 20220606, 20220607, 20220608, 20220609, 20220610, 20220611, 20220612, 20220613, 20220614, 20220615, 20220616, 20220617, 20220618, 20220619, 20220620, 20220621, 20220622, 20220623, 20220624, 20220625, 20220626, 20220627, 20220628, 20220629, 20220630, 20220701, 20220702, 20220703, 20220704, 20220705, 20220706, 20220707, 20220708, 20220709, 20220710, 20220711, 20220712, 20220713, 20220714, 20220715, 20220716, 20220817, 20220818, 20220819, 20220820, 20220821, 20220822, 20220823, 20220824, 20220825, 20220826, 20220827, 20220828, 20220829, 20220830, 20220831] partitions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6527:
URL: https://github.com/apache/hudi/issues/6527#issuecomment-1230649419
yes. it might get costly w/ the scale or if you plan to retain lot of file versions. I can file a jira to enhance this. but might need some intelligent design around this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6527:
URL: https://github.com/apache/hudi/issues/6527#issuecomment-1233256049
oh, just now looking at your solution. it might have some corner cases. For eg, if you choose cleaner policy to be LATEST_FILE_VERSION, and set commits retained to 10 (this config is supposed to work only for LATEST_COMMITS policy).
after 15 commits, we will consider only the last 10 commits to do clean up, what incase some file group never got updated from C5 to C15. So, those file groups should also be considered right when we are looking to schedule cleaning based on FILE_VERSIONS.
Open to jam more on this and find a good enhancement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] txl2017 commented on issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
txl2017 commented on issue #6527:
URL: https://github.com/apache/hudi/issues/6527#issuecomment-1231013851
> yes. it might get costly w/ the scale or if you plan to retain lot of file versions. I can file a jira to enhance this. but might need some intelligent design around this.
ok thanks! because this was in our production environment and is really affect our flink task. so we solved it through a simple treatment. and we are looking forward this intelligent design in next release. then we will upgrade.
here is our simple treatment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on issue #6527:
URL: https://github.com/apache/hudi/issues/6527#issuecomment-1236106235
In my opinion, read full partition in KEEP_LATEST_FILE_VERSIONS strategy when clean files **is required anyway,** since the commit instant will be cleaned up, you never know how many slices are in the fileGroup. All we can do is speed up listing partitions or files with metadata as far I am concerned
However, for KEEP_LATEST_COMMITS strategy, you only need to read the partitions that exist in the latest replate commits , rhat's what hudi is doing now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6527:
URL: https://github.com/apache/hudi/issues/6527#issuecomment-1233252514
I have made some perf fixes to the full cleaning when metadata is enabled.
https://issues.apache.org/jira/browse/HUDI-4749
Also, I have proposed a hybrid cleaning policy to enhance cleaning by LATEST_FILE_VERSIONS : https://issues.apache.org/jira/browse/HUDI-4750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
Posted by GitBox <gi...@apache.org>.
codope closed issue #6527: read full partition in KEEP_LATEST_FILE_VERSIONS Strategy when clean files
URL: https://github.com/apache/hudi/issues/6527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org