You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "AbhijeetSachdev1 (via GitHub)" <gi...@apache.org> on 2023/03/26 02:47:03 UTC
[GitHub] [hudi] AbhijeetSachdev1 opened a new issue, #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
AbhijeetSachdev1 opened a new issue, #8292:
URL: https://github.com/apache/hudi/issues/8292
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Hi Team, Could you please help me in understanding the uses of hudi-archived files.
What is the need of archiving commits ? Why will we ever need them ?
Can't we just delete old commit files, instead of archiving them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1512620727
In most of the cases, there should be no bad impact, if your incremenal source is fast enough or the hive meta sync is not in latency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1542018914
@AbhijeetSachdev1 Do you need any other support for this ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1580238095
Closing this out, @AbhijeetSachdev1 Feel free to reopen in case you need any support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan closed issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
URL: https://github.com/apache/hudi/issues/8292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1485062105
For most of the cases, we can remove it manually, the archived timeline can be used in some corner cases like Hive meta sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1486229963
The hive meta sync is being doing incrementally by checking the new instants since last meta sync, for each instant, if it checks that the instant had been archived, the archived metadata would be deserialized from the archiving log file, for your exception, it seems there are some missing variables for the metadata which result in some compatibility issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] AbhijeetSachdev1 commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "AbhijeetSachdev1 (via GitHub)" <gi...@apache.org>.
AbhijeetSachdev1 commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1512222922
@danny0405 Thanks for response, I am still not clear.
Could you help me understand, what worst can happen if we DELETE archived files ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] AbhijeetSachdev1 commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "AbhijeetSachdev1 (via GitHub)" <gi...@apache.org>.
AbhijeetSachdev1 commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1517120752
@danny0405
Thanks for the response !
Yes our hive meta sync is not in running behind.
And our incremental are also scheduled with the frequency of 1 hour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] AbhijeetSachdev1 commented on issue #8292: [SUPPORT] What all are the uses of hudi-archived files and consequences of deleting them manually.
Posted by "AbhijeetSachdev1 (via GitHub)" <gi...@apache.org>.
AbhijeetSachdev1 commented on issue #8292:
URL: https://github.com/apache/hudi/issues/8292#issuecomment-1485451002
Thanks danny fir the response, could you please elaborate how is it being used in hive_sync ?
Why I am asking is because our hive_sync is failing while reading these archived files.
Our related issue: https://github.com/apache/hudi/issues/8210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org