You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "njalan (via GitHub)" <gi...@apache.org> on 2023/10/01 14:49:48 UTC
[GitHub] [hudi] njalan opened a new issue, #9812: commits_.archive is not move to archived folder
njalan opened a new issue, #9812:
URL: https://github.com/apache/hudi/issues/9812
There are lots of .commits_.archive files under .hoodie directory but there is no files present under .hoodie/archived.
Environment Description
**Hudi version : 0.9.0
Spark version : 3.0.1
Hive version : 3
Hadoop version : 3.2.2
Storage (HDFS/S3/GCS..) : s3
Running on Docker? (yes/no) : no**
By the way these tables are upgraded from hudi 0.7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9812: commits_.archive is not move to archived folder
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1742328062
Did you ever configure the archive folder manually ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan closed issue #9812: commits_.archive is not move to archived folder
URL: https://github.com/apache/hudi/issues/9812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1744050515
@danny0405 It is weird that he latest file is generated two months ago and I didn't see any new archive files recently. My question is that can I remove these archive file manually? How to make new archive commit under archived directory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1743191318
@danny0405 No I never configure the archive folder manually. Do I need to config it manually? What is the parameter to config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1744008525
Are these files generated by the upgrated table or old version table, you can check the file modification time to get a clue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1744988210
metadata is not enable on hudi 0.9. I listed the details for one table in ticket https://github.com/apache/hudi/issues/9751. There are like around 10% of my tables has archive commits files in archived folder. I am using default parameter. But I didn't face any data issues for these tables without new archiving commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "njalan (via GitHub)" <gi...@apache.org>.
njalan commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1752044255
@ad1happy2go After remove these archive files steaming job is getting faster. But storage cluster's cpu is not getting down and it is still high because the count of http to access s3 is not reduced. Is it possible to tune it? In my case there is only job to upsert one table why it requires hundreds of s3 access for one spark streaming micro batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1744122986
@njalan Yes, You can clean up old commit files which got created with 0.7.
The issue here I see that with 0.12, why archive is not happening? Do you see any archival happening from last 2 months after you did the upgrade? Ideally new archival should happen automatically. How many active commits you see in active timeline now? Can you share configuration to triage more why archival is not happening?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1744084623
Did your commit metadata files get archiving in normal progress? I think it is okay you remove these legacy archiving files directly, they are currently useless except for CLI statistics purposes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] commits_.archive is not move to archived folder [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9812:
URL: https://github.com/apache/hudi/issues/9812#issuecomment-1746106261
@njalan Looks like its more of a upgrade issue. I dont see many active commits in the hoodie file list you pasted in another ticket. Archival will kick in for more than 20-30 commits. Let us know if removal of archived commits from .hoodie directory helped in reducing S3 operations (Your other issue) also.
Please let us know if you dont see archival happening when you have more than 30 commits in the timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org