You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/24 07:56:17 UTC
[GitHub] [hudi] eric9204 opened a new issue, #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
eric9204 opened a new issue, #5671:
URL: https://github.com/apache/hudi/issues/5671
**Describe the problem you faced**
Flink sql consume data from kafka and write to hudi, the sql statement enable metadata table, Archive can't be triggered.so ,there are 7044 files in the `.hoodie` directory
**To Reproduce**
Steps to reproduce the behavior:
1.Set the following parameters：
**Expected behavior**
These file should be archived which in the `.hoodie` directory
**Environment Description**
* Hudi version :
hudi-0.11.0
* Spark version :
--
* Hive version :
--
* Hadoop version :
3.1.0
* Storage (HDFS/S3/GCS..) :
hdfs
* Running on Docker? (yes/no) :
no
**Additional context**
flink-1.13.5
**Stacktrace**
```Add the stacktrace of the error.```
**There are 7044 files in the `.hoodie` directory**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1229659603
@nsivabalan When I use 0.11.1 for the same test, this problem does not appear again.
Thank you for your help, I will close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1149822369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1149280366
if there are any pending or failed commits in data table timeline, metadata table archival will be stalled. that is what Danny is suggesting to check the data table timeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 closed issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 closed issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
URL: https://github.com/apache/hudi/issues/5671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1138057358
> Another detail is that we tweaked the archive strategy when metadata table is enabled for 0.11, #4821, maybe it is relevant with your error.
After the program runs for eighteen hours a day，Metadata does not seem to be archived again.there are 7235 files in the `.hoodie `directory right now .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 closed issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 closed issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
URL: https://github.com/apache/hudi/issues/5671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1149869429
@nsivabalan
>if there are any pending or failed commits in data table timeline, metadata table archival will be stalled
Is that reasonable ? Is it possible that a rolled back instant still on the timeline ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1135694651
Actually,the program runs four days a day，all of compaction were successful and all log files were successfully compacted before the program stops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1136737349
Maybe You are right , i restart the program only change the table type from mor to cow yesterday，the meta file under `.hoodie` directory was archived .The program has been running for 22 hours and 47 minutes，there are 267 files in the `.hoodie` directory right now .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1136742468
I can't connect to the HDFS ,so I could't test it use HoodieTimelineArchiver. this should be related to compaction.I'm thinking about it,#4812.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1136716947
Another detail is that we tweaked the archive strategy when metadata table is enabled for 0.11, https://github.com/apache/hudi/pull/4821, maybe it is relevant with your error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1229359671
we made some fixes around failed roll backs in 0.11 or 0.11.1. essentially if a rollback is failed mid-way, next time if we restart the pipeline, hudi was creating a new rollback instant to rollback the partially failed commit. We also fixed an issue where in, even if no partialy failed commits are found, but if pending rollbacks are found, those will be attempted again.
Can you @eric9204 give 0.12 or 0.11.1 a try and let us know if you see the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1149398637
> if there are any pending or failed commits in data table timeline, metadata table archival will be stalled. that is what Danny is suggesting to check the data table timeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1149405745
@nsivabalan @danny0405 It's true that there are some anomalies.The program started at 10: 27，The first instant file was modified at 11: 01，It should be determined that the archive is triggered until a failed `deltacommit` transaction occurs.So after that, the archive was stopped forever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1149737683
But why the delta commit does not be rolled back ? When the job restarts, any failed writes expect to be rolled back first before we write new commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1149802860
> But why the delta commit does not be rolled back ? When the job restarts, any failed writes expect to be rolled back first before we write new commits.
Before this job restart ,the hudi data directory was deleted . The job started at 10: 27, that means , from 10: 27 to 11: 01 , the metadata archive was triggered normally until the failed `20220519110112118.deltacommit` occur.
I don't know why the delta commit does not be rolled back . But i know this , Maybe the delta commit failed due to which known reasons,that hudi table creates a new partition at a new hour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1135650145
Did you schedule the compaction but does not compact it actually ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #5671: [SUPPORT] Archive can't be triggered,when parameter of the metadata table was use in the program
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #5671:
URL: https://github.com/apache/hudi/issues/5671#issuecomment-1136083838
If you can connect to the HDFS on your local env, you can use the HoodieTimelineArchiver to test it locally and you can quickly find the reason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org