You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ytimetravel (via GitHub)" <gi...@apache.org> on 2024/04/26 11:42:20 UTC
[I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Ytimetravel opened a new issue, #11104:
URL: https://github.com/apache/hudi/issues/11104
Dear community,
I have discovered an issue when using Hudi.If multiple archive tasks run on COW table and set "hoodie.archive.automatic=false", it may cause data problems. If set hoodie.archive.automatic=true, then this issue will not occur.
And then I find that if the archive operation try lock failed, it will always release lock(if exist).
<img width="1378" alt="image" src="https://github.com/apache/hudi/assets/112808605/fa2b7017-c11e-42d6-bf40-a20bf59f7652">
<img width="1370" alt="image" src="https://github.com/apache/hudi/assets/112808605/7b9d18a1-e9dc-4e03-afb6-d084ae000f43">
<img width="1567" alt="image" src="https://github.com/apache/hudi/assets/112808605/fe49cb8a-46bd-493e-9c93-fdba254735fc">
I suspect that this lock release operation may have affected other normal operations.
Perhaps the problem could be avoided by doing it this way?
<img width="894" alt="image" src="https://github.com/apache/hudi/assets/112808605/cc8c0e9a-8b29-4cf9-b928-bdd79f8a7a4c">
Looking forward to your valuable suggestions.
Hudi version :0.14.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "Ytimetravel (via GitHub)" <gi...@apache.org>.
Ytimetravel commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081966490
<img width="1562" alt="image" src="https://github.com/apache/hudi/assets/112808605/16348f67-4899-45bc-acb0-ca2ba3e51912">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2440531511
@ad1happy2go The issue is fixed in [pr#11160](https://github.com/apache/hudi/pull/11160) and merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081770424
@danny0405 @Ytimetravel The behavior of method `TransactionManager#endTransaction` itself is correct, it would check whether the current lock is hold by itself before it unlock. However, there is a bug in `HoodieTimelineArchiver` because archiving itself is not a transaction and does not correspond to any instant in timeline. When an exception occurs, it might mistakenly deletes locks held by others.
<img width="1144" alt="image" src="https://github.com/apache/hudi/assets/1525333/b4fcafe3-baad-4034-9877-b8c95a5c70e9">
@Ytimetravel Would you like to fix this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "Ytimetravel (via GitHub)" <gi...@apache.org>.
Ytimetravel commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081840729
@beyond1920 Yes, I am very willing to fix this issue.
<img width="1122" alt="image" src="https://github.com/apache/hudi/assets/112808605/91d14ef0-ed7b-4b49-aa5b-ae27ccebc79a">
This fix has already been tested and verified~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2080266345
archive itself holds an transaction lock, so we need to release it in any case, what is the wrong case you have ecountered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081311704
> otherwise is there a chance of mistakenly releasing the lock of other operations and causing problems?
That's reasonable, we should ensure the lock been acquired in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081867583
What is the general reason that the trasanction start of archival is failing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "Ytimetravel (via GitHub)" <gi...@apache.org>.
Ytimetravel commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081302597
@danny0405 Sorry, I don't remember the details of the problem (I will confirm with my colleagues and provide the results later), but based on the logic here, wouldn't it be better to attempt to release the lock only when acquiring it, otherwise is there a chance of mistakenly releasing the lock of other operations and causing problems?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "Ytimetravel (via GitHub)" <gi...@apache.org>.
Ytimetravel commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081963052
@danny0405 Failed to acquire lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2419828235
@Ytimetravel Do we have a fix or JIRA on this?
cc @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081970456
Okay, it would be great if you can fire a fix for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "Ytimetravel (via GitHub)" <gi...@apache.org>.
Ytimetravel commented on issue #11104:
URL: https://github.com/apache/hudi/issues/11104#issuecomment-2081966052
![Uploading image.png…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Archive operation always release lock on the timeline when try lock failed [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 closed issue #11104: [SUPPORT] Archive operation always release lock on the timeline when try lock failed
URL: https://github.com/apache/hudi/issues/11104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org