You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ZhendongBai (via GitHub)" <gi...@apache.org> on 2024/11/18 08:16:07 UTC
[I] Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]
ZhendongBai opened a new issue, #11575:
URL: https://github.com/apache/iceberg/issues/11575
### Apache Iceberg version
1.1.0
### Query engine
Flink
### Please describe the bug 🐞
when calling checkCommitStatus method, unexpected errors maybe occur, such as OutOfMemoryError, during the checkCommitStatus method execution. the code show as below:
```
commitStatus =
checkCommitStatus(
viewName,
newMetadataLocation,
metadata.properties(),
() -> checkCurrentMetadataLocation(newMetadataLocation));
```
During the execution of the `org.apache.iceberg.hive.HiveViewOperations.checkCurrentMetadataLocation` method's refresh operation to download and update the current table metadata, memory consumption occurs, potentially leading to an `OutOfMemoryError`.
It is important to note that `Tasks` may not handle `Error` exceptions, instead throwing them directly.
Finally, in the `finally` block, the cleanupMetadataAndUnlock function may delete the table snapshot metadata file, even if it has been recently committed.
### Willingness to contribute
- [X] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]
Posted by "ZhendongBai (via GitHub)" <gi...@apache.org>.
ZhendongBai commented on issue #11575:
URL: https://github.com/apache/iceberg/issues/11575#issuecomment-2482311824
> @ZhendongBai: Could you please check if this issue still exists in the current connector version?
@pvary Yes, Despite multiple iterations, the core logic of `checkCommitStatus`, the logic of calling refresh to get the latest table metadata, has remained unchanged, the code in problem description section is main brach code, the the 1.1.0 version code link is https://github.com/apache/iceberg/blob/ede085d0f7529f24acd0c81dd0a43f7bb969b763/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java#L305, and the refresh method is called at https://github.com/apache/iceberg/blob/ede085d0f7529f24acd0c81dd0a43f7bb969b763/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java#L336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #11575:
URL: https://github.com/apache/iceberg/issues/11575#issuecomment-2482281011
@ZhendongBai: Could you please check if this issue still exists in the current connector version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer closed issue #11575: Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError
URL: https://github.com/apache/iceberg/issues/11575
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Incorrect Deletion of Snapshot Metadata Due to OutOfMemoryError [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #11575:
URL: https://github.com/apache/iceberg/issues/11575#issuecomment-2491927652
Fixed in #11576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org