You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lurnagao-dahua (via GitHub)" <gi...@apache.org> on 2024/03/30 02:33:07 UTC
[PR] Hive: Check e.getMessage() is not null [iceberg]
lurnagao-dahua opened a new pull request, #10069:
URL: https://github.com/apache/iceberg/pull/10069
Hi, I get an Metadata not found in metadata location for table error when trying to query the data generated by flink
flink: 1.16.2
iceberg: 1.4.3
The reason is that in some cases, the e.getMessage() return null and it will throw NullPointerException, then skip checkCommitStatus, it may be delete metadataLocation, actually, metadata commit succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547080134
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
throw e;
} catch (Throwable e) {
- if (e.getMessage()
- .contains(
+ if (e.getMessage() != null
Review Comment:
It'd better to create a util method for this pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547262874
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
throw e;
} catch (Throwable e) {
- if (e.getMessage()
- .contains(
+ if (e.getMessage() != null
Review Comment:
If we had the method in the first place, I don't think we would miss the null check here. With this method, we can ensure the null check will be applied in any other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2031250719
@lurnagao-dahua please check styles.
> The reason is that in some cases, the e.getMessage() return null and it will throw NullPointerException, then skip checkCommitStatus, it may be delete metadataLocation, actually, metadata commit succeed.
Is it possible to add a UT for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "lurnagao-dahua (via GitHub)" <gi...@apache.org>.
lurnagao-dahua commented on PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2032193393
> @lurnagao-dahua please check styles.
>
> > The reason is that in some cases, the e.getMessage() return null and it will throw NullPointerException, then skip checkCommitStatus, it may be delete metadataLocation, actually, metadata commit succeed.
>
> Is it possible to add a UT for this case?
Hi, I found the original pr similar to this one [701](https://github.com/apache/iceberg/pull/701), then I will try to write an UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "lurnagao-dahua (via GitHub)" <gi...@apache.org>.
lurnagao-dahua closed pull request #10069: Hive: Fix metadata file not found
URL: https://github.com/apache/iceberg/pull/10069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "lurnagao-dahua (via GitHub)" <gi...@apache.org>.
lurnagao-dahua commented on PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2033989359
> @lurnagao-dahua: If you check https://github.com/apache/iceberg/blob/3caa3a28d07a2d08b9a0e4196634126f1e016d6a/hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCommits.java, you can find plenty of examples for commit errors. Maybe if we could do something similar, like throwing an exception without a message. It would be nice to have a test.
>
> OTOH, if the test is more than 50 lines, it would cost us more in the upkeep of the test in the long run, than what we gain with testing a null check. In this case I would skip addig the extra code, following the example of #701.
Hi,Thank you for your suggestion
I added a unit test.It will throw runtimeException without message and result in throw CommitStateUnknownException,the meesage will be null + COMMON_INFO:
```
public CommitStateUnknownException(Throwable cause) {
super(cause.getMessage() + "\n" + COMMON_INFO, cause);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2033649191
@lurnagao-dahua: If you check https://github.com/apache/iceberg/blob/3caa3a28d07a2d08b9a0e4196634126f1e016d6a/hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCommits.java, you can find plenty of examples for commit errors. Maybe if we could do something similar, like throwing an exception without a message. It would be nice to have a test.
OTOH, if the test is more than 50 lines, it would cost us more in the upkeep of the test in the long run, than what we gain with testing a null check. In this case I would skip addig the extra code, following the example of #701.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "lurnagao-dahua (via GitHub)" <gi...@apache.org>.
lurnagao-dahua commented on PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2033424821
Thank you for your response!
In my case flink streaming write to iceberg
1.Hive metastore has been full GC continuously so it will throw SocketTimeoutException: Read timed out(`hive.metastore.client.socket.timeout` default 600s)
2.hiveTableOperations commit thread call `Thread.sleep(retryDelaySeconds * 1000)` to retry
3.The Flink checkpoint timeout time is less than 600s and Interrupt it, then throw InterruptedException and not message
I have been thinking for a while and I have some doubts about this UT. Can you give me some advice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547262874
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
throw e;
} catch (Throwable e) {
- if (e.getMessage()
- .contains(
+ if (e.getMessage() != null
Review Comment:
If we had the method in the first place, I don't think we would miss the null check here. With this method, we can ensure the null check will be applied in any other places. Just my two cents, which should not block merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "lurnagao-dahua (via GitHub)" <gi...@apache.org>.
lurnagao-dahua commented on code in PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#discussion_r1549380381
##########
hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCommits.java:
##########
@@ -397,6 +397,30 @@ public void testLockExceptionUnknownSuccessCommit() throws TException, Interrupt
.isTrue();
}
+ @Test
+ public void testCommitExceptionWithoutMessage() throws TException, InterruptedException {
Review Comment:
> @Test
> public void testCommitExceptionWithoutMessage() throws TException, InterruptedException {
> Table table = catalog.loadTable(TABLE_IDENTIFIER);
> HiveTableOperations ops = (HiveTableOperations) ((HasTableOperations) table).operations();
>
> TableMetadata metadataV1 = ops.current();
> table.updateSchema().addColumn("n", Types.IntegerType.get()).commit();
>
> ops.refresh();
>
> HiveTableOperations spyOps = spy(ops);
> doThrow(new RuntimeException()).when(spyOps).persistTable(any(), anyBoolean(), any());
>
> assertThatThrownBy(() -> spyOps.commit(ops.current(), metadataV1))
> .isInstanceOf(CommitStateUnknownException.class)
> .hasMessageStartingWith("null\nCannot determine whether the commit was successful or not");
> }
Thank you very much, I have simplified unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547189727
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
throw e;
} catch (Throwable e) {
- if (e.getMessage()
- .contains(
+ if (e.getMessage() != null
Review Comment:
Where would we use this new method?
If we would use it only in this class, and only in 2 places, then I think it is better not to "hide" what we do behind a method somewhere else in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#discussion_r1547262874
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -263,8 +263,8 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
throw e;
} catch (Throwable e) {
- if (e.getMessage()
- .contains(
+ if (e.getMessage() != null
Review Comment:
If we had the method in the first place, I don't think we would miss the null check here. With this method, we can ensure the null check will be applied in any other places. Just my two cents, which should not block this PR from being merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #10069:
URL: https://github.com/apache/iceberg/pull/10069
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#discussion_r1549340231
##########
hive-metastore/src/test/java/org/apache/iceberg/hive/TestHiveCommits.java:
##########
@@ -397,6 +397,30 @@ public void testLockExceptionUnknownSuccessCommit() throws TException, Interrupt
.isTrue();
}
+ @Test
+ public void testCommitExceptionWithoutMessage() throws TException, InterruptedException {
Review Comment:
can be simplified to
```
@Test
public void testCommitExceptionWithoutMessage() throws TException, InterruptedException {
Table table = catalog.loadTable(TABLE_IDENTIFIER);
HiveTableOperations ops = (HiveTableOperations) ((HasTableOperations) table).operations();
TableMetadata metadataV1 = ops.current();
table.updateSchema().addColumn("n", Types.IntegerType.get()).commit();
ops.refresh();
HiveTableOperations spyOps = spy(ops);
doThrow(new RuntimeException()).when(spyOps).persistTable(any(), anyBoolean(), any());
assertThatThrownBy(() -> spyOps.commit(ops.current(), metadataV1))
.isInstanceOf(CommitStateUnknownException.class)
.hasMessageStartingWith("null\nCannot determine whether the commit was successful or not");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Hive: Fix metadata file not found [iceberg]
Posted by "lurnagao-dahua (via GitHub)" <gi...@apache.org>.
lurnagao-dahua commented on PR #10069:
URL: https://github.com/apache/iceberg/pull/10069#issuecomment-2032181736
> @lurnagao-dahua请检查样式。
>
> > 原因是在某些情况下,e.getMessage()返回null,会抛出NullPointerException,然后跳过checkCommitStatus,可能是删除metadataLocation,实际上元数据提交成功。
>
> 是否可以为这种情况添加 UT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org