You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/03/26 07:28:04 UTC
[GitHub] [iceberg] openinx opened a new issue #2384: Flaky unit test: TestFlinkTableSink#testHashDistributeMode
openinx opened a new issue #2384:
URL: https://github.com/apache/iceberg/issues/2384
I encountered several failures when running the TestFlinkTableSink#testHashDistributeMode, I think it worth to take some time to dig what's reason that causing this UT to be flaky.
```java
org.apache.iceberg.flink.TestFlinkTableSink > testHashDistributeMode[catalogName=testhadoop_basenamespace, baseNamespace=l0.l1, format=AVRO, isStreaming=true] FAILED
java.lang.AssertionError: There should be only 1 data file in partition 'aaa' expected:<1> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.apache.iceberg.flink.TestFlinkTableSink.testHashDistributeMode(TestFlinkTableSink.java:280)
org.apache.flink.table.api.ValidationException: Could not execute DROP DATABASE IF EXISTS testhadoop_basenamespace.db RESTRICT
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:989)
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666)
at org.apache.iceberg.flink.FlinkTestBase.exec(FlinkTestBase.java:92)
at org.apache.iceberg.flink.FlinkTestBase.exec(FlinkTestBase.java:96)
at org.apache.iceberg.flink.FlinkTestBase.sql(FlinkTestBase.java:100)
at org.apache.iceberg.flink.TestFlinkTableSink.clean(TestFlinkTableSink.java:132)
Caused by:
org.apache.flink.table.catalog.exceptions.DatabaseNotEmptyException: Database db in catalog testhadoop_basenamespace is not empty.
at org.apache.iceberg.flink.FlinkCatalog.dropDatabase(FlinkCatalog.java:242)
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:983)
... 5 more
Caused by:
org.apache.iceberg.exceptions.NamespaceNotEmptyException: Namespace l0.l1.db is not empty.
at org.apache.iceberg.hadoop.HadoopCatalog.dropNamespace(HadoopCatalog.java:344)
at org.apache.iceberg.flink.FlinkCatalog.dropDatabase(FlinkCatalog.java:233)
... 6 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #2384: Flaky unit test: TestFlinkTableSink#testHashDistributeMode
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #2384:
URL: https://github.com/apache/iceberg/issues/2384#issuecomment-935843478
This failed again with the same issue in https://github.com/apache/iceberg/pull/3230/checks?check_run_id=3812504383
```
org.apache.iceberg.flink.TestFlinkTableSink > testHashDistributeMode[catalogName=testhadoop, baseNamespace=, format=PARQUET, isStreaming=true] FAILED
java.lang.AssertionError: There should be 1 data file in partition 'aaa' expected:<1> but was:<2>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.apache.iceberg.flink.TestFlinkTableSink.testHashDistributeMode(TestFlinkTableSink.java:281)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on issue #2384: Flaky unit test: TestFlinkTableSink#testHashDistributeMode
Posted by GitBox <gi...@apache.org>.
openinx commented on issue #2384:
URL: https://github.com/apache/iceberg/issues/2384#issuecomment-808007746
Checked the code path ( https://github.com/apache/iceberg/blob/master/flink/src/test/java/org/apache/iceberg/flink/TestFlinkTableSink.java#L290), If the flink job did a failover, then it will produce some data files under the partition directories, which has not been committed to apache iceberg table, so actually those files should not be visible to iceberg users. I think we'd better to change the way to get the partition files by `Table.newScan().planFiles()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #2384: Flaky unit test: TestFlinkTableSink#testHashDistributeMode
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #2384:
URL: https://github.com/apache/iceberg/issues/2384#issuecomment-945773969
already covered by #2575, so closinig this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on issue #2384: Flaky unit test: TestFlinkTableSink#testHashDistributeMode
Posted by GitBox <gi...@apache.org>.
openinx commented on issue #2384:
URL: https://github.com/apache/iceberg/issues/2384#issuecomment-808733308
Close this via https://github.com/apache/iceberg/pull/2385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx closed issue #2384: Flaky unit test: TestFlinkTableSink#testHashDistributeMode
Posted by GitBox <gi...@apache.org>.
openinx closed issue #2384:
URL: https://github.com/apache/iceberg/issues/2384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra closed issue #2384: Flaky unit test: TestFlinkTableSink#testHashDistributeMode
Posted by GitBox <gi...@apache.org>.
nastra closed issue #2384:
URL: https://github.com/apache/iceberg/issues/2384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org