You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/01/16 13:22:23 UTC
[GitHub] [iceberg] zhangjun0x01 opened a new issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
zhangjun0x01 opened a new issue #2098:
URL: https://github.com/apache/iceberg/issues/2098
I found that ,the iceberg-spark3-extensions module CI test often failed.
the exception like this
```
org.apache.iceberg.spark.extensions.TestCopyOnWriteDelete > testDeleteNonExistingRecords[catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, clients=1, parquet-enabled=false, cache-enabled=false}, format = avro, vectorized = false] FAILED
java.lang.RuntimeException: Failed to get table info from metastore default.table
Caused by:
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)
Caused by:
java.net.SocketException: Broken pipe (Write failed)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-762046265
Locally, when I increase the catalog config value for `clients` from 1 to 2 or higher, this has passed on the past 3 times I've tried it. This is obviously kind of a hack, and having a value for `clients` that is > 1 won't necessarily work for catalogs backed by derby. However, it would be helpful if other people can try upping these values and see if that helps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-762532916
> I started a thread titled as "test flakiness with SocketException of broken pipe in HiveMetaStoreClient" [1] in the dev mailing list two weeks ago. People have offered some insights there. Main suspicious is connection leak problem.
>
> [1] https://www.mail-archive.com/dev@iceberg.apache.org/msg01789.html
Cool thank you @stevenzwu. Connection leak would potentially explain why I’m much less able to reproduce when I increase the catalog’s `clients` config value. Also, knowing that it is (was?) happening in the flink tests helps a ton with guidance for how much time to put into investigating from the POV of a potential test inheritance pattern issue vs connection leak issue.
This is very helpful. I should be more active on the dev list, though I’m only able to contribute somewhat sporadically given my current work is not exactly iceberg related. I’ll follow up in that thread too with anything I find while investigating. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-761945486
+1 I also can reproduce in local with `./gradlew :iceberg-spark3-extensions:test`, but passed in IDEA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-762046265
Locally, when I increase the catalog config value for `clients` from 1 to 2 or higher, this has passed on the past 3 times I've tried it. This is obviously kind of a hack, and having a value for `clients` that is > 1 won't necessarily work for catalogs backed by derby. However, it would be helpful if other people can try upping these values and see if that helps.
I upped the value of clients from 1 in the following test parameters (in the immutable config map, using `"clients", "2"`).
https://github.com/apache/iceberg/blob/74bb38d810a0e046caa47e771804d4d14c06ff57/spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java#L57-L87
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-761978477
I've also been able to get CI to pass recently (as in today), but it's sporadic & non-deterministic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-762532916
> I started a thread titled as "test flakiness with SocketException of broken pipe in HiveMetaStoreClient" [1] in the dev mailing list two weeks ago. People have offered some insights there. Main suspicious is connection leak problem.
>
> [1] https://www.mail-archive.com/dev@iceberg.apache.org/msg01789.html
Cool thank you @stevenzwu. Connection leak would potentially explain why I’m much less able to reproduce when I increase the hive catalog’s `clients` value. Also, knowing that it is (was?) happening in the flink tests helps with determining how much time to put into investigating from the POV of a potential test inheritance pattern issue.
This is very helpful. I should be more active on the dev list, though I’m only able to contribute somewhat sporadically given my current work is not iceberg related. I’ll follow up in that thread too with anything I find while investigating. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-762046265
Locally, when I increase the catalog config value for `clients` from 1 to 2 or higher, this has passed on the past 3 times I've tried it. This is obviously kind of a hack, and having a value for `clients` that is > 1 won't necessarily work for catalogs backed by derby. However, it would be helpful if other people can try upping these values and see if that helps.
I upped the value of clients from 1 in the following test parameters (in the immutable config map, using `"clients", "2"`). I did not update it for the one value that is hard coded as 1.
https://github.com/apache/iceberg/blob/74bb38d810a0e046caa47e771804d4d14c06ff57/spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java#L57-L87
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] stevenzwu commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
stevenzwu commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-762425155
I started a thread titled as "test flakiness with SocketException of broken pipe in HiveMetaStoreClient" [1] in the dev mailing list two weeks ago. People have offered some insights there. Main suspicious is connection leak problem.
[1] https://www.mail-archive.com/dev@iceberg.apache.org/msg01789.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-761941980
Additionally, another option seems to be that we could use `TestSparkTableUtilWithInMemoryCatalog`, or essentially update the test suite to not use the Hive metastore for all of these tests given that its prone to connection leaks at times.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-761941085
I too have found that this test fails. However, it does not appear to be deterministic. I'm guessing it's possible that the tests are run in such an order that the table gets dropped during one of the other tests.
There's also multiple levels of inheritance used for this test suite, but I don't necessarily think that's the issue. However, it does offer some clues, as the non-abstract test suites hat inherit from the higher up classes such as `TestExtensionsTestBase` all seem to pass. The issue appears to only be in `TestCopyOnWriteDelete`, which is possibly conflicting in its `DROP TABLE` statements with the other test suite that extends `SparkRowLevelOperationsTestBase`, `TestCopyOnWrtieMerge`, or is conflicting with tests of its own.
Woud making the names of the tables used in each test somewhat random prevent this scenario? I think this would entail updating `createAndInitTable` to take in the table name and then use that table name instead of a class level variable. This is not exactly a trivial refactor though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhangjun0x01 commented on issue #2098: Spark : The iceberg-spark3-extensions module CI test failed
Posted by GitBox <gi...@apache.org>.
zhangjun0x01 commented on issue #2098:
URL: https://github.com/apache/iceberg/issues/2098#issuecomment-761949467
I found it is random, there are many times the test passed on my computer, but CI test failed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org