You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Steven Wu <st...@gmail.com> on 2021/01/06 23:43:41 UTC
test flakiness with SocketException of broken pipe in HiveMetaStoreClient
I have noticed some flakiness with Flink and Spark tests both locally and
in CI checks. @zhangjun0x01 also reported the same problem with
iceberg-spark3-extensions. Below is a full stack trace from a local run
for Flink tests.
The flakiness might be recent regression, as the tests were stable for me
until recently. Any recent hive dep change? Anyone have any ideas?
org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
java.lang.RuntimeException: Failed to get table info from metastore
default.test
at
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
at
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
at
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
at
org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
at
org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
at
org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
Caused by:
org.apache.thrift.transport.TTransportException:
java.net.SocketException: Broken pipe (Write failed)
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at
org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
at
org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
at
org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
at org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
at
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
... 5 more
Caused by:
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at
java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
... 13 more
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by Ryan Murray <ry...@dremio.com>.
I have come across a few similar issues while (mis)using the HiveCatalog.
My analysis was that the HiveCatalog owns the hive client connection pool
but shares it with the underlying TableOperations. Depending on the
application the HiveCatalog can be closed (or its finalize method run after
its GC-ed) but the underlying TableOperations still are being used. When a
TableOp next tries to talk to hive it finds its already been disconnected.
I have been experimenting with a more cautious method of tracking resource
ownership by removing the finalize method and using a ReferenceQueue but it
looks like a rather disruptive change. However it fixes a lot of issues
like this and it (marginally) improves GC performance. I can respond back
here when I post a candidate PR and we can discuss its value.
Best
Ryan Murray
On Fri, Jan 8, 2021 at 7:50 PM Ryan Blue <rb...@netflix.com.invalid> wrote:
> It could be that there are two separate flaky test issues with not
> releasing connections in Flink and Spark. I don't think that the
> HiveCatalog code has been changed much recently, which would point toward
> problems elsewhere.
>
> I think one good reason to use HiveCatalog is to catch problems like
> these, where the catalog should be closed but isn't.
>
> On Fri, Jan 8, 2021 at 9:23 AM Steven Wu <st...@gmail.com> wrote:
>
>> I use the try-with-resource pattern in the FLIP-27 dev branch. I saw this
>> problem in Flink tests with the master branch too (although less likely).
>> With the FLIP-27 dev branch and an additional DeleteReadTests, it almost
>> happened 100%.
>>
>> Also, the Spark module (in the master branch) also has this flaky test
>> problem. Like Ryan mentioned earlier, maybe some common code (or pattern)
>> causes the issue. It became more flaky recently probably because there are
>> more tests added.
>>
>> Regardless, I still have the question if most unit tests should use
>> HiveCatalog? Why not the cheaper HadoopCatalog except for tests
>> specifically targeting HiveCatalog? Can that help speed up the tests?
>>
>> On Fri, Jan 8, 2021 at 12:06 AM OpenInx <op...@gmail.com> wrote:
>>
>>> OK, there's a try-with-resource to close the TableLoader in
>>> FlinkInputFormat [1]. so we don't have to do the extra try-with-resource
>>> in PR 2051 ( I will close that).
>>>
>>> Under my host, I did not reproduce your connection leak issues when
>>> running TestFlinkInputFormatReaderDeletes. Did you have any extra usage
>>> about the table loader and forget to close it in your flip-27 dev branch ?
>>>
>>> [1].
>>> https://github.com/apache/iceberg/blob/7645ceba65044184be192a7194a38729133b2e50/flink/src/main/java/org/apache/iceberg/flink/source/FlinkInputFormat.java#L77
>>>
>>> On Fri, Jan 8, 2021 at 3:36 PM OpenInx <op...@gmail.com> wrote:
>>>
>>>> > I was able to almost 100% reproduce the HiveMetaStoreClient aborted
>>>> connection problem locally with Flink tests after adding
>>>> another DeleteReadTests for the new FLIP-27 source impl in my dev branch
>>>>
>>>> I think I found the cause why it's easy to fail. The
>>>> TestFlinkInputFormatReaderDeletes will create a new CatalogLoader [1] for
>>>> loading table purposes inside the FlinkInputFormat.
>>>>
>>>> TestHelpers.readRowData(inputFormat, rowType).forEach(rowData -> {
>>>> RowDataWrapper wrapper = new RowDataWrapper(rowType,
>>>> projected.asStruct());
>>>> set.add(wrapper.wrap(rowData));
>>>> });
>>>>
>>>> When TestHelpers#readRowData, it will open a new catalog ( that means
>>>> opening a new hive connection). But after we finished the read processing,
>>>> we did not close the TableLoader, which leaks the catalog connection. I
>>>> opened a PR [2] to fix this issue, will it work in your branch ?
>>>>
>>>> I think it's worth keeping those hive catalog unit tests so that we
>>>> could detect those connection leak issues in time.
>>>>
>>>> [1].
>>>> https://github.com/apache/iceberg/blob/4436c92928f4b3b90839a26bf6a656902733261f/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkInputFormatReaderDeletes.java#L114
>>>> [2]. https://github.com/apache/iceberg/pull/2051/files
>>>>
>>>> On Fri, Jan 8, 2021 at 5:48 AM Steven Wu <st...@gmail.com> wrote:
>>>>
>>>>> Ryan/OpenInx, thanks a lot for the pointers.
>>>>>
>>>>> I was able to almost 100% reproduce the HiveMetaStoreClient aborted
>>>>> connection problem locally with Flink tests after adding
>>>>> another DeleteReadTests for the new FLIP-27 source impl in my dev branch. I
>>>>> don't see the problem anymore after switching the Flink DeleteReadTests
>>>>> from the HiveCatalog (requiring expensive TestHiveMetastore) to
>>>>> HadoopCatalog.
>>>>>
>>>>> There is still a base test class FlinkTestBase using the HiveCatalog.
>>>>> I am wondering if there is a value for using the more expensive HiveCatalog
>>>>> than the HadoopCatalog?
>>>>>
>>>>> On Wed, Jan 6, 2021 at 6:22 PM OpenInx <op...@gmail.com> wrote:
>>>>>
>>>>>> I encountered a similar issue when supporting hive-site.xml for flink
>>>>>> hive catalog. Here is the discussion and solution before:
>>>>>> https://github.com/apache/iceberg/pull/1586#discussion_r509453461
>>>>>>
>>>>>> It's a connection leak issue.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 7, 2021 at 10:06 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> I've noticed this too. I haven't had a chance to track down what's
>>>>>>> causing it yet. I've seen it in Spark tests, so it looks like there may be
>>>>>>> a problem that affects both. Probably a connection leak in the common code.
>>>>>>>
>>>>>>> On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have noticed some flakiness with Flink and Spark tests both
>>>>>>>> locally and in CI checks. @zhangjun0x01 also reported the same problem with
>>>>>>>> iceberg-spark3-extensions. Below is a full stack trace from a
>>>>>>>> local run for Flink tests.
>>>>>>>>
>>>>>>>> The flakiness might be recent regression, as the tests were stable
>>>>>>>> for me until recently. Any recent hive dep change? Anyone have any ideas?
>>>>>>>>
>>>>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
>>>>>>>> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>>>>>>>>
>>>>>>>> java.lang.RuntimeException: Failed to get table info from
>>>>>>>> metastore default.test
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>>>>>>>>
>>>>>>>>
>>>>>>>> Caused by:
>>>>>>>>
>>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>>>>>>>>
>>>>>>>> ... 5 more
>>>>>>>>
>>>>>>>>
>>>>>>>> Caused by:
>>>>>>>>
>>>>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>>>>
>>>>>>>> at java.net.SocketOutputStream.socketWrite0(Native
>>>>>>>> Method)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>>
>>>>>>>> at
>>>>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>>>>>>
>>>>>>>> ... 13 more
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ryan Blue
>>>>>>> Software Engineer
>>>>>>> Netflix
>>>>>>>
>>>>>>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
It could be that there are two separate flaky test issues with not
releasing connections in Flink and Spark. I don't think that the
HiveCatalog code has been changed much recently, which would point toward
problems elsewhere.
I think one good reason to use HiveCatalog is to catch problems like these,
where the catalog should be closed but isn't.
On Fri, Jan 8, 2021 at 9:23 AM Steven Wu <st...@gmail.com> wrote:
> I use the try-with-resource pattern in the FLIP-27 dev branch. I saw this
> problem in Flink tests with the master branch too (although less likely).
> With the FLIP-27 dev branch and an additional DeleteReadTests, it almost
> happened 100%.
>
> Also, the Spark module (in the master branch) also has this flaky test
> problem. Like Ryan mentioned earlier, maybe some common code (or pattern)
> causes the issue. It became more flaky recently probably because there are
> more tests added.
>
> Regardless, I still have the question if most unit tests should use
> HiveCatalog? Why not the cheaper HadoopCatalog except for tests
> specifically targeting HiveCatalog? Can that help speed up the tests?
>
> On Fri, Jan 8, 2021 at 12:06 AM OpenInx <op...@gmail.com> wrote:
>
>> OK, there's a try-with-resource to close the TableLoader in
>> FlinkInputFormat [1]. so we don't have to do the extra try-with-resource
>> in PR 2051 ( I will close that).
>>
>> Under my host, I did not reproduce your connection leak issues when
>> running TestFlinkInputFormatReaderDeletes. Did you have any extra usage
>> about the table loader and forget to close it in your flip-27 dev branch ?
>>
>> [1].
>> https://github.com/apache/iceberg/blob/7645ceba65044184be192a7194a38729133b2e50/flink/src/main/java/org/apache/iceberg/flink/source/FlinkInputFormat.java#L77
>>
>> On Fri, Jan 8, 2021 at 3:36 PM OpenInx <op...@gmail.com> wrote:
>>
>>> > I was able to almost 100% reproduce the HiveMetaStoreClient aborted
>>> connection problem locally with Flink tests after adding
>>> another DeleteReadTests for the new FLIP-27 source impl in my dev branch
>>>
>>> I think I found the cause why it's easy to fail. The
>>> TestFlinkInputFormatReaderDeletes will create a new CatalogLoader [1] for
>>> loading table purposes inside the FlinkInputFormat.
>>>
>>> TestHelpers.readRowData(inputFormat, rowType).forEach(rowData -> {
>>> RowDataWrapper wrapper = new RowDataWrapper(rowType,
>>> projected.asStruct());
>>> set.add(wrapper.wrap(rowData));
>>> });
>>>
>>> When TestHelpers#readRowData, it will open a new catalog ( that means
>>> opening a new hive connection). But after we finished the read processing,
>>> we did not close the TableLoader, which leaks the catalog connection. I
>>> opened a PR [2] to fix this issue, will it work in your branch ?
>>>
>>> I think it's worth keeping those hive catalog unit tests so that we
>>> could detect those connection leak issues in time.
>>>
>>> [1].
>>> https://github.com/apache/iceberg/blob/4436c92928f4b3b90839a26bf6a656902733261f/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkInputFormatReaderDeletes.java#L114
>>> [2]. https://github.com/apache/iceberg/pull/2051/files
>>>
>>> On Fri, Jan 8, 2021 at 5:48 AM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> Ryan/OpenInx, thanks a lot for the pointers.
>>>>
>>>> I was able to almost 100% reproduce the HiveMetaStoreClient aborted
>>>> connection problem locally with Flink tests after adding
>>>> another DeleteReadTests for the new FLIP-27 source impl in my dev branch. I
>>>> don't see the problem anymore after switching the Flink DeleteReadTests
>>>> from the HiveCatalog (requiring expensive TestHiveMetastore) to
>>>> HadoopCatalog.
>>>>
>>>> There is still a base test class FlinkTestBase using the HiveCatalog. I
>>>> am wondering if there is a value for using the more expensive HiveCatalog
>>>> than the HadoopCatalog?
>>>>
>>>> On Wed, Jan 6, 2021 at 6:22 PM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> I encountered a similar issue when supporting hive-site.xml for flink
>>>>> hive catalog. Here is the discussion and solution before:
>>>>> https://github.com/apache/iceberg/pull/1586#discussion_r509453461
>>>>>
>>>>> It's a connection leak issue.
>>>>>
>>>>>
>>>>> On Thu, Jan 7, 2021 at 10:06 AM Ryan Blue <rb...@netflix.com.invalid>
>>>>> wrote:
>>>>>
>>>>>> I've noticed this too. I haven't had a chance to track down what's
>>>>>> causing it yet. I've seen it in Spark tests, so it looks like there may be
>>>>>> a problem that affects both. Probably a connection leak in the common code.
>>>>>>
>>>>>> On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I have noticed some flakiness with Flink and Spark tests both
>>>>>>> locally and in CI checks. @zhangjun0x01 also reported the same problem with
>>>>>>> iceberg-spark3-extensions. Below is a full stack trace from a
>>>>>>> local run for Flink tests.
>>>>>>>
>>>>>>> The flakiness might be recent regression, as the tests were stable
>>>>>>> for me until recently. Any recent hive dep change? Anyone have any ideas?
>>>>>>>
>>>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
>>>>>>> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>>>>>>>
>>>>>>> java.lang.RuntimeException: Failed to get table info from
>>>>>>> metastore default.test
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>>>>>>>
>>>>>>>
>>>>>>> Caused by:
>>>>>>>
>>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>>>>>>>
>>>>>>> ... 5 more
>>>>>>>
>>>>>>>
>>>>>>> Caused by:
>>>>>>>
>>>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>>>
>>>>>>> at java.net.SocketOutputStream.socketWrite0(Native
>>>>>>> Method)
>>>>>>>
>>>>>>> at
>>>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>>>>>>>
>>>>>>> at
>>>>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>>>>>>>
>>>>>>> at
>>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>>
>>>>>>> at
>>>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>>>>>
>>>>>>> ... 13 more
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ryan Blue
>>>>>> Software Engineer
>>>>>> Netflix
>>>>>>
>>>>>
--
Ryan Blue
Software Engineer
Netflix
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by Steven Wu <st...@gmail.com>.
I use the try-with-resource pattern in the FLIP-27 dev branch. I saw this
problem in Flink tests with the master branch too (although less likely).
With the FLIP-27 dev branch and an additional DeleteReadTests, it almost
happened 100%.
Also, the Spark module (in the master branch) also has this flaky test
problem. Like Ryan mentioned earlier, maybe some common code (or pattern)
causes the issue. It became more flaky recently probably because there are
more tests added.
Regardless, I still have the question if most unit tests should use
HiveCatalog? Why not the cheaper HadoopCatalog except for tests
specifically targeting HiveCatalog? Can that help speed up the tests?
On Fri, Jan 8, 2021 at 12:06 AM OpenInx <op...@gmail.com> wrote:
> OK, there's a try-with-resource to close the TableLoader in
> FlinkInputFormat [1]. so we don't have to do the extra try-with-resource
> in PR 2051 ( I will close that).
>
> Under my host, I did not reproduce your connection leak issues when
> running TestFlinkInputFormatReaderDeletes. Did you have any extra usage
> about the table loader and forget to close it in your flip-27 dev branch ?
>
> [1].
> https://github.com/apache/iceberg/blob/7645ceba65044184be192a7194a38729133b2e50/flink/src/main/java/org/apache/iceberg/flink/source/FlinkInputFormat.java#L77
>
> On Fri, Jan 8, 2021 at 3:36 PM OpenInx <op...@gmail.com> wrote:
>
>> > I was able to almost 100% reproduce the HiveMetaStoreClient aborted
>> connection problem locally with Flink tests after adding
>> another DeleteReadTests for the new FLIP-27 source impl in my dev branch
>>
>> I think I found the cause why it's easy to fail. The
>> TestFlinkInputFormatReaderDeletes will create a new CatalogLoader [1] for
>> loading table purposes inside the FlinkInputFormat.
>>
>> TestHelpers.readRowData(inputFormat, rowType).forEach(rowData -> {
>> RowDataWrapper wrapper = new RowDataWrapper(rowType,
>> projected.asStruct());
>> set.add(wrapper.wrap(rowData));
>> });
>>
>> When TestHelpers#readRowData, it will open a new catalog ( that means
>> opening a new hive connection). But after we finished the read processing,
>> we did not close the TableLoader, which leaks the catalog connection. I
>> opened a PR [2] to fix this issue, will it work in your branch ?
>>
>> I think it's worth keeping those hive catalog unit tests so that we could
>> detect those connection leak issues in time.
>>
>> [1].
>> https://github.com/apache/iceberg/blob/4436c92928f4b3b90839a26bf6a656902733261f/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkInputFormatReaderDeletes.java#L114
>> [2]. https://github.com/apache/iceberg/pull/2051/files
>>
>> On Fri, Jan 8, 2021 at 5:48 AM Steven Wu <st...@gmail.com> wrote:
>>
>>> Ryan/OpenInx, thanks a lot for the pointers.
>>>
>>> I was able to almost 100% reproduce the HiveMetaStoreClient aborted
>>> connection problem locally with Flink tests after adding
>>> another DeleteReadTests for the new FLIP-27 source impl in my dev branch. I
>>> don't see the problem anymore after switching the Flink DeleteReadTests
>>> from the HiveCatalog (requiring expensive TestHiveMetastore) to
>>> HadoopCatalog.
>>>
>>> There is still a base test class FlinkTestBase using the HiveCatalog. I
>>> am wondering if there is a value for using the more expensive HiveCatalog
>>> than the HadoopCatalog?
>>>
>>> On Wed, Jan 6, 2021 at 6:22 PM OpenInx <op...@gmail.com> wrote:
>>>
>>>> I encountered a similar issue when supporting hive-site.xml for flink
>>>> hive catalog. Here is the discussion and solution before:
>>>> https://github.com/apache/iceberg/pull/1586#discussion_r509453461
>>>>
>>>> It's a connection leak issue.
>>>>
>>>>
>>>> On Thu, Jan 7, 2021 at 10:06 AM Ryan Blue <rb...@netflix.com.invalid>
>>>> wrote:
>>>>
>>>>> I've noticed this too. I haven't had a chance to track down what's
>>>>> causing it yet. I've seen it in Spark tests, so it looks like there may be
>>>>> a problem that affects both. Probably a connection leak in the common code.
>>>>>
>>>>> On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com> wrote:
>>>>>
>>>>>> I have noticed some flakiness with Flink and Spark tests both locally
>>>>>> and in CI checks. @zhangjun0x01 also reported the same problem with
>>>>>> iceberg-spark3-extensions. Below is a full stack trace from a local
>>>>>> run for Flink tests.
>>>>>>
>>>>>> The flakiness might be recent regression, as the tests were stable
>>>>>> for me until recently. Any recent hive dep change? Anyone have any ideas?
>>>>>>
>>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
>>>>>> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>>>>>>
>>>>>> java.lang.RuntimeException: Failed to get table info from
>>>>>> metastore default.test
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>>>>>>
>>>>>>
>>>>>> Caused by:
>>>>>>
>>>>>> org.apache.thrift.transport.TTransportException:
>>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>>
>>>>>> at
>>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>>>>
>>>>>> at
>>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>>>>>
>>>>>> at
>>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>>>>>>
>>>>>> at
>>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>>>>>>
>>>>>> ... 5 more
>>>>>>
>>>>>>
>>>>>> Caused by:
>>>>>>
>>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>>
>>>>>> at java.net.SocketOutputStream.socketWrite0(Native
>>>>>> Method)
>>>>>>
>>>>>> at
>>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>>>>>>
>>>>>> at
>>>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>>>>>>
>>>>>> at
>>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>>
>>>>>> at
>>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>>>
>>>>>> at
>>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>>>>
>>>>>> ... 13 more
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ryan Blue
>>>>> Software Engineer
>>>>> Netflix
>>>>>
>>>>
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by OpenInx <op...@gmail.com>.
OK, there's a try-with-resource to close the TableLoader in
FlinkInputFormat [1]. so we don't have to do the extra try-with-resource
in PR 2051 ( I will close that).
Under my host, I did not reproduce your connection leak issues when
running TestFlinkInputFormatReaderDeletes. Did you have any extra usage
about the table loader and forget to close it in your flip-27 dev branch ?
[1].
https://github.com/apache/iceberg/blob/7645ceba65044184be192a7194a38729133b2e50/flink/src/main/java/org/apache/iceberg/flink/source/FlinkInputFormat.java#L77
On Fri, Jan 8, 2021 at 3:36 PM OpenInx <op...@gmail.com> wrote:
> > I was able to almost 100% reproduce the HiveMetaStoreClient aborted
> connection problem locally with Flink tests after adding
> another DeleteReadTests for the new FLIP-27 source impl in my dev branch
>
> I think I found the cause why it's easy to fail. The
> TestFlinkInputFormatReaderDeletes will create a new CatalogLoader [1] for
> loading table purposes inside the FlinkInputFormat.
>
> TestHelpers.readRowData(inputFormat, rowType).forEach(rowData -> {
> RowDataWrapper wrapper = new RowDataWrapper(rowType,
> projected.asStruct());
> set.add(wrapper.wrap(rowData));
> });
>
> When TestHelpers#readRowData, it will open a new catalog ( that means
> opening a new hive connection). But after we finished the read processing,
> we did not close the TableLoader, which leaks the catalog connection. I
> opened a PR [2] to fix this issue, will it work in your branch ?
>
> I think it's worth keeping those hive catalog unit tests so that we could
> detect those connection leak issues in time.
>
> [1].
> https://github.com/apache/iceberg/blob/4436c92928f4b3b90839a26bf6a656902733261f/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkInputFormatReaderDeletes.java#L114
> [2]. https://github.com/apache/iceberg/pull/2051/files
>
> On Fri, Jan 8, 2021 at 5:48 AM Steven Wu <st...@gmail.com> wrote:
>
>> Ryan/OpenInx, thanks a lot for the pointers.
>>
>> I was able to almost 100% reproduce the HiveMetaStoreClient aborted
>> connection problem locally with Flink tests after adding
>> another DeleteReadTests for the new FLIP-27 source impl in my dev branch. I
>> don't see the problem anymore after switching the Flink DeleteReadTests
>> from the HiveCatalog (requiring expensive TestHiveMetastore) to
>> HadoopCatalog.
>>
>> There is still a base test class FlinkTestBase using the HiveCatalog. I
>> am wondering if there is a value for using the more expensive HiveCatalog
>> than the HadoopCatalog?
>>
>> On Wed, Jan 6, 2021 at 6:22 PM OpenInx <op...@gmail.com> wrote:
>>
>>> I encountered a similar issue when supporting hive-site.xml for flink
>>> hive catalog. Here is the discussion and solution before:
>>> https://github.com/apache/iceberg/pull/1586#discussion_r509453461
>>>
>>> It's a connection leak issue.
>>>
>>>
>>> On Thu, Jan 7, 2021 at 10:06 AM Ryan Blue <rb...@netflix.com.invalid>
>>> wrote:
>>>
>>>> I've noticed this too. I haven't had a chance to track down what's
>>>> causing it yet. I've seen it in Spark tests, so it looks like there may be
>>>> a problem that affects both. Probably a connection leak in the common code.
>>>>
>>>> On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com> wrote:
>>>>
>>>>> I have noticed some flakiness with Flink and Spark tests both locally
>>>>> and in CI checks. @zhangjun0x01 also reported the same problem with
>>>>> iceberg-spark3-extensions. Below is a full stack trace from a local
>>>>> run for Flink tests.
>>>>>
>>>>> The flakiness might be recent regression, as the tests were stable for
>>>>> me until recently. Any recent hive dep change? Anyone have any ideas?
>>>>>
>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
>>>>> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>>>>>
>>>>> java.lang.RuntimeException: Failed to get table info from
>>>>> metastore default.test
>>>>>
>>>>> at
>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>>>>>
>>>>>
>>>>> Caused by:
>>>>>
>>>>> org.apache.thrift.transport.TTransportException:
>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>
>>>>> at
>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>>>
>>>>> at
>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>>>>
>>>>> at
>>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>>>>>
>>>>> at
>>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>>>>>
>>>>> ... 5 more
>>>>>
>>>>>
>>>>> Caused by:
>>>>>
>>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>>
>>>>> at java.net.SocketOutputStream.socketWrite0(Native
>>>>> Method)
>>>>>
>>>>> at
>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>>>>>
>>>>> at
>>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>>>>>
>>>>> at
>>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>>
>>>>> at
>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>>
>>>>> at
>>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>>>
>>>>> ... 13 more
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Software Engineer
>>>> Netflix
>>>>
>>>
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by OpenInx <op...@gmail.com>.
> I was able to almost 100% reproduce the HiveMetaStoreClient aborted
connection problem locally with Flink tests after adding
another DeleteReadTests for the new FLIP-27 source impl in my dev branch
I think I found the cause why it's easy to fail. The
TestFlinkInputFormatReaderDeletes will create a new CatalogLoader [1] for
loading table purposes inside the FlinkInputFormat.
TestHelpers.readRowData(inputFormat, rowType).forEach(rowData -> {
RowDataWrapper wrapper = new RowDataWrapper(rowType,
projected.asStruct());
set.add(wrapper.wrap(rowData));
});
When TestHelpers#readRowData, it will open a new catalog ( that means
opening a new hive connection). But after we finished the read processing,
we did not close the TableLoader, which leaks the catalog connection. I
opened a PR [2] to fix this issue, will it work in your branch ?
I think it's worth keeping those hive catalog unit tests so that we could
detect those connection leak issues in time.
[1].
https://github.com/apache/iceberg/blob/4436c92928f4b3b90839a26bf6a656902733261f/flink/src/test/java/org/apache/iceberg/flink/source/TestFlinkInputFormatReaderDeletes.java#L114
[2]. https://github.com/apache/iceberg/pull/2051/files
On Fri, Jan 8, 2021 at 5:48 AM Steven Wu <st...@gmail.com> wrote:
> Ryan/OpenInx, thanks a lot for the pointers.
>
> I was able to almost 100% reproduce the HiveMetaStoreClient aborted
> connection problem locally with Flink tests after adding
> another DeleteReadTests for the new FLIP-27 source impl in my dev branch. I
> don't see the problem anymore after switching the Flink DeleteReadTests
> from the HiveCatalog (requiring expensive TestHiveMetastore) to
> HadoopCatalog.
>
> There is still a base test class FlinkTestBase using the HiveCatalog. I am
> wondering if there is a value for using the more expensive HiveCatalog than
> the HadoopCatalog?
>
> On Wed, Jan 6, 2021 at 6:22 PM OpenInx <op...@gmail.com> wrote:
>
>> I encountered a similar issue when supporting hive-site.xml for flink
>> hive catalog. Here is the discussion and solution before:
>> https://github.com/apache/iceberg/pull/1586#discussion_r509453461
>>
>> It's a connection leak issue.
>>
>>
>> On Thu, Jan 7, 2021 at 10:06 AM Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>>> I've noticed this too. I haven't had a chance to track down what's
>>> causing it yet. I've seen it in Spark tests, so it looks like there may be
>>> a problem that affects both. Probably a connection leak in the common code.
>>>
>>> On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com> wrote:
>>>
>>>> I have noticed some flakiness with Flink and Spark tests both locally
>>>> and in CI checks. @zhangjun0x01 also reported the same problem with
>>>> iceberg-spark3-extensions. Below is a full stack trace from a local
>>>> run for Flink tests.
>>>>
>>>> The flakiness might be recent regression, as the tests were stable for
>>>> me until recently. Any recent hive dep change? Anyone have any ideas?
>>>>
>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
>>>> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>>>>
>>>> java.lang.RuntimeException: Failed to get table info from
>>>> metastore default.test
>>>>
>>>> at
>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>>>>
>>>> at
>>>> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>>>>
>>>> at
>>>> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>>>>
>>>> at
>>>> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>>>>
>>>> at
>>>> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>>>>
>>>> at
>>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>>>>
>>>>
>>>> Caused by:
>>>>
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>
>>>> at
>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>>
>>>> at
>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>>>
>>>> at
>>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>>>
>>>> at
>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>>>>
>>>> at
>>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>>>>
>>>> at
>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>>>>
>>>> at
>>>> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>>>>
>>>> at
>>>> org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>>>>
>>>> at
>>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>>>>
>>>> ... 5 more
>>>>
>>>>
>>>> Caused by:
>>>>
>>>> java.net.SocketException: Broken pipe (Write failed)
>>>>
>>>> at java.net.SocketOutputStream.socketWrite0(Native
>>>> Method)
>>>>
>>>> at
>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>>>>
>>>> at
>>>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>>>>
>>>> at
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>>
>>>> at
>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>>
>>>> at
>>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>>
>>>> ... 13 more
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>>
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by Steven Wu <st...@gmail.com>.
Ryan/OpenInx, thanks a lot for the pointers.
I was able to almost 100% reproduce the HiveMetaStoreClient aborted
connection problem locally with Flink tests after adding
another DeleteReadTests for the new FLIP-27 source impl in my dev branch. I
don't see the problem anymore after switching the Flink DeleteReadTests
from the HiveCatalog (requiring expensive TestHiveMetastore) to
HadoopCatalog.
There is still a base test class FlinkTestBase using the HiveCatalog. I am
wondering if there is a value for using the more expensive HiveCatalog than
the HadoopCatalog?
On Wed, Jan 6, 2021 at 6:22 PM OpenInx <op...@gmail.com> wrote:
> I encountered a similar issue when supporting hive-site.xml for flink hive
> catalog. Here is the discussion and solution before:
> https://github.com/apache/iceberg/pull/1586#discussion_r509453461
>
> It's a connection leak issue.
>
>
> On Thu, Jan 7, 2021 at 10:06 AM Ryan Blue <rb...@netflix.com.invalid>
> wrote:
>
>> I've noticed this too. I haven't had a chance to track down what's
>> causing it yet. I've seen it in Spark tests, so it looks like there may be
>> a problem that affects both. Probably a connection leak in the common code.
>>
>> On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com> wrote:
>>
>>> I have noticed some flakiness with Flink and Spark tests both locally
>>> and in CI checks. @zhangjun0x01 also reported the same problem with
>>> iceberg-spark3-extensions. Below is a full stack trace from a local
>>> run for Flink tests.
>>>
>>> The flakiness might be recent regression, as the tests were stable for
>>> me until recently. Any recent hive dep change? Anyone have any ideas?
>>>
>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
>>> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>>>
>>> java.lang.RuntimeException: Failed to get table info from metastore
>>> default.test
>>>
>>> at
>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>>>
>>> at
>>> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>>>
>>> at
>>> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>>>
>>> at
>>> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>>>
>>> at
>>> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>>>
>>> at
>>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>>>
>>>
>>> Caused by:
>>>
>>> org.apache.thrift.transport.TTransportException:
>>> java.net.SocketException: Broken pipe (Write failed)
>>>
>>> at
>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>>
>>> at
>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>>
>>> at
>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>>
>>> at
>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>>>
>>> at
>>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>>>
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>>>
>>> at
>>> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>>>
>>> at
>>> org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>>>
>>> at
>>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>>>
>>> ... 5 more
>>>
>>>
>>> Caused by:
>>>
>>> java.net.SocketException: Broken pipe (Write failed)
>>>
>>> at java.net.SocketOutputStream.socketWrite0(Native
>>> Method)
>>>
>>> at
>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>>>
>>> at
>>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>>>
>>> at
>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>
>>> at
>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>
>>> at
>>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>>
>>> ... 13 more
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix
>>
>
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by OpenInx <op...@gmail.com>.
I encountered a similar issue when supporting hive-site.xml for flink hive
catalog. Here is the discussion and solution before:
https://github.com/apache/iceberg/pull/1586#discussion_r509453461
It's a connection leak issue.
On Thu, Jan 7, 2021 at 10:06 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> I've noticed this too. I haven't had a chance to track down what's causing
> it yet. I've seen it in Spark tests, so it looks like there may be a
> problem that affects both. Probably a connection leak in the common code.
>
> On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com> wrote:
>
>> I have noticed some flakiness with Flink and Spark tests both locally and
>> in CI checks. @zhangjun0x01 also reported the same problem with
>> iceberg-spark3-extensions. Below is a full stack trace from a local run
>> for Flink tests.
>>
>> The flakiness might be recent regression, as the tests were stable for me
>> until recently. Any recent hive dep change? Anyone have any ideas?
>>
>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
>> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>>
>> java.lang.RuntimeException: Failed to get table info from metastore
>> default.test
>>
>> at
>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>>
>> at
>> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>>
>> at
>> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>>
>> at
>> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>>
>> at
>> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>>
>> at
>> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>>
>>
>> Caused by:
>>
>> org.apache.thrift.transport.TTransportException:
>> java.net.SocketException: Broken pipe (Write failed)
>>
>> at
>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>>
>> at
>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>
>> at
>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>
>> at
>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>>
>> at
>> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>>
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>>
>> at
>> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>>
>> at org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>>
>> at
>> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>>
>> ... 5 more
>>
>>
>> Caused by:
>>
>> java.net.SocketException: Broken pipe (Write failed)
>>
>> at java.net.SocketOutputStream.socketWrite0(Native
>> Method)
>>
>> at
>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>>
>> at
>> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>>
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>
>> at
>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>
>> at
>> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>>
>> ... 13 more
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
I've noticed this too. I haven't had a chance to track down what's causing
it yet. I've seen it in Spark tests, so it looks like there may be a
problem that affects both. Probably a connection leak in the common code.
On Wed, Jan 6, 2021 at 3:44 PM Steven Wu <st...@gmail.com> wrote:
> I have noticed some flakiness with Flink and Spark tests both locally and
> in CI checks. @zhangjun0x01 also reported the same problem with
> iceberg-spark3-extensions. Below is a full stack trace from a local run
> for Flink tests.
>
> The flakiness might be recent regression, as the tests were stable for me
> until recently. Any recent hive dep change? Anyone have any ideas?
>
> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes >
> testMixedPositionAndEqualityDeletes[fileFormat=ORC] FAILED
>
> java.lang.RuntimeException: Failed to get table info from metastore
> default.test
>
> at
> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:142)
>
> at
> org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:86)
>
> at
> org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:69)
>
> at
> org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:92)
>
> at
> org.apache.iceberg.flink.TableLoader$CatalogTableLoader.loadTable(TableLoader.java:113)
>
> at
> org.apache.iceberg.flink.source.TestIcebergSourceReaderDeletes.rowSet(TestIcebergSourceReaderDeletes.java:90)
>
>
> Caused by:
>
> org.apache.thrift.transport.TTransportException:
> java.net.SocketException: Broken pipe (Write failed)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
>
> at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>
> at
> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_table_req(ThriftHiveMetastore.java:1561)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1553)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1350)
>
> at
> org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:130)
>
> at org.apache.iceberg.hive.ClientPool.run(ClientPool.java:65)
>
> at
> org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:130)
>
> ... 5 more
>
>
> Caused by:
>
> java.net.SocketException: Broken pipe (Write failed)
>
> at java.net.SocketOutputStream.socketWrite0(Native Method)
>
> at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>
> at
> java.net.SocketOutputStream.write(SocketOutputStream.java:155)
>
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>
> at
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>
> at
> org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
>
> ... 13 more
>
--
Ryan Blue
Software Engineer
Netflix