You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2026/02/24 16:59:17 UTC
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24029
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
JVM threads are being reused across JNI invocations. This means
if the interrupt flag is being set for a thread when the JNI call
returns, the next JNI call might get the interrupt (if it checks for
it).
It can be reproduced manually by adding
Thread.currentThread().interrupt() calls at the end of our methods
(e.g. JniFrontend.convertTable(), JniCatalog.updateCatalog()).
Doing so we can trigger errors we can see in IMPALA-12621, IMPALA-10633,
IMPALA-10924, IMPALA-10540, IMPALA-12261. All these issues have
a stack trace similar to the following:
W20260221 07:53:31.855443 1324125 DataStreamer.java:832] DataStreamer Exception
Java exception follows:
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:781)
I20260221 07:53:31.863165 1312138 jni-util.cc:321] fa4015e85de1494e:e1586d1500000000] org.apache.iceberg.exceptions.RuntimeIOException: Failed to write json to file: hdfs://localhost:20500/test-warehouse/lineitem_sixblocks_iceberg/metadata/00000-93a5f622-d41e-4ccb-87a4-eb26d9bd7a5c.metadata.json
at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133)
at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:173)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75)
at org.apache.impala.catalog.iceberg.IcebergHiveCatalog.createTable(IcebergHiveCatalog.java:74)
at org.apache.impala.util.MigrateTableUtil.migrateToIcebergTable(MigrateTableUtil.java:99)
at org.apache.impala.service.Frontend.convertTable(Frontend.java:1004)
at org.apache.impala.service.JniFrontend.convertTable(JniFrontend.java:243)
I.e., something interrupted the thread when it tried to create a new
JSON file for an Iceberg table. Looking at the HDFS logs, the file
was created successfully:
org.apache.hadoop.hdfs.StateChange: DIR* completeFile: ...metadata.json
There is also no thread interruption logic in the code path of
JniFrontend.convertTable() that could explain the interruption.
So probably the best explanation to the above is:
* JNI thread gets interrupted in a way its interrupt flag remains set
* New JNI call on thread runs Iceberg code that checks interruption
* Above exception being thrown
Testing:
* tested manually by adding Thread.currentThread().interrupt() calls
at the end of JniCatalog/JniFronted methods.
Generated-by: Gemini Pro
Generated-by: Claude Sonnet 4.5
Change-Id: Iaec6860433431064737e994999dd57a63f223a20
---
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M fe/src/main/java/org/apache/impala/common/JniUtil.java
3 files changed, 45 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/24029/1
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21799/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 14:32:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13173/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 16:59:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 2:
I've run two exhaustive tests, one failed with IMPALA-10667.
The other one failed with IMPALA-10667 also, and two unrelated issues:
- test_skipping_older_events (probably IMPALA-12416)
- FE test: LdapSearchBindImpalaShellTest.testCookieRefresh
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 10:05:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 16:58:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 21:15:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 2: Code-Review+1
Willing to go to +2 if exhaustive tests pass.
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 16:27:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 16:59:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21774/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Feb 2026 17:25:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 1: Code-Review+1
This seems useful. I was wondering if IMPALA-915 might contribute to this, but the timing looks wrong.
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Feb 2026 18:22:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24029/1/be/src/util/jni-util.h
File be/src/util/jni-util.h:
http://gerrit.cloudera.org:8080/#/c/24029/1/be/src/util/jni-util.h@539
PS1, Line 539: // Clear interrupt status of JNI thread.
: JniUtil::ClearThreadInterruptStatus(env_);
For debug builds my preference would be to check both before and after the call, and treat it as a DCHECK if it is interrupted.
This would help in pinpointing the culprit - if the "before" DCHECK is hit, then some other Java call was setting interruption, e.g. libhdfs file operations. If the "after" DCHECK is hit, then we know that it happened during the given JNI call.
I am not sure about release - calling to JNI has some overhead (practically two calls to JVM instead of one), but our JNI interface is pretty course grained, so probably it wouldn't be noticable.
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Feb 2026 18:30:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 14:25:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 2:
(1 comment)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/24029/1/be/src/util/jni-util.h
File be/src/util/jni-util.h:
http://gerrit.cloudera.org:8080/#/c/24029/1/be/src/util/jni-util.h@539
PS1, Line 539: // Clear interrupt status of JNI thread.
: JniUtil::ClearThreadInterruptStatus(env_);
> For debug builds my preference would be to check both before and after the
Core tests passed with DCHECKs, so it's probably not that frequent. Currently I'm running exhaustive tests.
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 14:07:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
JVM threads are being reused across JNI invocations. This means
if the interrupt flag is being set for a thread when the JNI call
returns, the next JNI call might get the interrupt (if it checks for
it).
It can be reproduced manually by adding
Thread.currentThread().interrupt() calls at the end of our methods
(e.g. JniFrontend.convertTable(), JniCatalog.updateCatalog()).
Doing so we can trigger errors we can see in IMPALA-12621, IMPALA-10633,
IMPALA-10924, IMPALA-10540, IMPALA-12261. All these issues have
a stack trace similar to the following:
W20260221 07:53:31.855443 1324125 DataStreamer.java:832] DataStreamer Exception
Java exception follows:
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:781)
I20260221 07:53:31.863165 1312138 jni-util.cc:321] fa4015e85de1494e:e1586d1500000000] org.apache.iceberg.exceptions.RuntimeIOException: Failed to write json to file: hdfs://localhost:20500/test-warehouse/lineitem_sixblocks_iceberg/metadata/00000-93a5f622-d41e-4ccb-87a4-eb26d9bd7a5c.metadata.json
at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133)
at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:173)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75)
at org.apache.impala.catalog.iceberg.IcebergHiveCatalog.createTable(IcebergHiveCatalog.java:74)
at org.apache.impala.util.MigrateTableUtil.migrateToIcebergTable(MigrateTableUtil.java:99)
at org.apache.impala.service.Frontend.convertTable(Frontend.java:1004)
at org.apache.impala.service.JniFrontend.convertTable(JniFrontend.java:243)
I.e., something interrupted the thread when it tried to create a new
JSON file for an Iceberg table. Looking at the HDFS logs, the file
was created successfully:
org.apache.hadoop.hdfs.StateChange: DIR* completeFile: ...metadata.json
There is also no thread interruption logic in the code path of
JniFrontend.convertTable() that could explain the interruption.
So probably the best explanation to the above is:
* JNI thread gets interrupted in a way its interrupt flag remains set
* New JNI call on thread runs Iceberg code that checks interruption
* Above exception being thrown
Testing:
* tested manually by adding Thread.currentThread().interrupt() calls
at the end of JniCatalog/JniFronted methods.
Generated-by: Gemini Pro
Generated-by: Claude Sonnet 4.5
Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Reviewed-on: http://gerrit.cloudera.org:8080/24029
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M fe/src/main/java/org/apache/impala/common/JniUtil.java
3 files changed, 51 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24029
to look at the new patch set (#3).
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
JVM threads are being reused across JNI invocations. This means
if the interrupt flag is being set for a thread when the JNI call
returns, the next JNI call might get the interrupt (if it checks for
it).
It can be reproduced manually by adding
Thread.currentThread().interrupt() calls at the end of our methods
(e.g. JniFrontend.convertTable(), JniCatalog.updateCatalog()).
Doing so we can trigger errors we can see in IMPALA-12621, IMPALA-10633,
IMPALA-10924, IMPALA-10540, IMPALA-12261. All these issues have
a stack trace similar to the following:
W20260221 07:53:31.855443 1324125 DataStreamer.java:832] DataStreamer Exception
Java exception follows:
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:781)
I20260221 07:53:31.863165 1312138 jni-util.cc:321] fa4015e85de1494e:e1586d1500000000] org.apache.iceberg.exceptions.RuntimeIOException: Failed to write json to file: hdfs://localhost:20500/test-warehouse/lineitem_sixblocks_iceberg/metadata/00000-93a5f622-d41e-4ccb-87a4-eb26d9bd7a5c.metadata.json
at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133)
at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:173)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75)
at org.apache.impala.catalog.iceberg.IcebergHiveCatalog.createTable(IcebergHiveCatalog.java:74)
at org.apache.impala.util.MigrateTableUtil.migrateToIcebergTable(MigrateTableUtil.java:99)
at org.apache.impala.service.Frontend.convertTable(Frontend.java:1004)
at org.apache.impala.service.JniFrontend.convertTable(JniFrontend.java:243)
I.e., something interrupted the thread when it tried to create a new
JSON file for an Iceberg table. Looking at the HDFS logs, the file
was created successfully:
org.apache.hadoop.hdfs.StateChange: DIR* completeFile: ...metadata.json
There is also no thread interruption logic in the code path of
JniFrontend.convertTable() that could explain the interruption.
So probably the best explanation to the above is:
* JNI thread gets interrupted in a way its interrupt flag remains set
* New JNI call on thread runs Iceberg code that checks interruption
* Above exception being thrown
Testing:
* tested manually by adding Thread.currentThread().interrupt() calls
at the end of JniCatalog/JniFronted methods.
Generated-by: Gemini Pro
Generated-by: Claude Sonnet 4.5
Change-Id: Iaec6860433431064737e994999dd57a63f223a20
---
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M fe/src/main/java/org/apache/impala/common/JniUtil.java
3 files changed, 51 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/24029/3
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24029
to look at the new patch set (#2).
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
JVM threads are being reused across JNI invocations. This means
if the interrupt flag is being set for a thread when the JNI call
returns, the next JNI call might get the interrupt (if it checks for
it).
It can be reproduced manually by adding
Thread.currentThread().interrupt() calls at the end of our methods
(e.g. JniFrontend.convertTable(), JniCatalog.updateCatalog()).
Doing so we can trigger errors we can see in IMPALA-12621, IMPALA-10633,
IMPALA-10924, IMPALA-10540, IMPALA-12261. All these issues have
a stack trace similar to the following:
W20260221 07:53:31.855443 1324125 DataStreamer.java:832] DataStreamer Exception
Java exception follows:
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:781)
I20260221 07:53:31.863165 1312138 jni-util.cc:321] fa4015e85de1494e:e1586d1500000000] org.apache.iceberg.exceptions.RuntimeIOException: Failed to write json to file: hdfs://localhost:20500/test-warehouse/lineitem_sixblocks_iceberg/metadata/00000-93a5f622-d41e-4ccb-87a4-eb26d9bd7a5c.metadata.json
at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133)
at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:115)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:170)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadataIfRequired(BaseMetastoreTableOperations.java:160)
at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:173)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201)
at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75)
at org.apache.impala.catalog.iceberg.IcebergHiveCatalog.createTable(IcebergHiveCatalog.java:74)
at org.apache.impala.util.MigrateTableUtil.migrateToIcebergTable(MigrateTableUtil.java:99)
at org.apache.impala.service.Frontend.convertTable(Frontend.java:1004)
at org.apache.impala.service.JniFrontend.convertTable(JniFrontend.java:243)
I.e., something interrupted the thread when it tried to create a new
JSON file for an Iceberg table. Looking at the HDFS logs, the file
was created successfully:
org.apache.hadoop.hdfs.StateChange: DIR* completeFile: ...metadata.json
There is also no thread interruption logic in the code path of
JniFrontend.convertTable() that could explain the interruption.
So probably the best explanation to the above is:
* JNI thread gets interrupted in a way its interrupt flag remains set
* New JNI call on thread runs Iceberg code that checks interruption
* Above exception being thrown
Testing:
* tested manually by adding Thread.currentThread().interrupt() calls
at the end of JniCatalog/JniFronted methods.
Generated-by: Gemini Pro
Generated-by: Claude Sonnet 4.5
Change-Id: Iaec6860433431064737e994999dd57a63f223a20
---
M be/src/util/jni-util.cc
M be/src/util/jni-util.h
M fe/src/main/java/org/apache/impala/common/JniUtil.java
3 files changed, 49 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/24029/2
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24029 )
Change subject: IMPALA-12621: Prevent Thread interrupt flag poisoning of JVM threads
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21831/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24029
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaec6860433431064737e994999dd57a63f223a20
Gerrit-Change-Number: 24029
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Mar 2026 15:08:44 +0000
Gerrit-HasComments: No