You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/02/24 07:56:55 UTC
[GitHub] [spark] wangyum opened a new pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
wangyum opened a new pull request #35646:
URL: https://github.com/apache/spark/pull/35646
This is a backport of https://github.com/apache/spark/pull/34362 to branch 3.2.
### What changes were proposed in this pull request?
This PR ported HIVE-21498, HIVE-25098 and upgraded libthrift to 0.16.0.
The CHANGES list for libthrift 0.16.0 is available at: https://github.com/apache/thrift/blob/v0.16.0/CHANGES.md
### Why are the changes needed?
To address [CVE-2020-13949](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056333571
Ya, @sunchao is right.
To @srowen and @wangyum . This patch is not released yet. I'm going to revert this from all branches.
We can land this back. However, before that, at least, we need to make it sure that the following three new `abstract` methods (at https://github.com/apache/thrift/commit/63213c17ad3fece91fdaaca8f59165ca3f41c5c1 ) are handled correctly.
- https://github.com/apache/thrift/blob/8ab86c3303a8157ecfed6ff588d71e6e13dd7017/lib/java/src/org/apache/thrift/transport/TTransport.java#L205-L209
```java
public abstract TConfiguration getConfiguration();
public abstract void updateKnownMessageSize(long size) throws TTransportException;
public abstract void checkReadBytesAvailable(long numBytes) throws TTransportException;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056125525
To be clear, I didn't take a look at those two huge patches, but I'm not sure those patches are able to land at Apache Hive 2.3.10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056101781
Hi, @wangyum . Thanks, but, before releasing a new Hive, we need to revert libthrift 0.16.0 from master/3.2/3.1.
Apache Spark 3.3 branch cut is planned on March 15th. We have only two weeks.
Could you revert SPARK-37090 from master/3.2/3.1 as a committer and author, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35646:
URL: https://github.com/apache/spark/pull/35646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056328220
> Could we backport https://issues.apache.org/jira/browse/HIVE-21498 and https://issues.apache.org/jira/browse/HIVE-25098 to branch-2.3 and release a new version?
@wangyum I can give it a try but it could be challenging given the amount of changes in these two. It'll take some time and most likely won't be ready before Spark 3.3 release.
> ... what's different about where it fails than what the tests run?
@srowen I think this scenario is not covered in any test in Spark - it requires a remote secure HMS but Spark Hive tests only use embedded HMS through Derby.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1052199808
Merged to 3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056124406
Hi, @srowen .
- As of today, Apache Hive 2.3.x is built with libthrift 0.9.3 and it turned out that it is incompatible with libthrift 0.14.1+.
```
<libthrift.version>0.9.3</libthrift.version>
```
- @wangyum suggested two Hive 4.0 huge patches
- https://github.com/apache/hive/pull/1455 (HIVE-21498: Upgrade Thrift to 0.13.0, **+539,057 −310,147**)
- https://github.com/apache/hive/pull/2330 (HIVE-25098: Upgrade thrift from 0.13.0 to 0.14.1, **+9,012 −10,943**)
Given the size of Hive patches, I don't think we can afford these in Apache Spark 3.3/3.2/3.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1057168412
I added [my comment on your Hive PR](https://github.com/apache/hive/pull/3066#pullrequestreview-897886062), too.
> Just a question: What about branch-3.1 and branch-3.0? Are these patches released officially in any Apache Hive artifacts after vote?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056101781
Hi, @wangyum . Thanks, but, before releasing a new Hive, we need to revert libthrift 0.16.0 from master/3.2/3.1.
Apache Spark 3.3 branch cut is planned on March 15th. We have only two weeks.
Could you revert them as a committer and author, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056128029
@srowen . For your questions, @sunchao found that `libthrift 0.14.1+ requires an implementation of abstract method, `checkReadBytesAvailable`, and Apache Hive branch-2.3 didn't implemented it.
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport does not define or inherit an implementation of the resolved method 'abstract void checkReadBytesAvailable(long)' of abstract class org.apache.thrift.transport.TTransport.
org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:444)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056128029
@srowen . For your questions, @sunchao found that `libthrift 0.14.1+` requires an implementation of abstract method, `checkReadBytesAvailable`, and Apache Hive branch-2.3 didn't implemented it.
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport does not define or inherit an implementation of the resolved method 'abstract void checkReadBytesAvailable(long)' of abstract class org.apache.thrift.transport.TTransport.
org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:444)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056032697
We are using the same Hive version 2.3.9 as in upstream Spark. I checked `TUGIAssumingTransport` [there](https://github.com/apache/hive/blob/branch-2.3/shims/common/src/main/java/org/apache/hadoop/hive/thrift/client/TUGIAssumingTransport.java) and it doesn't implement `checkReadBytesAvailable` which is from Thrift 0.14 and up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056333571
Ya, @sunchao is right.
To @srowen and @wangyum . SPARK-37090 is not released yet. I'm going to revert this from all branches.
We can land this back. However, before that, at least, we need to make it sure that the following three new `abstract` methods (at https://github.com/apache/thrift/commit/63213c17ad3fece91fdaaca8f59165ca3f41c5c1 ) are handled correctly.
- https://github.com/apache/thrift/blob/8ab86c3303a8157ecfed6ff588d71e6e13dd7017/lib/java/src/org/apache/thrift/transport/TTransport.java#L205-L209
```java
public abstract TConfiguration getConfiguration();
public abstract void updateKnownMessageSize(long size) throws TTransportException;
public abstract void checkReadBytesAvailable(long numBytes) throws TTransportException;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
wangyum edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056417289
Thank you @dongjoon-hyun This PR(https://github.com/apache/hive/pull/3066) tries to backport HIVE-21498 and HIVE-25098 to branch-2.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056070319
@sunchao Could we backport https://issues.apache.org/jira/browse/HIVE-21498 and https://issues.apache.org/jira/browse/HIVE-25098 to branch-2.3 and release a new version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056124406
Hi, @srowen .
- As of today, Apache Hive 2.3.x is built with libthrift 0.9.3 and is incompatible with libthrift 0.14.1+.
```
<libthrift.version>0.9.3</libthrift.version>
```
- @wangyum suggesting two Hive 4.0 huge patches
- https://github.com/apache/hive/pull/1455 (HIVE-21498: Upgrade Thrift to 0.13.0, **+539,057 −310,147**)
- https://github.com/apache/hive/pull/2330 (HIVE-25098: Upgrade thrift from 0.13.0 to 0.14.1, **+9,012 −10,943**)
Given the size of Hive patches, I don't think we can afford these in Apache Spark 3.3/3.2/3.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056125525
To be clear, I didn't take a look at those two huge pages, but I'm not sure those patches are able to land at Apache Hive 2.3.10.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1057157986
Thank you, @wangyum !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1050913647
Weird, is this running tests? I don't see the test workflow executed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1052139515
Triggered the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056372669
In addition, I added those two Hive 4.0 JIRA links as a blocker for SPARK-37090.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056025564
Hi @wangyum @srowen , after picking up this change and deploy it internally, we found an issue that seems to be related:
```
Receiver class org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport does not define or inherit an implementation of the resolved method 'abstract void checkReadBytesAvailable(long)' of abstract class org.apache.thrift.transport.TTransport.
org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:444)
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:415)
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:411)
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:251)
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1514)
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1500)
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1370)
org.apache.iceberg.hive.HiveTableOperations.lambda$doRefresh$0(HiveTableOperations.java:182)
org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:182)
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:94)
```
still investigating what exactly caused the error though. The `TUGIAssumingTransport` class is from Hive client side and seems it's used when talking to a secure HMS via either delegation token or kerberos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056417289
Thank you @dongjoon-hyun This PR(https://github.com/apache/hive/pull/3066) try to backport HIVE-21498 and HIVE-25098 to branch-2.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056030870
This seems like a Hive version issue - what are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056101781
Hi, @wangyum . Before releasing a new Hive, we need to revert libthrift 0.16.0 from master/3.2/3.1.
Apache Spark 3.3 branch cut is planned on March 15th. We have only two weeks.
Could you revert them as a committer and author, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056101781
Hi, @wangyum . Before release a new Hive, we need to revert libthrift 0.16.0 from master/3.2/3.1.
Apache Spark 3.3 branch cut is planned on March 15th. We have only two weeks.
Could you revert them as a committer and author, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056114609
(Could someone briefly explain the issue - what's different about where it fails than what the tests run? not something that can be just fixed-forward?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #35646: [SPARK-37090][BUILD][3.2] Upgrade libthrift to 0.16.0 to avoid security vulnerabilities
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #35646:
URL: https://github.com/apache/spark/pull/35646#issuecomment-1056128029
For your questions, @sunchao found that `libthrift 0.14.1+ requires an implementation of abstract method, `checkReadBytesAvailable`, and Apache Hive branch-2.3 didn't implemented it.
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport does not define or inherit an implementation of the resolved method 'abstract void checkReadBytesAvailable(long)' of abstract class org.apache.thrift.transport.TTransport.
org.apache.thrift.protocol.TBinaryProtocol.checkStringReadLength(TBinaryProtocol.java:444)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org