You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "zabetak (via GitHub)" <gi...@apache.org> on 2023/06/02 15:04:26 UTC
[GitHub] [hive] zabetak opened a new pull request, #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
zabetak opened a new pull request, #4386:
URL: https://github.com/apache/hive/pull/4386
### What changes were proposed in this pull request?
The topicPartitionBookkeeper field was removed from TransactionManager in the latest Kafka version and was replaced by txnPartitionMap.
Below the commit which applied the respective refactoring in Kafka: https://github.com/apache/kafka/commit/3ea7b418fb3d7e9fc74c27751c1b02b04877f197
### Why are the changes needed?
Resolve `NoSuchFieldException` when running INSERT queries in Kafka tables
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
```
mvn -pl itests/qtest -Pitests test -Dtest=TestMiniHiveKafkaCliDriver -Dqfile=kafka_storage_handler.q
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4386:
URL: https://github.com/apache/hive/pull/4386#issuecomment-1574249650
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4386)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4386&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4386&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4386&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4386&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4386&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] rkirtir commented on a diff in pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "rkirtir (via GitHub)" <gi...@apache.org>.
rkirtir commented on code in PR #4386:
URL: https://github.com/apache/hive/pull/4386#discussion_r1216171512
##########
kafka-handler/src/java/org/apache/hadoop/hive/kafka/HiveKafkaProducer.java:
##########
Review Comment:
We need to check on tests like HiveKafkaProducerTest, TransactionalKafkaWriterTest which are currently ignored from test suit but those partially/completely fails on " Incompatible KafkaProducer version"
Caused by: java.lang.NoSuchFieldException: topicPartitionBookkeeper
at java.lang.Class.getDeclaredField(Class.java:2070)
As these tests are Ignored at present, we might miss out on those
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on PR #4386:
URL: https://github.com/apache/hive/pull/4386#issuecomment-1634453211
@zabetak any changes required in this post downgrade?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
URL: https://github.com/apache/hive/pull/4386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4386:
URL: https://github.com/apache/hive/pull/4386#issuecomment-1581079530
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4386)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4386&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4386&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4386&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=CODE_SMELL) [19 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4386&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4386&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4386&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] akshat0395 commented on pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "akshat0395 (via GitHub)" <gi...@apache.org>.
akshat0395 commented on PR #4386:
URL: https://github.com/apache/hive/pull/4386#issuecomment-1574694518
Hi @zabetak will this change break any the backwards compatibility with older kafka version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4386:
URL: https://github.com/apache/hive/pull/4386#issuecomment-1714782704
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4386:
URL: https://github.com/apache/hive/pull/4386#discussion_r2108949899
##########
kafka-handler/src/java/org/apache/hadoop/hive/kafka/HiveKafkaProducer.java:
##########
Review Comment:
@akshat0395 enabled the HiveKafkaProducerTest, only TransactionalKafkaWriterTest remained
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
URL: https://github.com/apache/hive/pull/4386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on pull request #4386: HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #4386:
URL: https://github.com/apache/hive/pull/4386#issuecomment-1576517710
> Hi @zabetak will this change break any the backwards compatibility with older kafka version?
Any code that relies on reflection and internal classes is unlike to be backward compatible in the long term. This change does not break backward compatibility; this already happened in HIVE-27105 where Kafka was upgraded from 2.5.0 to 3.4.0.
Hive does not guarantee that it will be compatible with multiple Kafka, Hadoop, Tez, etc. versions. We should just ensure that the one used in the `pom.xml` is supported/working. In other words, I don't think we should worry to much about previous Kafka versions but just make sure that things work for the one we actually use i.e., 3.4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4386:
URL: https://github.com/apache/hive/pull/4386#discussion_r2108949899
##########
kafka-handler/src/java/org/apache/hadoop/hive/kafka/HiveKafkaProducer.java:
##########
Review Comment:
@akshat0395 enabled the `HiveKafkaProducerTest`, only `TransactionalKafkaWriterTest` remained
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27407: INSERT INTO external Kafka table fails with NoSuchFieldException [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4386:
URL: https://github.com/apache/hive/pull/4386#issuecomment-3123651516
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org