You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/01/12 11:06:12 UTC
[GitHub] [hive] abstractdog opened a new pull request #2944: HIVE-25637: Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
abstractdog opened a new pull request #2944:
URL: https://github.com/apache/hive/pull/2944
### What changes were proposed in this pull request?
Supporting INSERT into a kafka table in a secure cluster. Further details are in JIRA description.
### Why are the changes needed?
INSERT into a kafka table failed.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Tested on cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a change in pull request #2944: HIVE-25637: Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #2944:
URL: https://github.com/apache/hive/pull/2944#discussion_r783097207
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
##########
@@ -311,10 +325,18 @@ private void getKafkaCredentials(MapWork work, DAG dag, JobConf conf) {
break;
}
}
+
+ for (TableDesc tableDesc : fileSinkTableDescs) {
+ kafkaBrokers = (String) tableDesc.getProperties().get("kafka.bootstrap.servers"); //FIXME: KafkaTableProperties
+ if (kafkaBrokers != null && !kafkaBrokers.isEmpty()) {
Review comment:
thanks @maheshk114, makes sense, introduced collectKafkaDelegationTokenForTableDesc, I've just force pushed a new commit including this, please let me know if it's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog merged pull request #2944: HIVE-25637: Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
Posted by GitBox <gi...@apache.org>.
abstractdog merged pull request #2944:
URL: https://github.com/apache/hive/pull/2944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] maheshk114 commented on a change in pull request #2944: HIVE-25637: Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
Posted by GitBox <gi...@apache.org>.
maheshk114 commented on a change in pull request #2944:
URL: https://github.com/apache/hive/pull/2944#discussion_r783073682
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
##########
@@ -311,10 +325,18 @@ private void getKafkaCredentials(MapWork work, DAG dag, JobConf conf) {
break;
}
}
+
+ for (TableDesc tableDesc : fileSinkTableDescs) {
+ kafkaBrokers = (String) tableDesc.getProperties().get("kafka.bootstrap.servers"); //FIXME: KafkaTableProperties
+ if (kafkaBrokers != null && !kafkaBrokers.isEmpty()) {
Review comment:
The common code can be reused instead of another copy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #2944: HIVE-25637: Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #2944:
URL: https://github.com/apache/hive/pull/2944#issuecomment-1012233987
merged to master, thanks @maheshk114 for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #2944: HIVE-25637: Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #2944:
URL: https://github.com/apache/hive/pull/2944#issuecomment-1012233309
TestReplWithJsonMessageFormat failure is unrelated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a change in pull request #2944: HIVE-25637: Hive on Tez: inserting data failing into the non native hive external table managed by kafka storage handler
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #2944:
URL: https://github.com/apache/hive/pull/2944#discussion_r783097207
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
##########
@@ -311,10 +325,18 @@ private void getKafkaCredentials(MapWork work, DAG dag, JobConf conf) {
break;
}
}
+
+ for (TableDesc tableDesc : fileSinkTableDescs) {
+ kafkaBrokers = (String) tableDesc.getProperties().get("kafka.bootstrap.servers"); //FIXME: KafkaTableProperties
+ if (kafkaBrokers != null && !kafkaBrokers.isEmpty()) {
Review comment:
thanks @maheshk114, makes sense, I've just force pushed a new commit including this, please let me know if it's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org