You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "ChengJie1053 (via GitHub)" <gi...@apache.org> on 2023/05/19 06:27:17 UTC
[GitHub] [linkis] ChengJie1053 opened a new pull request, #4563: spark etl support kafka
ChengJie1053 opened a new pull request, #4563:
URL: https://github.com/apache/linkis/pull/4563
spark etl support kafka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on a diff in pull request #4563: spark etl support kafka
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on code in PR #4563:
URL: https://github.com/apache/linkis/pull/4563#discussion_r1205192149
##########
tool/dependencies/known-dependencies.txt:
##########
@@ -491,7 +491,11 @@ snakeyaml-1.33.jar
snappy-java-1.1.4.jar
snappy-java-1.1.7.7.jar
snappy-java-1.1.8.2.jar
+snappy-java-1.1.8.4.jar
spark-redis_2.12-2.6.0.jar
+spark-sql-kafka-0-10_2.12-3.2.1.jar
+spark-tags_2.12-3.2.1.jar
+spark-token-provider-kafka-0-10_2.12-3.2.1.jar
Review Comment:
can delete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] ChengJie1053 commented on a diff in pull request #4563: spark etl support kafka
Posted by "ChengJie1053 (via GitHub)" <gi...@apache.org>.
ChengJie1053 commented on code in PR #4563:
URL: https://github.com/apache/linkis/pull/4563#discussion_r1203589688
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -388,6 +388,17 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql-kafka-0-10_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
Ok, thank you for reviewing the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] ChengJie1053 commented on a diff in pull request #4563: spark etl support kafka
Posted by "ChengJie1053 (via GitHub)" <gi...@apache.org>.
ChengJie1053 commented on code in PR #4563:
URL: https://github.com/apache/linkis/pull/4563#discussion_r1205205537
##########
tool/dependencies/known-dependencies.txt:
##########
@@ -491,7 +491,11 @@ snakeyaml-1.33.jar
snappy-java-1.1.4.jar
snappy-java-1.1.7.7.jar
snappy-java-1.1.8.2.jar
+snappy-java-1.1.8.4.jar
spark-redis_2.12-2.6.0.jar
+spark-sql-kafka-0-10_2.12-3.2.1.jar
+spark-tags_2.12-3.2.1.jar
+spark-token-provider-kafka-0-10_2.12-3.2.1.jar
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong merged pull request #4563: spark etl support kafka
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong merged PR #4563:
URL: https://github.com/apache/linkis/pull/4563
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on a diff in pull request #4563: spark etl support kafka
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on code in PR #4563:
URL: https://github.com/apache/linkis/pull/4563#discussion_r1203388099
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -388,6 +388,17 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql-kafka-0-10_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
There is a kafka package in the Spark jars, which needs to be provided
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -388,6 +388,17 @@
</exclusion>
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql-kafka-0-10_${scala.binary.version}</artifactId>
+ <version>${spark.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-client-runtime</artifactId>
+ </exclusion>
+ </exclusions>
Review Comment:
There is no reference in the code, you can directly delete this dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org