You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/08/30 18:05:15 UTC
[GitHub] [incubator-seatunnel] zhaomin1423 opened a new pull request, #2574: Support seatunnel-translation-spark-3.3
zhaomin1423 opened a new pull request, #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] JinJiDeJinMu commented on pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by GitBox <gi...@apache.org>.
JinJiDeJinMu commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1358811687
how to build new_spark_3.3 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1240471608
Hi, please reslove confilct then I will merge the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] JinJiDeJinMu commented on pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by GitBox <gi...@apache.org>.
JinJiDeJinMu commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1359070747
@Hisoka-X I want to join in the translation of Spark3. x based on v2, What can I do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964639910
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
<img width="920" alt="image" src="https://user-images.githubusercontent.com/49054376/188850114-dfbc4b61-1bda-468f-8eee-d5e8601472d9.png">
Here,can put it to the dir. The e2e is using by the style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964635922
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
How to choose? Seem like you should add jar choose logic in seatunnel-spark-starter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1238852981
> Hi, Please fix UT
The fail is related with flink. The pr don't update, can you help me rerun it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] JinJiDeJinMu commented on pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by GitBox <gi...@apache.org>.
JinJiDeJinMu commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1358930803
> > > <img alt="截屏2022-12-20 14 52 40" width="694" src="https://user-images.githubusercontent.com/33084817/208602197-40d5a60c-e691-4d9d-ae3f-71cd9928c630.png">
> > > Who can help me look at this problem
> >
> >
> > for test spark3 by use spark3_sink build
>
> @JinJiDeJinMu Hi, Support Spark3 not finished at now. But @nishuihanqiu have a branch for spark3 in his repository. [#875 (comment)](https://github.com/apache/incubator-seatunnel/issues/875#issuecomment-1312735235)
@Hisoka-X thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1238747839
Please help me review it. I only add e2e in spark 3.3 for FakeSourceToConsoleIT, will add more after merge it, because the module change frequently, I fixed conflicts repeatedly. @Hisoka-X
The spark scope is changed to provided in example module, because the example don't add the release package, the compile scope need handle licenses, it's necessary, spark has a lot of dependencies. @ashulin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1240582683
> > Sorry, it seems that this approval disappeared due to force push.
>
> Never mind, after ci passed, I will approval again.
please help me rerun failed ci, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] JinJiDeJinMu commented on pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by GitBox <gi...@apache.org>.
JinJiDeJinMu commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1358915006
> <img alt="截屏2022-12-20 14 52 40" width="694" src="https://user-images.githubusercontent.com/33084817/208602197-40d5a60c-e691-4d9d-ae3f-71cd9928c630.png">
>
> Who can help me look at this problem
for test spark3 by use spark3_sink build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1257870858
Can we support all spark e2e run both 3.3 and 2.4 without configure anything? I find we need write same code twice if we want test job both on 3.3 and 2.4. If we do this, and all e2e passed. I think we can merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964641061
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
Typically, users choose one spark version, so they only use one translation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1239169576
> How user switch spark2.4 and spark3.3 use shell command?
According to spark client version, choose the jar. But, we can handle it later, the pr only add spark3.3 translation. There are frequent conflicts if we add more content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1239048792
How user switch spark2.4 and spark3.3 use shell command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964530314
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
How user switch seatunnel-transform-spark(2.4、3.3) list?
##########
seatunnel-translation/seatunnel-translation-spark/pom.xml:
##########
@@ -26,13 +26,15 @@
<artifactId>seatunnel-translation-spark</artifactId>
<packaging>pom</packaging>
<modules>
- <module>seatunnel-translation-spark-2.4</module>
<module>seatunnel-translation-spark-common</module>
+ <module>seatunnel-translation-spark-2.4</module>
+ <module>seatunnel-translation-spark-3.3</module>
+ <module>seatunnel-translation-spark-2.4-dist</module>
+ <module>seatunnel-translation-spark-3.3-dist</module>
</modules>
-
+
<dependencyManagement>
<dependencies>
- <!--spark-->
<dependency>
Review Comment:
move to spark-2.4 module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964595616
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
> How user switch seatunnel-transform-spark(2.4、3.3) list?
Our release package include them, user choose the jar according to their spark client version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer closed pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer closed pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
URL: https://github.com/apache/incubator-seatunnel/pull/2574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1232438067
Can you add e2e from spark3 and spark2?
reference:https://github.com/apache/incubator-seatunnel/pull/2499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1237273474
> Please add licenses according to `Dependency licenses action`. you can see https://seatunnel.apache.org/docs/contribution/new-license
Thanks, I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1239173829
> > How user switch spark2.4 and spark3.3 use shell command?
>
> According to spark client version, choose the jar. But, we can handle it later, the pr only add spark3.3 translation. There are frequent conflicts if we add more content.
OK for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1240487131
Sorry, it seems that this approval disappeared due to force push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1260254822
> Can we support all spark e2e run both 3.3 and 2.4 without configure anything? I find we need write same code twice if we want test job both on 3.3 and 2.4. If we do this, and all e2e passed. I think we can merge this PR.
I only add one test for 3.3, I will fix conflicts later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1238830290
Hi, Please fix UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1240499548
> Sorry, it seems that this approval disappeared due to force push.
Never mind, after ci passed, I will approval again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] dik111 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
dik111 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1297929162
spark version :3.3.1
hive version: 3.0.0
mysql versin: 5.7
I had test mysql to hive
it throws some exception:
```
client token: N/A
diagnostics: User class threw exception: java.util.ServiceConfigurationError: org.apache.seatunnel.spark.BaseSparkSink: Provider org.apache.seatunnel.spark.hive.sink.Hive could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.loadPluginInstance(AbstractPluginDiscovery.java:128)
at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:99)
at org.apache.seatunnel.core.spark.config.SparkExecutionContext.lambda$getSinks$2(SparkExecutionContext.java:95)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.seatunnel.core.spark.config.SparkExecutionContext.getSinks(SparkExecutionContext.java:98)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:57)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:40)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
at org.apache.seatunnel.spark.hive.sink.Hive.<init>(Hive.scala:29)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 28 more
ApplicationMaster host: jtbihdp04.sogal.com
ApplicationMaster RPC port: 39125
queue: default
start time: 1667267960414
final status: FAILED
tracking URL: http://jtbihdp13.sogal.com:8088/proxy/application_1667009280185_5845/
user: suofy
22/11/01 10:00:51 ERROR Client: Application diagnostics message: User class threw exception: java.util.ServiceConfigurationError: org.apache.seatunnel.spark.BaseSparkSink: Provider org.apache.seatunnel.spark.hive.sink.Hive could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.loadPluginInstance(AbstractPluginDiscovery.java:128)
at org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:99)
at org.apache.seatunnel.core.spark.config.SparkExecutionContext.lambda$getSinks$2(SparkExecutionContext.java:95)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.seatunnel.core.spark.config.SparkExecutionContext.getSinks(SparkExecutionContext.java:98)
at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:57)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:40)
at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
at org.apache.seatunnel.spark.hive.sink.Hive.<init>(Hive.scala:29)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 21 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 28 more
Exception in thread "main" org.apache.spark.SparkException: Application application_1667009280185_5845 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1342)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1358917701
> > <img alt="截屏2022-12-20 14 52 40" width="694" src="https://user-images.githubusercontent.com/33084817/208602197-40d5a60c-e691-4d9d-ae3f-71cd9928c630.png">
> > Who can help me look at this problem
>
> for test spark3 by use spark3_sink build
@JinJiDeJinMu Hi, Support Spark3 not finished at now. But @nishuihanqiu have a branch for spark3 in his repository. https://github.com/apache/incubator-seatunnel/issues/875#issuecomment-1312735235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1359078815
> @Hisoka-X I want to join in the translation of Spark3. x based on v2, What can I do?
@JinJiDeJinMu Wonderful! You can continue do your work with this PR. Fix conflict and do some test, if have some problem then fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964631965
##########
seatunnel-translation/seatunnel-translation-spark/pom.xml:
##########
@@ -26,13 +26,15 @@
<artifactId>seatunnel-translation-spark</artifactId>
<packaging>pom</packaging>
<modules>
- <module>seatunnel-translation-spark-2.4</module>
<module>seatunnel-translation-spark-common</module>
+ <module>seatunnel-translation-spark-2.4</module>
+ <module>seatunnel-translation-spark-3.3</module>
+ <module>seatunnel-translation-spark-2.4-dist</module>
+ <module>seatunnel-translation-spark-3.3-dist</module>
</modules>
-
+
<dependencyManagement>
<dependencies>
- <!--spark-->
<dependency>
Review Comment:
Because seatunnel-translation-spark-common depend on spark, if move it, the common module need add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964654816
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
Good idea. The arg can control to choose which plugin dir. We can do it in a new pr. How about you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964647675
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
I prefer a more automated way, such as by judging the version of SPARK_HOME or using SPARK2 by default, you can switch versions by adding parameters to the shell script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1238902034
<img width="1375" alt="image" src="https://user-images.githubusercontent.com/49054376/188790426-ca1a65de-702a-41aa-83e1-8fdd3a923fb2.png">
The fail is strange, I don't how to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1240481822
> Hi, please reslove confilct then I will merge the code. Thanks!
done, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964668110
##########
seatunnel-core/seatunnel-spark-starter/pom.xml:
##########
@@ -74,7 +74,7 @@
<dependency>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-translation-spark-2.4</artifactId>
+ <artifactId>seatunnel-translation-spark-common</artifactId>
Review Comment:
Ok for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on a diff in pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on code in PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#discussion_r964643389
##########
seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-spark-e2e-base/src/test/java/org/apache/seatunnel/e2e/spark/Spark33Container.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.e2e.spark;
+
+import java.nio.file.Paths;
+
+/**
+ * This class is the base class of SparkEnvironment test. The before method will create a Spark master, and after method will close the Spark master.
+ * You can use {@link AbstractSparkContainer#executeSeaTunnelSparkJob} to submit a seatunnel conf and a seatunnel spark job.
+ */
+public abstract class Spark33Container extends AbstractSparkContainer {
+
+ private final String translationJarName = "seatunnel-translation-spark-3.3-dist.jar";
+
+ @Override
+ protected String getTranslationJarPath() {
+ return Paths.get(PROJECT_ROOT_PATH.toString(),
+ "seatunnel-translation", "seatunnel-translation-spark",
+ "seatunnel-translation-spark-3.3-dist", "target",
+ translationJarName).toString();
+ }
+
+ @Override
+ protected String getTranslationJarTargetPath() {
+ return Paths.get(SEATUNNEL_HOME, "plugins", "translation-spark-3.3", "lib",
Review Comment:
cc @hailin0 @Hisoka-X.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1241445362
Could you help me review it, merging is blocking, the conflicts are frequent. @ashulin @CalvinKirs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhaomin1423 commented on pull request #2574: Support seatunnel-translation-spark-3.3
Posted by GitBox <gi...@apache.org>.
zhaomin1423 commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1237273088
> Can you add e2e from spark3 and spark2? reference:#2499
ok, I will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] JinJiDeJinMu commented on pull request #2574: [Feature][Connector-V2][Translation] Support spark 3.3
Posted by GitBox <gi...@apache.org>.
JinJiDeJinMu commented on PR #2574:
URL: https://github.com/apache/incubator-seatunnel/pull/2574#issuecomment-1358913943
<img width="694" alt="截屏2022-12-20 14 52 40" src="https://user-images.githubusercontent.com/33084817/208602197-40d5a60c-e691-4d9d-ae3f-71cd9928c630.png">
Who can help me look at this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org