You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by "GuoPhilipse (via GitHub)" <gi...@apache.org> on 2023/03/05 13:25:50 UTC
[GitHub] [linkis] GuoPhilipse opened a new pull request, #4301: [feat]Support spark3.3+ and spark2.2- compile
GuoPhilipse opened a new pull request, #4301:
URL: https://github.com/apache/linkis/pull/4301
### What is the purpose of the change
1、after spark 3.3.0 JdbcUtils#createConnectionFactory was moved, we can improve code to support spark3.3 +
2、OrcFileFormat may be lack(take before spark2.2.1) ,we can improve code to support lower spark version
3、HiveTableRelation may be lack(take before spark2.2.1) ,we can improve code to support lower spark version
4、querytimeout may be lack(take before spark2.4.0) ,we can improve code to support lower spark version
5、authtoken may be lack(before spark2.4.0) ,we can improve code to support lower spark version
### Related issues/PRs
Related issues: #4298
Related pr: #
### Brief change log
- Define the core abstraction and interfaces of the EngineConn Factory;
- Define the core abstraction and interfaces of Executor Manager.
### Checklist
- [x] I have read the [Contributing Guidelines on pull requests](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#pull-requests).
- [ ] I have explained the need for this PR and the problem it solves
- [ ] I have explained the changes or the new features added to this PR
- [ ] I have added tests corresponding to this change
- [ ] I have updated the documentation to reflect this change
- [ ] I have verified that this change is backward compatible (If not, please discuss on the [Linkis mailing list](https://linkis.apache.org/community/how-to-subscribe) first)
- [ ] **If this is a code change**: I have written unit tests to fully verify the new behavior.
<!--
Note
1. Mark the PR title as `[WIP] title` until it's ready to be reviewed.
如果PR还未准备好被review，请在标题上添加[WIP]标识(WIP work in progress)
2. Always add/update tests for any changes unless you have a good reason.
除非您有充分的理由，否则任何修改都需要添加/更新测试
3. Always update the documentation to reflect the changes made in the PR.
始终更新文档以反映 PR 中所做的更改
4. After the PR is submitted, please pay attention to the execution result of git action check.
If there is any failure, please adjust it in time
PR提交后，请关注git action check 执行结果，关键的check失败时，请及时修正
5. Before the pr is merged, if the commit is missing, you can continue to commit the code
在未合并前，如果提交有遗漏，您可以继续提交代码
6. After you submit PR, you can add assistant WeChat, the WeChat QR code is
https://user-images.githubusercontent.com/7869972/176336986-d6b9be8f-d1d3-45f1-aa45-8e6adf5dd244.png
您提交pr后，可以添加助手微信，微信二维码为
https://user-images.githubusercontent.com/7869972/176336986-d6b9be8f-d1d3-45f1-aa45-8e6adf5dd244.png
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on a diff in pull request #4301: [feat]Support spark3.3+ and spark2.2- compile
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on code in PR #4301:
URL: https://github.com/apache/linkis/pull/4301#discussion_r1125868738
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -187,12 +187,21 @@
<artifactId>linkis-rpc</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>net.sf.py4j</groupId>
+ <artifactId>py4j</artifactId>
Review Comment:
Here py4j is compatible and should not be introduced separately, because spark jars already have this dependency, which will cause low version conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] peacewong commented on pull request #4301: [feat]Support spark3.3+ and spark2.2- compile
Posted by "peacewong (via GitHub)" <gi...@apache.org>.
peacewong commented on PR #4301:
URL: https://github.com/apache/linkis/pull/4301#issuecomment-1455402231
@rarexixi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] GuoPhilipse commented on a diff in pull request #4301: [feat]Support spark3.3+ and spark2.2- compile
Posted by "GuoPhilipse (via GitHub)" <gi...@apache.org>.
GuoPhilipse commented on code in PR #4301:
URL: https://github.com/apache/linkis/pull/4301#discussion_r1126601715
##########
linkis-engineconn-plugins/spark/pom.xml:
##########
@@ -187,12 +187,21 @@
<artifactId>linkis-rpc</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>net.sf.py4j</groupId>
+ <artifactId>py4j</artifactId>
Review Comment:
have set the dependency as provided
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] rarexixi merged pull request #4301: [feat]Support spark3.3+ and spark2.2- compile
Posted by "rarexixi (via GitHub)" <gi...@apache.org>.
rarexixi merged PR #4301:
URL: https://github.com/apache/linkis/pull/4301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] rarexixi commented on a diff in pull request #4301: [feat]Support spark3.3+ and spark2.2- compile
Posted by "rarexixi (via GitHub)" <gi...@apache.org>.
rarexixi commented on code in PR #4301:
URL: https://github.com/apache/linkis/pull/4301#discussion_r1125926436
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/datacalc/sink/JdbcSink.scala:
##########
@@ -58,7 +60,7 @@ class JdbcSink extends DataCalcSink[JdbcSinkConfig] with Logging {
.repartition(1)
.foreachPartition((_: Iterator[Row]) => {
val jdbcOptions = new JDBCOptions(options)
- val conn: Connection = JdbcUtils.createConnectionFactory(jdbcOptions)()
+ val conn: Connection = createConnectionFactory(jdbcOptions)()
Review Comment:
Replace with `DriverManager.getConnection(config.getUrl, config.getUser, config.getPassword)` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] rarexixi commented on pull request #4301: [feat]Support spark3.3+ and spark2.2- compile
Posted by "rarexixi (via GitHub)" <gi...@apache.org>.
rarexixi commented on PR #4301:
URL: https://github.com/apache/linkis/pull/4301#issuecomment-1459610266
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [linkis] rarexixi commented on a diff in pull request #4301: [feat]Support spark3.3+ and spark2.2- compile
Posted by "rarexixi (via GitHub)" <gi...@apache.org>.
rarexixi commented on code in PR #4301:
URL: https://github.com/apache/linkis/pull/4301#discussion_r1125959895
##########
linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/datacalc/sink/HiveSink.scala:
##########
@@ -184,17 +184,26 @@ class HiveSink extends DataCalcSink[HiveSinkConfig] with Logging {
logicalRelation.relation match {
case hadoopFsRelation: HadoopFsRelation =>
hadoopFsRelation.fileFormat match {
- case _: org.apache.spark.sql.execution.datasources.orc.OrcFileFormat =>
- fileFormat = FileFormat.ORC
case _: org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat =>
fileFormat = FileFormat.PARQUET
case dataSourceRegister: DataSourceRegister =>
fileFormat = FileFormat.withName(dataSourceRegister.shortName.toUpperCase)
case _ =>
+ val allSubClasses = ClassUtils.reflections.getSubTypesOf(classOf[FileFormat])
+ breakable {
+ allSubClasses.asScala
+ .filter(!ClassUtils.isInterfaceOrAbstract(_))
+ .foreach(subclass => {
+ if (subclass.getSimpleName.equals("OrcFileFormat")) {
+ fileFormat = FileFormat.ORC
+ break()
+ }
+ })
+ }
Review Comment:
Is this right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org