You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "liuxiaocs7 (via GitHub)" <gi...@apache.org> on 2024/04/07 14:54:15 UTC
[PR] Drop support for Spark 3.1 [kyuubi]
liuxiaocs7 opened a new pull request, #6273:
URL: https://github.com/apache/kyuubi/pull/6273
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #6250
## Describe Your Solution ๐ง
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist ๐
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1555155472
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala:
##########
@@ -76,27 +75,13 @@ class FetchOrcStatement(spark: SparkSession) {
private def getOrcDeserializer(orcSchema: StructType, colId: Array[Int]): OrcDeserializer = {
try {
- if (SPARK_ENGINE_RUNTIME_VERSION >= "3.2") {
- // SPARK-34535 changed the constructor signature of OrcDeserializer
- DynConstructors.builder()
- .impl(classOf[OrcDeserializer], classOf[StructType], classOf[Array[Int]])
- .build[OrcDeserializer]()
- .newInstance(
- orcSchema,
- colId)
- } else {
- DynConstructors.builder()
- .impl(
- classOf[OrcDeserializer],
- classOf[StructType],
- classOf[StructType],
- classOf[Array[Int]])
- .build[OrcDeserializer]()
- .newInstance(
- new StructType,
- orcSchema,
- colId)
- }
+ // SPARK-34535 changed the constructor signature of OrcDeserializer
+ DynConstructors.builder()
Review Comment:
we don't need reflection then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#issuecomment-2041759323
LGTM if CI pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1555021698
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala:
##########
@@ -197,8 +190,7 @@ object SparkDatasetHelper extends Logging {
private def doCommandResultExec(command: SparkPlan): Array[Array[Byte]] = {
Review Comment:
`doCommandResultExec` and `doLocalTableScan` are similar now, do the common parts need to be extracted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "liuxiaocs7 (via GitHub)" <gi...@apache.org>.
liuxiaocs7 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1555163151
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala:
##########
@@ -76,27 +75,13 @@ class FetchOrcStatement(spark: SparkSession) {
private def getOrcDeserializer(orcSchema: StructType, colId: Array[Int]): OrcDeserializer = {
try {
- if (SPARK_ENGINE_RUNTIME_VERSION >= "3.2") {
- // SPARK-34535 changed the constructor signature of OrcDeserializer
- DynConstructors.builder()
- .impl(classOf[OrcDeserializer], classOf[StructType], classOf[Array[Int]])
- .build[OrcDeserializer]()
- .newInstance(
- orcSchema,
- colId)
- } else {
- DynConstructors.builder()
- .impl(
- classOf[OrcDeserializer],
- classOf[StructType],
- classOf[StructType],
- classOf[Array[Int]])
- .build[OrcDeserializer]()
- .newInstance(
- new StructType,
- orcSchema,
- colId)
- }
+ // SPARK-34535 changed the constructor signature of OrcDeserializer
+ DynConstructors.builder()
Review Comment:
done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1555156507
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala:
##########
@@ -223,30 +210,6 @@ object SparkDatasetHelper extends Logging {
spark.sessionState.conf.sessionLocalTimeZone).toArray
}
- /**
- * This method provides a reflection-based implementation of
- * [[AdaptiveSparkPlanExec.finalPhysicalPlan]] that enables us to adapt to the Spark runtime
- * without patching SPARK-41914.
- *
- * TODO: Once we drop support for Spark 3.1.x, we can directly call
- * [[AdaptiveSparkPlanExec.finalPhysicalPlan]].
- */
- def finalPhysicalPlan(adaptiveSparkPlanExec: AdaptiveSparkPlanExec): SparkPlan = {
- withFinalPlanUpdate(adaptiveSparkPlanExec, identity)
- }
-
- /**
- * A reflection-based implementation of [[AdaptiveSparkPlanExec.withFinalPlanUpdate]].
- */
- private def withFinalPlanUpdate[T](
- adaptiveSparkPlanExec: AdaptiveSparkPlanExec,
- fun: SparkPlan => T): T = {
- val plan = invokeAs[SparkPlan](adaptiveSparkPlanExec, "getFinalPhysicalPlan")
- val result = fun(plan)
- invokeAs[Unit](adaptiveSparkPlanExec, "finalPlanUpdate")
- result
- }
-
/**
* offset support was add since Spark-3.4(set SPARK-28330), to ensure backward compatibility with
Review Comment:
```suggestion
* offset support was add in SPARK-28330(3.4.0), to ensure backward compatibility with
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #6273: [KYUUBI #6250] Drop support for Spark 3.1
URL: https://github.com/apache/kyuubi/pull/6273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#issuecomment-2041501641
thanks for making this PR, I noticed there are some places missed, you can search codebase of `kyuubi-spark-sql-engine` module with keywords "3.2"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#issuecomment-2041527824
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6273?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 58.36%. Comparing base [(`4fcc5c7`)](https://app.codecov.io/gh/apache/kyuubi/commit/4fcc5c72a2d9c9300f452935449900cae19cc8ea?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`7d755a8`)](https://app.codecov.io/gh/apache/kyuubi/pull/6273?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 4 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #6273 +/- ##
============================================
- Coverage 58.37% 58.36% -0.01%
Complexity 24 24
============================================
Files 651 651
Lines 39560 39567 +7
Branches 5447 5451 +4
============================================
+ Hits 23092 23095 +3
- Misses 13982 13996 +14
+ Partials 2486 2476 -10
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6273?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1554998623
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala:
##########
@@ -197,8 +190,7 @@ object SparkDatasetHelper extends Logging {
private def doCommandResultExec(command: SparkPlan): Array[Array[Byte]] = {
Review Comment:
this method could be further changed to
```suggestion
private def doCommandResultExec(commandResult: CommandResultExec): Array[Array[Byte]] = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1554999483
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala:
##########
@@ -197,8 +190,7 @@ object SparkDatasetHelper extends Logging {
private def doCommandResultExec(command: SparkPlan): Array[Array[Byte]] = {
Review Comment:
one more place: SPARK-35798 also applicable for `doLocalTableScan`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1555163472
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/sql/kyuubi/SparkDatasetHelper.scala:
##########
@@ -197,8 +190,7 @@ object SparkDatasetHelper extends Logging {
private def doCommandResultExec(command: SparkPlan): Array[Array[Byte]] = {
Review Comment:
the current state is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#discussion_r1555155472
##########
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/FetchOrcStatement.scala:
##########
@@ -76,27 +75,13 @@ class FetchOrcStatement(spark: SparkSession) {
private def getOrcDeserializer(orcSchema: StructType, colId: Array[Int]): OrcDeserializer = {
try {
- if (SPARK_ENGINE_RUNTIME_VERSION >= "3.2") {
- // SPARK-34535 changed the constructor signature of OrcDeserializer
- DynConstructors.builder()
- .impl(classOf[OrcDeserializer], classOf[StructType], classOf[Array[Int]])
- .build[OrcDeserializer]()
- .newInstance(
- orcSchema,
- colId)
- } else {
- DynConstructors.builder()
- .impl(
- classOf[OrcDeserializer],
- classOf[StructType],
- classOf[StructType],
- classOf[Array[Int]])
- .build[OrcDeserializer]()
- .newInstance(
- new StructType,
- orcSchema,
- colId)
- }
+ // SPARK-34535 changed the constructor signature of OrcDeserializer
+ DynConstructors.builder()
Review Comment:
we don't need reflection then, just inline this method `getOrcDeserializer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #6250] Drop support for Spark 3.1 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6273:
URL: https://github.com/apache/kyuubi/pull/6273#issuecomment-2042548555
Thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org