You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2023/09/28 08:50:48 UTC
[GitHub] [pinot] gortiz opened a new issue, #11701: Upgrade to Spark 3.5.0
gortiz opened a new issue, #11701:
URL: https://github.com/apache/pinot/issues/11701
This issue is created to analyze whether we can upgrade our Spark 3.x dependencies to 3.5.0.
Apache Pinot connects to Apache Spark in two different types of modules:
- batch ingestion plugins (modules `pinot-batch-ingestion-spark-3.2` and `pinot-batch-ingestion-spark-2.4`), used to ingest data from Spark.
- connectors (modules `pinot-spark-3-connector` and `pinot-spark-2-connector`), used to use Spark to query Pinot.
Note that we need to support these two major Spark versions for compatibility reasons.
As explained in https://github.com/apache/pinot/issues/11656, one of the reasons we cannot officially support Java 21 is that Spark 3.x do not support Java 21 until version 3.5. It seems there is no issues with Spark 2.x versions.
It is not clear to me exactly which Spark version we are using. As expected, `pinot-batch-ingestion-spark-3.2` uses spark 3.2.x (specifically 3.2.1) while `pinot-spark-3-connector` uses 3.4.0. It also seems that Pinot distribution does not includes the connectors (which makes sense), and the final jars will contain the 3.2.1 version.
Spark versioning semantics are defined [here](https://spark.apache.org/versioning-policy.html). Notice that they do not follow semantic versioning. Specifically:
> An API is any public class or interface exposed in Spark that is not marked as “developer API” or “experimental”. Release A is API compatible with release B if code compiled against release A compiles cleanly against B. Currently, does not guarantee that a compiled application that is linked against version A will link cleanly against version B without re-compiling. Link-level compatibility is something we’ll try to guarantee in future releases.
There is also [a document](https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-32-to-33) where they specify the changes from version to version. We would need to understand the changes from 3.2 to 3.3, 3.3 to 3.4 and finally 3.4 to 3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11701:
URL: https://github.com/apache/pinot/issues/11701#issuecomment-2283409218
> While upgrading Spark 3.3.0 to 3.5.0 facing errors. { using java 17 , scala 2.13} for building spark 3.5.0
step to produce error..
>
> clone spark 3.5.0 from apache spark github
> Go to spark directory and run the command. ----> ./build/mvn -DskipTests clean package
I don't think we compile Spark our self. You may need to go to the Spark github repo to look for support compiling it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11701:
URL: https://github.com/apache/pinot/issues/11701#issuecomment-1746287017
I've modified #11702 as ready to review.
I was able to run all our tests without issues with Spark 3.5 instead of 3.2, but I didn't tried my branch in an actual scenario where spark is being used. Reading Spark documentation, it seems that the changes are backward compatible.
It would be nice to have a review from contributors with experience in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11701:
URL: https://github.com/apache/pinot/issues/11701#issuecomment-1772897972
Mostly curious because don't actually use the connector. But why don't we split the connector modules into spark3.x versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "jadami10 (via GitHub)" <gi...@apache.org>.
jadami10 commented on issue #11701:
URL: https://github.com/apache/pinot/issues/11701#issuecomment-1775290443
> But Spark 3.5 should be backward compatible with 3.2
ah my bad. I should have read the [spark docs](https://spark.apache.org/versioning-policy.html) first. It does seem they expect all the same major versions to be compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "indreshverma (via GitHub)" <gi...@apache.org>.
indreshverma commented on issue #11701:
URL: https://github.com/apache/pinot/issues/11701#issuecomment-2267634877
While upgrading Spark 3.3.0 to 3.5.0 facing errors. { using java 17 , scala 2.13} for building spark 3.5.0
step to produce error..
1. clone spark 3.5.0 from apache spark github
2. Go to spark directory and run the command. ----> ./build/mvn -DskipTests clean package
Tried with diff versions of scala also same issue.
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.8.0:compile (scala-compile-first) on project spark-core_2.13: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.8.0:compile failed: org.apache.commons.exec.ExecuteException.
Please help me on this, i'm stuck here, did research , did googling all but could not solve it .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11701:
URL: https://github.com/apache/pinot/issues/11701#issuecomment-1774677528
You mean having one 3.2 and one 3.5 project? We don't usually do that. I mean, imagine we needed to create one single plugin for each technology and each version. It would be very difficult to maintain!
For some wildly used technologies like Kafka and Spark we maintain several backward incompatible projects. For example we have one plugin for kafka 0.9 and another for kafka 2.0. We also had a spark 2.4 and 3.2, so users that use the incompatible 2.x and 3.x versions can actually deploy the plugin they need.
But Spark 3.5 should be backward compatible with 3.2, so we don't see a reason to have to maintain 3 different spark plugins (2.4, 3.2 and 3.5). Therefore we decided to rename 3.2 as 3.x and upgrade spark client version from 3.2 to 3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11701:
URL: https://github.com/apache/pinot/issues/11701#issuecomment-1767796587
We can consider this done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Upgrade to Spark 3.5.0 [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz closed issue #11701: Upgrade to Spark 3.5.0
URL: https://github.com/apache/pinot/issues/11701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org