You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/31 10:50:12 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #37741: [SPARK-40283][INFRA] MiMa updates after 3.3.0 release: bump previousSparkVersion to 3.3.0
LuciferYang opened a new pull request, #37741:
URL: https://github.com/apache/spark/pull/37741
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'core/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37741:
URL: https://github.com/apache/spark/pull/37741#issuecomment-1233598133
> > Maybe, do you think you can clean up more like [SPARK-36004](https://issues.apache.org/jira/browse/SPARK-36004)?
>
> like [f2ed5d8](https://github.com/apache/spark/pull/37741/commits/f2ed5d8118062bb382a1bf8631db3fedad5dfc92)?
Seems failed. Let me check again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960184187
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
Review Comment:
<img width="892" alt="image" src="https://user-images.githubusercontent.com/1475305/187824896-760b26b6-e559-4bbb-a61e-9def53aa2538.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @amaliujia and @cloud-fan for SPARK-39506
- https://github.com/apache/spark/pull/36904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37741: [SPARK-40283][INFRA] Make MiMa check default exclude `private object` and bump `previousSparkVersion` to 3.3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37741:
URL: https://github.com/apache/spark/pull/37741#issuecomment-1235919725
Merged to master for Apache Spark 3.4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r961838473
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc.JdbcDialect as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.JdbcDialect.*"),
Review Comment:
`JdbcDialect` is a developer API and should be tracked. Can we track it explicitly instead of skipping the entire package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960201922
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +133,24 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
- },
-
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
- // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs")
+ }
)
def excludes(version: String) = version match {
case v if v.startsWith("3.4") => v34excludes
case v if v.startsWith("3.3") => v33excludes
Review Comment:
~~`org.apache.spark.sql.catalog.*` also add to default excludes~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960184187
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
Review Comment:
<img width="892" alt="image" src="https://user-images.githubusercontent.com/1475305/187824896-760b26b6-e559-4bbb-a61e-9def53aa2538.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @amaliujia and @cloud-fan for SPARK-39506
- [[SPARK-39506][SQL] Make CacheTable, isCached, UncacheTable, setCurrentCatalog, currentCatalog, listCatalogs 3l namespace compatible](https://github.com/apache/spark/pull/36904#top)
#36904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960206275
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +133,24 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
- },
-
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
- // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs")
+ }
)
def excludes(version: String) = version match {
case v if v.startsWith("3.4") => v34excludes
case v if v.startsWith("3.3") => v33excludes
Review Comment:
`case v if v.startsWith("3.3") => v33excludes` deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960199402
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +133,24 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
- },
-
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
- // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs")
+ }
)
def excludes(version: String) = version match {
case v if v.startsWith("3.4") => v34excludes
case v if v.startsWith("3.3") => v33excludes
Review Comment:
Shall we remove this line too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960185199
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-39704][SQL] Implement createIndex & dropIndex & indexExists in JDBC (H2 dialect)
Review Comment:
cc @panbingkun and @huaxingao for SPARK-39704
- https://github.com/apache/spark/pull/37112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37741: [SPARK-40283][INFRA] Make MiMa check default exclude `private object` and bump `previousSparkVersion` to 3.3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37741:
URL: https://github.com/apache/spark/pull/37741#issuecomment-1235905398
MiMa passed. Merged to master for Apache Spark 3.4.0.
<img width="244" alt="Screen Shot 2022-09-02 at 2 17 35 PM" src="https://user-images.githubusercontent.com/9700541/188236630-547b2c99-574b-4592-88ee-d9f49f32bfa3.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #37741: [SPARK-40283][INFRA] Make MiMa check default exclude `private object` and bump `previousSparkVersion` to 3.3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #37741: [SPARK-40283][INFRA] Make MiMa check default exclude `private object` and bump `previousSparkVersion` to 3.3.0
URL: https://github.com/apache/spark/pull/37741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960245752
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
So we want to skip all `private object`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960182874
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
Review Comment:
<img width="823" alt="image" src="https://user-images.githubusercontent.com/1475305/187824035-1ca93e22-1ac2-4dcb-b38f-f87115948b94.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @vkorukanti and @cloud-fan SPARK-38679:
- [[SPARK-38679][CORE] Expose the number partitions in a stage to TaskContext](https://github.com/apache/spark/pull/35995#top)
#35995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960182874
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
Review Comment:
<img width="823" alt="image" src="https://user-images.githubusercontent.com/1475305/187824035-1ca93e22-1ac2-4dcb-b38f-f87115948b94.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @vkorukanti and @cloud-fan SPARK-38679:
- [[SPARK-38679][CORE] Expose the number partitions in a stage to TaskContext](https://github.com/apache/spark/pull/35995/files#top)
#35995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960139158
##########
project/MimaExcludes.scala:
##########
@@ -124,63 +170,12 @@ object MimaExcludes {
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
Review Comment:
wrong place, should in `v34excludes `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960184187
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
Review Comment:
<img width="892" alt="image" src="https://user-images.githubusercontent.com/1475305/187824896-760b26b6-e559-4bbb-a61e-9def53aa2538.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @amaliujia and @cloud-fan for SPARK-39506
- [[SPARK-39506][SQL] Make CacheTable, isCached, UncacheTable, setCurrentCatalog, currentCatalog, listCatalogs 3l namespace compatible](https://github.com/apache/spark/pull/36904#top)
#36904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960198725
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,26 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-36511][MINOR][SQL] Remove ColumnIOUtil
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.parquet.io.ColumnIOUtil")
)
// Exclude rules for 3.3.x from 3.2.0
- lazy val v33excludes = v32excludes ++ Seq(
+ lazy val v33excludes = defaultExcludes ++ Seq(
Review Comment:
I thought we can remove `v33excludes` completely except `defaultExcludes` because it is a different from 3.2.0. Do we need this when we compare with v3.3.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960210967
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
I have add `org.apache.spark.sql.jdbc.*` into default excludes list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960272640
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
Yes, that's reasonable. Let me check why `GenerateMIMAIgnore` didn't ignore them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960298499
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
@cloud-fan
[ad543c0](https://github.com/apache/spark/pull/37741/commits/ad543c057bed529e36172e1e6e463b9c2c71369d) add `moduleSymbol.isPrivate` condition to `directlyPrivateSpark` check, then `private object` will ignore as default.
But we still need explicitly add `ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.JdbcDialect.*")` due to JdbcDialect is a `abstract class` now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960184187
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
Review Comment:
![Uploading image.png…]()
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @amaliujia and @cloud-fan for SPARK-39506
- [[SPARK-39506][SQL] Make CacheTable, isCached, UncacheTable, setCurrentCatalog, currentCatalog, listCatalogs 3l namespace compatible](https://github.com/apache/spark/pull/36904#top)
#36904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960182874
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
Review Comment:
<img width="823" alt="image" src="https://user-images.githubusercontent.com/1475305/187824035-1ca93e22-1ac2-4dcb-b38f-f87115948b94.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @vkorukanti and @cloud-fan SPARK-38679:
- [SPARK-38679][CORE] Expose the number partitions in a stage to TaskContext#35995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960229821
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
@cloud-fan Are all `XXDialect extends JdbcDialect ` identified as `@Private`? Let me test it first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960190241
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-39704][SQL] Implement createIndex & dropIndex & indexExists in JDBC (H2 dialect)
Review Comment:
[c2d9f37](https://github.com/apache/spark/pull/37741/commits/c2d9f3724914eb1706db010ad7daa91643f8ef28) add `org.apache.spark.sql.jdbc.*` as default excludes, is this ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960205832
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +133,24 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
- },
-
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
- // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs")
+ }
)
def excludes(version: String) = version match {
case v if v.startsWith("3.4") => v34excludes
case v if v.startsWith("3.3") => v33excludes
Review Comment:
I've got the wrong idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960186203
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-39704][SQL] Implement createIndex & dropIndex & indexExists in JDBC (H2 dialect)
Review Comment:
they should not be checked by mima... Does mima have a list to mark private APIs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960213062
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,26 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-36511][MINOR][SQL] Remove ColumnIOUtil
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.parquet.io.ColumnIOUtil")
)
// Exclude rules for 3.3.x from 3.2.0
- lazy val v33excludes = v32excludes ++ Seq(
+ lazy val v33excludes = defaultExcludes ++ Seq(
Review Comment:
`v33excludes ` should not be needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
wangyum commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r959648298
##########
project/MimaBuild.scala:
##########
@@ -86,7 +86,7 @@ object MimaBuild {
def mimaSettings(sparkHome: File, projectRef: ProjectRef): Seq[Setting[_]] = {
val organization = "org.apache.spark"
- val previousSparkVersion = "3.2.0"
+ val previousSparkVersion = "3.3.0"
Review Comment:
Please fix the PR description.
> Bump MiMa's previousSparkVersion to 3.2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r959650281
##########
project/MimaBuild.scala:
##########
@@ -86,7 +86,7 @@ object MimaBuild {
def mimaSettings(sparkHome: File, projectRef: ProjectRef): Seq[Setting[_]] = {
val organization = "org.apache.spark"
- val previousSparkVersion = "3.2.0"
+ val previousSparkVersion = "3.3.0"
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960201922
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +133,24 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
- },
-
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
- // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs")
+ }
)
def excludes(version: String) = version match {
case v if v.startsWith("3.4") => v34excludes
case v if v.startsWith("3.3") => v33excludes
Review Comment:
`org.apache.spark.sql.catalog.*` also default excludes
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +133,24 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
- },
-
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
- // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs")
+ }
)
def excludes(version: String) = version match {
case v if v.startsWith("3.4") => v34excludes
case v if v.startsWith("3.3") => v33excludes
Review Comment:
`org.apache.spark.sql.catalog.*` also add to default excludes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37741:
URL: https://github.com/apache/spark/pull/37741#issuecomment-1232891257
cc @wangyum @HyukjinKwon @srowen @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960186185
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-39704][SQL] Implement createIndex & dropIndex & indexExists in JDBC (H2 dialect)
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DB2Dialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DB2Dialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DB2Dialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DerbyDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DerbyDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DerbyDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.JdbcDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.JdbcDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.JdbcDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MsSqlServerDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MsSqlServerDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MsSqlServerDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MySQLDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MySQLDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MySQLDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.NoopDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.NoopDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.NoopDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.OracleDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.OracleDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.OracleDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.PostgresDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.PostgresDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.PostgresDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.TeradataDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.TeradataDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.TeradataDialect.indexExists"),
+
+ // [SPARK-39759][SQL] Implement listIndexes in JDBC (H2 dialect)
Review Comment:
cc @panbingkun and @huaxingao for SPARK-39759
- https://github.com/apache/spark/pull/37172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960247865
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
yea, mima is for checking binary compatibility and why do we care about private APIs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960197251
##########
project/MimaExcludes.scala:
##########
@@ -102,8 +157,8 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI")
)
- // Exclude rules for 3.2.x from 3.1.1
- lazy val v32excludes = Seq(
+ // Defulat exclude rules
+ lazy val defaultExcludes = Seq(
Review Comment:
Looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960227179
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
ignore the class with `org.apache.spark.annotation.Private` ? The code should be feasible, what do you think about this @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960330730
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
And `@DeveloperApi(for example JdbcDialect)` can not be ignore as default due to https://github.com/apache/spark/pull/11751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960202573
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,26 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-36511][MINOR][SQL] Remove ColumnIOUtil
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.parquet.io.ColumnIOUtil")
)
// Exclude rules for 3.3.x from 3.2.0
- lazy val v33excludes = v32excludes ++ Seq(
+ lazy val v33excludes = defaultExcludes ++ Seq(
Review Comment:
Do you mean `val v34excludes = defaultExcludes ++ Seq(...)` directly?
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,26 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-36511][MINOR][SQL] Remove ColumnIOUtil
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.parquet.io.ColumnIOUtil")
)
// Exclude rules for 3.3.x from 3.2.0
- lazy val v33excludes = v32excludes ++ Seq(
+ lazy val v33excludes = defaultExcludes ++ Seq(
Review Comment:
Do you mean `val v34excludes = defaultExcludes ++ Seq(...)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960244646
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
they are `private object`...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960187731
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-39704][SQL] Implement createIndex & dropIndex & indexExists in JDBC (H2 dialect)
Review Comment:
Sure? we can add `org.apache.spark.sql.jdbc.*` as defaultExcludes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960185778
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
Review Comment:
cc @anchovYu and @MaxGekk for SPARK-38929
- https://github.com/apache/spark/pull/36241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960209742
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
We do have developer APIs in this package (e.g. `JdbcDialect`), but dialect implementations are private (e.g. `private object DB2Dialect extends JdbcDialect`). I have no idea why mima still checks these dialect implementations...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r961866213
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc.JdbcDialect as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.JdbcDialect.*"),
Review Comment:
[34fb8c4](https://github.com/apache/spark/pull/37741/commits/34fb8c4d2e56cb3244ce8c1ffa7cb927fbe4c204) fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960186703
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.currentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.setCurrentCatalog"),
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.sql.catalog.Catalog.listCatalogs"),
+
+ // [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI"),
+ ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI$default$3"),
+
+ // [SPARK-39704][SQL] Implement createIndex & dropIndex & indexExists in JDBC (H2 dialect)
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DB2Dialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DB2Dialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DB2Dialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DerbyDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DerbyDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DerbyDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.JdbcDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.JdbcDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.JdbcDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MsSqlServerDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MsSqlServerDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MsSqlServerDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MySQLDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MySQLDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MySQLDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.NoopDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.NoopDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.NoopDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.OracleDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.OracleDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.OracleDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.PostgresDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.PostgresDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.PostgresDialect.indexExists"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.TeradataDialect.createIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.TeradataDialect.dropIndex"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.TeradataDialect.indexExists"),
+
+ // [SPARK-39759][SQL] Implement listIndexes in JDBC (H2 dialect)
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DB2Dialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.DerbyDialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.JdbcDialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MsSqlServerDialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.MySQLDialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.NoopDialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.OracleDialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.PostgresDialect.listIndexes"),
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.jdbc.TeradataDialect.listIndexes"),
+
+ // [SPARK-36511][MINOR][SQL] Remove ColumnIOUtil
+ ProblemFilters.exclude[MissingClassProblem]("org.apache.parquet.io.ColumnIOUtil")
Review Comment:
cc @panbingkun and @srowen for SPARK-36511
- https://github.com/apache/spark/pull/37529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960184187
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
+
+ // [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
Review Comment:
<img width="892" alt="image" src="https://user-images.githubusercontent.com/1475305/187824896-760b26b6-e559-4bbb-a61e-9def53aa2538.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @amaliujia and @cloud-fan for SPARK-39506
- [[SPARK-39506][SQL] Make CacheTable, isCached, UncacheTable, setCurrentCatalog, currentCatalog, listCatalogs 3l namespace compatible](https://github.com/apache/spark/pull/36904#top)
#36904
https://github.com/apache/spark/pull/36904#top
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960182874
##########
project/MimaExcludes.scala:
##########
@@ -62,11 +62,66 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
+
+ // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
Review Comment:
<img width="823" alt="image" src="https://user-images.githubusercontent.com/1475305/187824035-1ca93e22-1ac2-4dcb-b38f-f87115948b94.png">
It was placed in `v32excludes` before, and move to `v34excludes` in this pr.
cc @vkorukanti and @cloud-fan SPARK-38679:
- https://github.com/apache/spark/pull/35995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0 and clean up expired rules
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960214675
##########
project/MimaExcludes.scala:
##########
@@ -118,79 +96,23 @@ object MimaExcludes {
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
+ // SPARK-40283: add jdbc as default excludes
+ ProblemFilters.exclude[Problem]("org.apache.spark.sql.jdbc.*"),
Review Comment:
We have developer APIs there so we can't simply exclude the entire package. If we can't figure out how to fix mima (will it help to add `@Private` for these private classes/objects?), then we have to add exclude rules one by one for each class in `org.apache.spark.sql.jdbc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960136975
##########
project/MimaExcludes.scala:
##########
@@ -124,63 +170,12 @@ object MimaExcludes {
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
},
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
// [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
Review Comment:
wrong place, will correct it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960138266
##########
project/MimaExcludes.scala:
##########
@@ -124,63 +170,12 @@ object MimaExcludes {
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
- // [SPARK-34848][CORE] Add duration to TaskMetricDistributions
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.status.api.v1.TaskMetricDistributions.this"),
-
- // [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions
- // in the REST API call for a specified stage
- ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.status.api.v1.StageData.this"),
-
- // [SPARK-36173][CORE] Support getting CPU number in TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.cpus"),
-
- // [SPARK-38679][CORE] Expose the number of partitions in a stage to TaskContext
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.TaskContext.numPartitions"),
-
- // [SPARK-35896] Include more granular metrics for stateful operators in StreamingQueryProgress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StateOperatorProgress.this"),
-
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
},
- // [SPARK-33808][SQL] DataSource V2: Build logical writes in the optimizer
- ProblemFilters.exclude[MissingClassProblem]("org.apache.spark.sql.connector.write.V1WriteBuilder"),
-
- // [SPARK-33955] Add latest offsets to source progress
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.SourceProgress.this"),
-
- // [SPARK-34862][SQL] Support nested column in ORC vectorized reader
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBoolean"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getByte"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getShort"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getInt"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getLong"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getFloat"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDouble"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getDecimal"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getUTF8String"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getBinary"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getArray"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getMap"),
- ProblemFilters.exclude[DirectAbstractMethodProblem]("org.apache.spark.sql.vectorized.ColumnVector.getChild"),
-
- // [SPARK-35135][CORE] Turn WritablePartitionedIterator from trait into a default implementation class
- ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.util.collection.WritablePartitionedIterator"),
-
- // [SPARK-35757][CORE] Add bitwise AND operation and functionality for intersecting bloom filters
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.util.sketch.BloomFilter.intersectInPlace"),
-
- // [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
- ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
- ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions"),
-
// [SPARK-39506] In terms of 3 layer namespace effort, add currentCatalog, setCurrentCatalog and listCatalogs API to Catalog interface
Review Comment:
should be `v34excludes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r960187254
##########
project/MimaExcludes.scala:
##########
@@ -102,8 +157,8 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.types.Decimal.fromStringANSI")
)
- // Exclude rules for 3.2.x from 3.1.1
- lazy val v32excludes = Seq(
+ // Defulat exclude rules
+ lazy val defaultExcludes = Seq(
Review Comment:
@dongjoon-hyun
I keep
```
// Spark Internals
ProblemFilters.exclude[Problem]("org.apache.spark.rpc.*"),
ProblemFilters.exclude[Problem]("org.spark-project.jetty.*"),
ProblemFilters.exclude[Problem]("org.spark_project.jetty.*"),
ProblemFilters.exclude[Problem]("org.sparkproject.jetty.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.unused.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.unsafe.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.memory.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.util.collection.unsafe.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.catalyst.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.execution.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.internal.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.errors.*"),
// DSv2 catalog and expression APIs are unstable yet. We should enable this back.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.catalog.*"),
ProblemFilters.exclude[Problem]("org.apache.spark.sql.connector.expressions.*"),
// Avro source implementation is internal.
ProblemFilters.exclude[Problem]("org.apache.spark.sql.v2.avro.*"),
(problem: Problem) => problem match {
case MissingClassProblem(cls) => !cls.fullName.startsWith("org.sparkproject.jpmml") &&
!cls.fullName.startsWith("org.sparkproject.dmg.pmml")
case _ => true
}
```
as `defaultExcludes`, is that right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #37741: [WIP][SPARK-40283][INFRA] Update MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #37741:
URL: https://github.com/apache/spark/pull/37741#discussion_r959456336
##########
project/MimaExcludes.scala:
##########
@@ -62,7 +62,52 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy"),
ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.copy$default$2"),
ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.this"),
- ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply")
+ ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.deploy.DeployMessages#RequestExecutors.apply"),
Review Comment:
Pass GA first, add comments later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37741: [SPARK-40283][INFRA] Bump MiMa's previousSparkVersion to 3.3.0
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37741:
URL: https://github.com/apache/spark/pull/37741#issuecomment-1233590169
> f2ed5d8
like [f2ed5d8](https://github.com/apache/spark/pull/37741/commits/f2ed5d8118062bb382a1bf8631db3fedad5dfc92)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org