You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2025/12/29 16:20:31 UTC
[PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
baibaichen opened a new pull request, #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342
# [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE
Fixes #11341
## What changes were proposed in this pull request?
This PR includes several improvements for Scala 2.13 compatibility and code modernization:
| Change | Description | Files |
|--------|-------------|-------|
| Update Java compiler release version | Changed from hardcoded `8` to `${java.version}` in the Scala 2.13 profiler to align with `maven.compiler.target` | `pom.xml` |
| Replace deprecated file reading method | Migrated from deprecated `fileToString` to `Files.readString` in `GlutenSQLQueryTestSuite` (see https://github.com/apache/spark/pull/51911) | `gluten-ut/spark40/src/test/scala/org/apache/spark/GlutenSQLQueryTestSuite.scala` |
| Fix symbol literal deprecation | Refactored to use `Symbol()` for column references to fix compilation error in Scala 2.13 with IntelliJ compiler | `GlutenDataSourceV2Suite.scala`, `ClickHouseAdaptiveQueryExecSuite.scala`, `VeloxAdaptiveQueryExecSuite.scala` |
| Suppress warnings | Added suppressions for `ContextAwareIterator` initialization and `ContentFile::path` warnings in Scala 2.13 | `ColumnarArrowEvalPythonExec.scala`, `GlutenIcebergSourceUtil.scala` |
| Remove obsolete suppressions | Removed suppression for lint-multiarg-infix warnings from pom.xml (see https://github.com/apache/spark/pull/43332) | `pom.xml` |
## Why are the changes needed?
These changes are necessary to:
- Maintain compatibility with Scala 2.13 and IntelliJ compiler to accelerate development speed
- Remove deprecated code that was removed from Spark codebase
- Improve build configuration consistency
## How was this patch tested?
- Existing test suites
- Manual testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652117895
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -50,6 +51,7 @@ object GlutenIcebergSourceUtil {
}
}
+ @nowarn
Review Comment:
> As the comment indicates, please use location
>
> ```
> /**
> * Returns fully qualified path to the file, suitable for constructing a Hadoop Path.
> *
> * @deprecated since 1.7.0, will be removed in 2.0.0; use {@link #location()} instead.
> */
> @Deprecated
> CharSequence path();
>
> /** Return the fully qualified path to the file. */
> default String location() {
> return path().toString();
> }
> ```
No, `location` doesn't work in profiles in Spark 3.3 and earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652052155
##########
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/adaptive/velox/VeloxAdaptiveQueryExecSuite.scala:
##########
@@ -986,15 +987,16 @@ class VeloxAdaptiveQueryExecSuite extends AdaptiveQueryExecSuite with GlutenSQLT
// Repartition with non-default partition num specified.
checkSMJ(
- df.repartition(4, 'b),
+ df.repartition(4, Symbol("b")),
Review Comment:
Can we use col("b")? The warning may be caused by b with single quote
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652053039
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -50,6 +51,7 @@ object GlutenIcebergSourceUtil {
}
}
+ @nowarn
Review Comment:
What's the warning message? Can we fix the warning?
##########
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/adaptive/velox/VeloxAdaptiveQueryExecSuite.scala:
##########
@@ -986,15 +987,16 @@ class VeloxAdaptiveQueryExecSuite extends AdaptiveQueryExecSuite with GlutenSQLT
// Repartition with non-default partition num specified.
checkSMJ(
- df.repartition(4, 'b),
+ df.repartition(4, Symbol("b")),
Review Comment:
Can we use "b"? The warning may be caused by b with single quote
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#issuecomment-3699270030
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652097773
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -50,6 +51,7 @@ object GlutenIcebergSourceUtil {
}
}
+ @nowarn
Review Comment:
There is shim for iceberg like this PR, https://github.com/apache/incubator-gluten/pull/11146, we can accept current workaround, and fix it after https://github.com/apache/incubator-gluten/pull/11146 merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#issuecomment-3696952472
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652072066
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -50,6 +51,7 @@ object GlutenIcebergSourceUtil {
}
}
+ @nowarn
Review Comment:
`org.apache.iceberg.ContentFile::path()` is marked as `deprecated`. Without introducing a shim, it's hard to fix properly, so we'll suppress the warning for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#issuecomment-3698138099
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#issuecomment-3698453068
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652054745
##########
gluten-ut/spark40/src/test/scala/org/apache/spark/sql/execution/adaptive/velox/VeloxAdaptiveQueryExecSuite.scala:
##########
@@ -986,15 +987,16 @@ class VeloxAdaptiveQueryExecSuite extends AdaptiveQueryExecSuite with GlutenSQLT
// Repartition with non-default partition num specified.
checkSMJ(
- df.repartition(4, 'b),
+ df.repartition(4, Symbol("b")),
Review Comment:
Syntax | Valid | Notes
-- | -- | --
df.repartition(4, 'b) | ✅ | Old-style, works
df.repartition(4, Symbol("b")) | ✅ | Same as above
df.repartition(4, col("b")) | ✅ | Recommended
df.repartition(4, 'b), | ❌ | Syntax error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652072066
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -50,6 +51,7 @@ object GlutenIcebergSourceUtil {
}
}
+ @nowarn
Review Comment:
`org.apache.iceberg.ContentFile::path()` is marked as `deprecated`. Without introducing a shim, it's hard to fix properly, so we'll suppress the warning for now.
```scala
asFileScanTask(tasks.toList).foreach {
task =>
paths.add(
BackendsApiManager.getTransformerApiInstance
.encodeFilePathIfNeed(task.file().path().toString)) // => deprecated
```
Additionally, `@nowarn` can only be added before method names; it cannot be applied before individual statements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342#discussion_r2652094097
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -50,6 +51,7 @@ object GlutenIcebergSourceUtil {
}
}
+ @nowarn
Review Comment:
As the comment indicates, please use location
```
/**
* Returns fully qualified path to the file, suitable for constructing a Hadoop Path.
*
* @deprecated since 1.7.0, will be removed in 2.0.0; use {@link #location()} instead.
*/
@Deprecated
CharSequence path();
/** Return the fully qualified path to the file. */
default String location() {
return path().toString();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11341][CORE] Support Scala 2.13 with IntelliJ IDE [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen merged PR #11342:
URL: https://github.com/apache/incubator-gluten/pull/11342
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org