You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2024/03/26 06:02:20 UTC
[PR] Test scala-maven-plugin 4.9.1 [spark]
LuciferYang opened a new pull request, #45716:
URL: https://github.com/apache/spark/pull/45716
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2029564793
This PR seems to break local environment for maven in IntelliJ. Couple of us are getting this response when trying to run anything using maven, even after rebuilding.
```An exception or error caused a run to abort.
java.lang.ExceptionInInitializerError
at org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTime$.apply(finishAnalysis.scala:111)
at org.apache.spark.sql.catalyst.optimizer.ComputeCurrentTime$.apply(finishAnalysis.scala:108)
at org.apache.spark.sql.catalyst.optimizer.Optimizer$FinishAnalysis$.$anonfun$apply$1(Optimizer.scala:306)
at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
at scala.collection.immutable.List.foldLeft(List.scala:79)
at org.apache.spark.sql.catalyst.optimizer.Optimizer$FinishAnalysis$.apply(Optimizer.scala:306)
at org.apache.spark.sql.catalyst.optimizer.Optimizer$FinishAnalysis$.apply(Optimizer.scala:286)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222)
at scala.collection.immutable.ArraySeq.foldLeft(ArraySeq.scala:222)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:219)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:211)
at scala.collection.immutable.List.foreach(List.scala:334)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:211)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:182)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:89)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:182)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$optimizedPlan$1(QueryExecution.scala:166)
at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:233)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:571)
at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:233)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:918)
at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:232)
at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:162)
at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:158)
at org.apache.spark.sql.execution.QueryExecution.assertOptimized(QueryExecution.scala:176)
at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:196)
at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:193)
at org.apache.spark.sql.execution.QueryExecution.simpleString(QueryExecution.scala:252)
at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$explainString(QueryExecution.scala:298)
at org.apache.spark.sql.execution.QueryExecution.explainString(QueryExecution.scala:266)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:138)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:241)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:116)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:918)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:72)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:196)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:120)
at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:571)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:119)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:109)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:442)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:83)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:442)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:330)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:326)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:34)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:418)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:109)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:94)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:224)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:918)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:92)
at org.apache.spark.sql.Dataset.withPlan(Dataset.scala:4453)
at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3978)
at org.apache.spark.sql.connector.DatasourceV2SQLBase.$anonfun$$init$$1(DatasourceV2SQLBase.scala:47)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.scalatest.BeforeAndAfter.runTest(BeforeAndAfter.scala:210)
at org.scalatest.BeforeAndAfter.runTest$(BeforeAndAfter.scala:203)
at org.apache.spark.sql.CollationSuite.runTest(CollationSuite.scala:37)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:334)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
at org.scalatest.Suite.run(Suite.scala:1114)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:69)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at org.apache.spark.sql.CollationSuite.org$scalatest$BeforeAndAfter$$super$run(CollationSuite.scala:37)
at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
at org.apache.spark.sql.CollationSuite.run(CollationSuite.scala:37)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:334)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.run(Runner.scala:798)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)
Caused by: java.lang.IllegalAccessException: symbolic reference class is not accessible: class sun.util.calendar.ZoneInfo, from interface org.apache.spark.sql.catalyst.util.SparkDateTimeUtils (unnamed module @5cee5251)
at java.base/java.lang.invoke.MemberName.makeAccessException(MemberName.java:894)
at java.base/java.lang.invoke.MethodHandles$Lookup.checkSymbolicClass(MethodHandles.java:3787)
at java.base/java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:3747)
at java.base/java.lang.invoke.MethodHandles$Lookup.findVirtual(MethodHandles.java:2767)
at org.apache.spark.sql.catalyst.util.SparkDateTimeUtils.$init$(SparkDateTimeUtils.scala:206)
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.<clinit>(DateTimeUtils.scala:41)
... 102 more```
I am looking into how to fix this for me locally, but haven't found a solution yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2029587279
This seems to have solved the problem. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542333424
##########
core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:
##########
@@ -68,8 +67,8 @@ private[spark] object SerializationDebugger extends Logging {
}
private[serializer] var enableDebugging: Boolean = {
- !AccessController.doPrivileged(new sun.security.action.GetBooleanAction(
- "sun.io.serialization.extendedDebugInfo")).booleanValue()
+ !sun.security.action.GetBooleanAction
+ .privilegedGetProperty("sun.io.serialization.extendedDebugInfo")
Review Comment:
try fix
```
/home/runner/work/spark/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: object security is not a member of package sun
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2026773657
1. If it's not a good time to do this work now, please let me know and I will close this PR.
2. If the PR needs to be split, such as submitting the code change part first, then changing the compile parameters and upgrading the plugin, please let me know too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542333284
##########
core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:
##########
@@ -68,8 +67,8 @@ private[spark] object SerializationDebugger extends Logging {
}
private[serializer] var enableDebugging: Boolean = {
- !AccessController.doPrivileged(new sun.security.action.GetBooleanAction(
- "sun.io.serialization.extendedDebugInfo")).booleanValue()
+ !sun.security.action.GetBooleanAction
Review Comment:
try fix
```
/home/runner/work/spark/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: object security is not a member of package sun
```
##########
core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:
##########
@@ -68,8 +67,8 @@ private[spark] object SerializationDebugger extends Logging {
}
private[serializer] var enableDebugging: Boolean = {
- !AccessController.doPrivileged(new sun.security.action.GetBooleanAction(
- "sun.io.serialization.extendedDebugInfo")).booleanValue()
+ !sun.security.action.GetBooleanAction
Review Comment:
try fix
```
/home/runner/work/spark/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: object security is not a member of package sun
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542548283
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
@@ -215,8 +224,10 @@ trait SparkDateTimeUtils {
val rebasedDays = rebaseGregorianToJulianDays(days)
val localMillis = Math.multiplyExact(rebasedDays, MILLIS_PER_DAY)
val timeZoneOffset = TimeZone.getDefault match {
- case zoneInfo: ZoneInfo => zoneInfo.getOffsetsByWall(localMillis, null)
- case timeZone: TimeZone => timeZone.getOffset(localMillis - timeZone.getRawOffset)
+ case zoneInfo: TimeZone if zoneInfo.getClass.getName == zoneInfoClassName =>
Review Comment:
for fix:
```
Error: ] /home/runner/work/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:27: object util is not a member of package sun
Error: ] /home/runner/work/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:218: not found: type ZoneInfo
Error: ] /home/runner/work/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:218: value getOffsetsByWall is not a member of java.util.TimeZone
```
Maybe we can just use `val timeZoneOffset = TimeZone.getDefault.getOffset(localMillis)` , but I'm not sure which case can verify the compatibility issue with 2.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2029581332
Could you try the operation above? This should not be a new issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542666338
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/CreateXmlParser.scala:
##########
@@ -75,13 +74,13 @@ object CreateXmlParser extends Serializable {
// a reader with specific encoding.
// The method creates a reader for an array with given encoding and sets size of internal
// decoding buffer according to size of input array.
- private def getStreamDecoder(enc: String, in: Array[Byte], length: Int): StreamDecoder = {
+ private def getStreamDecoder(enc: String, in: Array[Byte], length: Int): Reader = {
val bais = new ByteArrayInputStream(in, 0, length)
val byteChannel = Channels.newChannel(bais)
val decodingBufferSize = Math.min(length, 8192)
val decoder = Charset.forName(enc).newDecoder()
- StreamDecoder.forDecoder(byteChannel, decoder, decodingBufferSize)
+ Channels.newReader(byteChannel, decoder, decodingBufferSize)
Review Comment:
for fix:
```
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/CreateXmlParser.scala:27: object nio is not a member of package sun
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/CreateXmlParser.scala:78: not found: type StreamDecoder
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/CreateXmlParser.scala:84: not found: value StreamDecoder
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/CreateXmlParser.scala:27: Unused import
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2027417034
Could you make the CI happy, @LuciferYang ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test compile scala with `-release 17` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1543077176
##########
core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:
##########
@@ -68,8 +70,13 @@ private[spark] object SerializationDebugger extends Logging {
}
private[serializer] var enableDebugging: Boolean = {
- !AccessController.doPrivileged(new sun.security.action.GetBooleanAction(
- "sun.io.serialization.extendedDebugInfo")).booleanValue()
+ val lookup = MethodHandles.lookup()
+ val clazz = SparkClassUtils.classForName("sun.security.action.GetBooleanAction")
+ val constructor = clazz.getConstructor(classOf[String])
+ val mh = lookup.unreflectConstructor(constructor)
+ val action = mh.invoke("sun.io.serialization.extendedDebugInfo")
+ .asInstanceOf[PrivilegedAction[Boolean]]
+ !AccessController.doPrivileged(action).booleanValue()
Review Comment:
Use MethodHandle to ensure logical consistency. Considering that `sun.security` is not an exported package, you can also directly use `!JBoolean.getBoolean("sun.io.serialization.extendedDebugInfo")`.
##########
core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:
##########
@@ -68,8 +70,13 @@ private[spark] object SerializationDebugger extends Logging {
}
private[serializer] var enableDebugging: Boolean = {
- !AccessController.doPrivileged(new sun.security.action.GetBooleanAction(
- "sun.io.serialization.extendedDebugInfo")).booleanValue()
+ val lookup = MethodHandles.lookup()
+ val clazz = SparkClassUtils.classForName("sun.security.action.GetBooleanAction")
+ val constructor = clazz.getConstructor(classOf[String])
+ val mh = lookup.unreflectConstructor(constructor)
+ val action = mh.invoke("sun.io.serialization.extendedDebugInfo")
+ .asInstanceOf[PrivilegedAction[Boolean]]
+ !AccessController.doPrivileged(action).booleanValue()
Review Comment:
Use MethodHandle to ensure logical consistency. Considering that `sun.security` is not an exported package, we can also directly use `!JBoolean.getBoolean("sun.io.serialization.extendedDebugInfo")`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2029138539
Merged to master. Thank you, @LuciferYang and all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1538639261
##########
.github/workflows/build_and_test.yml:
##########
@@ -844,9 +844,9 @@ jobs:
run: |
export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
export MAVEN_CLI_OPTS="--no-transfer-progress"
- export JAVA_VERSION=${{ matrix.java }}
+ # export JAVA_VERSION=${{ matrix.java }}
# It uses Maven's 'install' intentionally, see https://github.com/apache/spark/pull/26414.
- ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud -Djava.version=${JAVA_VERSION/-ea} install
+ ./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano -Phive -Phive-thriftserver -Phadoop-cloud install
Review Comment:
Mainly used to verify the cross-compilation scenario of Maven, using Java 21 to compile with `-release 17`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #45716: [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target`
URL: https://github.com/apache/spark/pull/45716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2029567736
@mihailom-db Could you provide a specific build command? The maven compilation in the GA test has passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542666094
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala:
##########
@@ -58,13 +57,13 @@ object CreateJacksonParser extends Serializable {
// a reader with specific encoding.
// The method creates a reader for an array with given encoding and sets size of internal
// decoding buffer according to size of input array.
- private def getStreamDecoder(enc: String, in: Array[Byte], length: Int): StreamDecoder = {
+ private def getStreamDecoder(enc: String, in: Array[Byte], length: Int): Reader = {
val bais = new ByteArrayInputStream(in, 0, length)
val byteChannel = Channels.newChannel(bais)
val decodingBufferSize = Math.min(length, 8192)
val decoder = Charset.forName(enc).newDecoder()
- StreamDecoder.forDecoder(byteChannel, decoder, decodingBufferSize)
+ Channels.newReader(byteChannel, decoder, decodingBufferSize)
Review Comment:
for fix:
```
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala:27: object nio is not a member of package sun
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala:61: not found: type StreamDecoder
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala:67: not found: value StreamDecoder
Error: ] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala:27: Unused import
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542858092
##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -17,7 +17,7 @@
package org.apache.spark.deploy.yarn
-import java.io.{FileSystem => _, _}
+import java.io.{File, FileFilter, FileNotFoundException, FileOutputStream, InterruptedIOException, IOException, OutputStreamWriter}
Review Comment:
fix
```
Error: ] /home/runner/work/spark/spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:20: object FileSystem is not a member of package java.io
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2027434639
> Could you make the CI happy, @LuciferYang ?
re-triggered the failed task, let's see if it can pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2029152050
Thanks @dongjoon-hyun ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #45716:
URL: https://github.com/apache/spark/pull/45716#issuecomment-2019810706
4.9.1 not in maven repository, let's wait
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542270189
##########
pom.xml:
##########
@@ -176,7 +176,7 @@
<scala.binary.version>2.13</scala.binary.version>
<scalatest-maven-plugin.version>2.2.0</scalatest-maven-plugin.version>
<!-- don't upgrade scala-maven-plugin to version 4.7.2 or higher, see SPARK-45144 for details -->
- <scala-maven-plugin.version>4.7.1</scala-maven-plugin.version>
+ <scala-maven-plugin.version>4.8.1</scala-maven-plugin.version>
Review Comment:
re-test 4.8.1 first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Test scala-maven-plugin 4.9.1 [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1542333424
##########
core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:
##########
@@ -68,8 +67,8 @@ private[spark] object SerializationDebugger extends Logging {
}
private[serializer] var enableDebugging: Boolean = {
- !AccessController.doPrivileged(new sun.security.action.GetBooleanAction(
- "sun.io.serialization.extendedDebugInfo")).booleanValue()
+ !sun.security.action.GetBooleanAction
+ .privilegedGetProperty("sun.io.serialization.extendedDebugInfo")
Review Comment:
try fix
```
/home/runner/work/spark/spark/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: object security is not a member of package sun
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47645][BUILD][CORE][SQL][YARN] Make Spark build with `-release` instead of `-target` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1546342471
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:
##########
@@ -215,8 +224,10 @@ trait SparkDateTimeUtils {
val rebasedDays = rebaseGregorianToJulianDays(days)
val localMillis = Math.multiplyExact(rebasedDays, MILLIS_PER_DAY)
val timeZoneOffset = TimeZone.getDefault match {
- case zoneInfo: ZoneInfo => zoneInfo.getOffsetsByWall(localMillis, null)
- case timeZone: TimeZone => timeZone.getOffset(localMillis - timeZone.getRawOffset)
+ case zoneInfo: TimeZone if zoneInfo.getClass.getName == zoneInfoClassName =>
Review Comment:
Let me investigate whether there are other open APIs that can be used as a substitute here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP] Make Spark build with `-release` instead of `-target` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #45716:
URL: https://github.com/apache/spark/pull/45716#discussion_r1544136677
##########
pom.xml:
##########
@@ -175,8 +174,7 @@
<scala.version>2.13.13</scala.version>
<scala.binary.version>2.13</scala.binary.version>
<scalatest-maven-plugin.version>2.2.0</scalatest-maven-plugin.version>
- <!-- don't upgrade scala-maven-plugin to version 4.7.2 or higher, see SPARK-45144 for details -->
- <scala-maven-plugin.version>4.7.1</scala-maven-plugin.version>
+ <scala-maven-plugin.version>4.8.1</scala-maven-plugin.version>
Review Comment:
To fix `Error: -release cannot be less than -target`, it seems necessary to upgrade to use 4.8.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org