You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/23 09:35:35 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
LuciferYang opened a new pull request, #36636:
URL: https://github.com/apache/spark/pull/36636
### What changes were proposed in this pull request?
This pr is a minor fix: `JavaUtils#deleteRecursivelyUsingJavaIO` method performs multiple file attribute calls, this pr change to use `Files.readAttributes` instead of them.
### Why are the changes needed?
Reduce multiple file attribute calls.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass Github Actions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36636:
URL: https://github.com/apache/spark/pull/36636#issuecomment-1138402259
thanks @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #36636:
URL: https://github.com/apache/spark/pull/36636#issuecomment-1134773728
Looks fine, but can you try to rerun tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36636:
URL: https://github.com/apache/spark/pull/36636#issuecomment-1135303165
[127a515](https://github.com/apache/spark/pull/36636/commits/127a51581da94995b51ece11caf181200416d3bf) merge with master and rerun all tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36636:
URL: https://github.com/apache/spark/pull/36636#issuecomment-1134818967
seems master UTs break
```
2022-05-23T09:48:27.7668925Z [0m[[0m[0minfo[0m] [0m[0m[31m- Hide credentials in show create table *** FAILED *** (43 milliseconds)[0m[0m
2022-05-23T09:48:27.7696056Z [0m[[0m[0minfo[0m] [0m[0m[31m "[0,10000000d5,5420455441455243,62617420454c4241,414e20200a282031,4e4952545320454d,45485420200a2c47,a29544e49204449,726f20474e495355,6568636170612e67,732e6b726170732e,a6362646a2e6c71,20534e4f4954504f,7462642720200a28,203d2027656c6261,45502e5453455427,200a2c27454c504f,6f77737361702720,2a27203d20276472,2a2a2a2a2a2a2a2a,6574636164657228,2720200a2c272964,27203d20276c7275,2a2a2a2a2a2a2a2a,746361646572282a,20200a2c27296465,3d20277265737527,7355747365742720,a29277265]" did not contain "TEST.PEOPLE" (JDBCSuite.scala:1146)[0m[0m
2022-05-23T09:48:27.7697358Z [0m[[0m[0minfo[0m] [0m[0m[31m org.scalatest.exceptions.TestFailedException:[0m[0m
2022-05-23T09:48:27.7697954Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)[0m[0m
2022-05-23T09:48:27.7698605Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)[0m[0m
2022-05-23T09:48:27.7699211Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)[0m[0m
2022-05-23T09:48:27.7699785Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)[0m[0m
2022-05-23T09:48:27.7700358Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.$anonfun$new$117(JDBCSuite.scala:1146)[0m[0m
2022-05-23T09:48:27.7700946Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)[0m[0m
2022-05-23T09:48:27.7701557Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)[0m[0m
2022-05-23T09:48:27.7702126Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)[0m[0m
2022-05-23T09:48:27.7702799Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.$anonfun$new$116(JDBCSuite.scala:1144)[0m[0m
2022-05-23T09:48:27.7703346Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)[0m[0m
2022-05-23T09:48:27.7703870Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1481)[0m[0m
2022-05-23T09:48:27.7704451Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:306)[0m[0m
2022-05-23T09:48:27.7705082Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:304)[0m[0m
2022-05-23T09:48:27.7705692Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.withTable(JDBCSuite.scala:48)[0m[0m
2022-05-23T09:48:27.7706491Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.$anonfun$new$115(JDBCSuite.scala:1131)[0m[0m
2022-05-23T09:48:27.7707032Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)[0m[0m
2022-05-23T09:48:27.7707551Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)[0m[0m
2022-05-23T09:48:27.7708053Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)[0m[0m
2022-05-23T09:48:27.7708544Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)[0m[0m
2022-05-23T09:48:27.7709026Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Transformer.apply(Transformer.scala:22)[0m[0m
2022-05-23T09:48:27.7709522Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Transformer.apply(Transformer.scala:20)[0m[0m
2022-05-23T09:48:27.7710081Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)[0m[0m
2022-05-23T09:48:27.7710740Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:203)[0m[0m
2022-05-23T09:48:27.7711377Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)[0m[0m
2022-05-23T09:48:27.7712015Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:200)[0m[0m
2022-05-23T09:48:27.7712561Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)[0m[0m
2022-05-23T09:48:27.7713121Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:200)[0m[0m
2022-05-23T09:48:27.7713713Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:182)[0m[0m
2022-05-23T09:48:27.7714354Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:64)[0m[0m
2022-05-23T09:48:27.7714968Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)[0m[0m
2022-05-23T09:48:27.7715541Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)[0m[0m
2022-05-23T09:48:27.7716158Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.org$scalatest$BeforeAndAfter$$super$runTest(JDBCSuite.scala:48)[0m[0m
2022-05-23T09:48:27.7716733Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfter.runTest(BeforeAndAfter.scala:213)[0m[0m
2022-05-23T09:48:27.7717270Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfter.runTest$(BeforeAndAfter.scala:203)[0m[0m
2022-05-23T09:48:27.7717813Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.runTest(JDBCSuite.scala:48)[0m[0m
2022-05-23T09:48:27.7718398Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:233)[0m[0m
2022-05-23T09:48:27.7718974Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)[0m[0m
2022-05-23T09:48:27.7719487Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.immutable.List.foreach(List.scala:431)[0m[0m
2022-05-23T09:48:27.7720016Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)[0m[0m
2022-05-23T09:48:27.7720554Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)[0m[0m
2022-05-23T09:48:27.7721076Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)[0m[0m
2022-05-23T09:48:27.7721641Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:233)[0m[0m
2022-05-23T09:48:27.7722297Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:232)[0m[0m
2022-05-23T09:48:27.7722872Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563)[0m[0m
2022-05-23T09:48:27.7723355Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Suite.run(Suite.scala:1112)[0m[0m
2022-05-23T09:48:27.7723787Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.Suite.run$(Suite.scala:1094)[0m[0m
2022-05-23T09:48:27.7734912Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563)[0m[0m
2022-05-23T09:48:27.7735578Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:237)[0m[0m
2022-05-23T09:48:27.7736113Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.SuperEngine.runImpl(Engine.scala:535)[0m[0m
2022-05-23T09:48:27.7736675Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:237)[0m[0m
2022-05-23T09:48:27.7737441Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:236)[0m[0m
2022-05-23T09:48:27.7738075Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:64)[0m[0m
2022-05-23T09:48:27.7738690Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)[0m[0m
2022-05-23T09:48:27.7739261Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)[0m[0m
2022-05-23T09:48:27.7739809Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)[0m[0m
2022-05-23T09:48:27.7740411Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.org$scalatest$BeforeAndAfter$$super$run(JDBCSuite.scala:48)[0m[0m
2022-05-23T09:48:27.7740976Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)[0m[0m
2022-05-23T09:48:27.7741492Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)[0m[0m
2022-05-23T09:48:27.7742012Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.sql.jdbc.JDBCSuite.run(JDBCSuite.scala:48)[0m[0m
2022-05-23T09:48:27.7742583Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)[0m[0m
2022-05-23T09:48:27.7743347Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)[0m[0m
2022-05-23T09:48:27.7743842Z [0m[[0m[0minfo[0m] [0m[0m[31m at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)[0m[0m
2022-05-23T09:48:27.7744321Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.FutureTask.run(FutureTask.java:266)[0m[0m
2022-05-23T09:48:27.7744909Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[0m[0m
2022-05-23T09:48:27.7745525Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[0m[0m
2022-05-23T09:48:27.7746016Z [0m[[0m[0minfo[0m] [0m[0m[31m at java.lang.Thread.run(Thread.java:750)[0m[0m
```
https://github.com/apache/spark/runs/6552100534?check_suite_focus=true
https://github.com/apache/spark/runs/6553260387?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #36636:
URL: https://github.com/apache/spark/pull/36636#issuecomment-1136000004
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36636:
URL: https://github.com/apache/spark/pull/36636#issuecomment-1135504746
> Looks fine, but can you try to rerun tests?
GA passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #36636: [SPARK-39256][CORE] Reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`
URL: https://github.com/apache/spark/pull/36636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org