You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/22 03:11:58 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request, #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
HeartSaVioR opened a new pull request, #37248:
URL: https://github.com/apache/spark/pull/37248
### What changes were proposed in this pull request?
This PR proposes to effectively revert SPARK-39748 but include the origin stats and constraints instead in LogicalRDD if it comes from DataFrame, to help optimizer figuring out better plan.
### Why are the changes needed?
We figured out several issues from [SPARK-39748](https://issues.apache.org/jira/browse/SPARK-39748):
1. One of major use case for DataFrame.checkpoint is ML, especially "iterative algorithm", which purpose is to "prune" the logical plan. That is against the purpose of including origin logical plan and we have a risk to have nested LogicalRDDs which grows the size of logical plan infinitely.
2. We leverage logical plan to carry over stats, but the correct stats information is in optimized plan.
3. (Not an issue but missing spot) constraints is also something we can carry over.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing and new UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37248:
URL: https://github.com/apache/spark/pull/37248#issuecomment-1196393417
CI build pass. Thanks all for reviewing! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37248:
URL: https://github.com/apache/spark/pull/37248#issuecomment-1192135264
cc. @cloud-fan @juliuszsompolski @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37248:
URL: https://github.com/apache/spark/pull/37248#discussion_r927769131
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala:
##########
@@ -86,19 +86,22 @@ case class ExternalRDDScanExec[T](
/**
* Logical plan node for scanning data from an RDD of InternalRow.
*
- * It is advised to set the field `originLogicalPlan` if the RDD is directly built from DataFrame,
- * as the stat can be inherited from `originLogicalPlan`.
+ * It is advised to set the field `originStats` and `originConstraints` if the RDD is directly
+ * built from DataFrame, so that Spark can make better optimizations.
*/
case class LogicalRDD(
output: Seq[Attribute],
rdd: RDD[InternalRow],
- originLogicalPlan: Option[LogicalPlan] = None,
outputPartitioning: Partitioning = UnknownPartitioning(0),
override val outputOrdering: Seq[SortOrder] = Nil,
- override val isStreaming: Boolean = false)(session: SparkSession)
+ override val isStreaming: Boolean = false)(
+ session: SparkSession,
+ originStats: Option[Statistics] = None,
+ originConstraints: Option[ExpressionSet] = None)
Review Comment:
can we add a comment to explain why they are in the curry constructor? because we don't want other rules to mistakenly transform and rewrite them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37248:
URL: https://github.com/apache/spark/pull/37248#issuecomment-1195436914
I think we can ignore it but mind retriggering one more time please? I made some fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37248:
URL: https://github.com/apache/spark/pull/37248#issuecomment-1193118952
```
2022-07-23T10:08:45.3143193Z [0m[[0m[0minfo[0m] [0m[0mScalaTest[0m
2022-07-23T10:08:45.3164162Z [0m[[0m[0minfo[0m] [0m[0m[36mRun completed in 1 hour, 22 minutes, 26 seconds.[0m[0m
2022-07-23T10:08:45.3164676Z [0m[[0m[0minfo[0m] [0m[0m[36mTotal number of tests run: 8885[0m[0m
2022-07-23T10:08:45.3165119Z [0m[[0m[0minfo[0m] [0m[0m[36mSuites: completed 470, aborted 0[0m[0m
2022-07-23T10:08:45.3165653Z [0m[[0m[0minfo[0m] [0m[0m[36mTests: succeeded 8885, failed 0, canceled 6, ignored 28, pending 0[0m[0m
2022-07-23T10:08:45.3169564Z [0m[[0m[0minfo[0m] [0m[0m[32mAll tests passed.[0m[0m
2022-07-23T10:08:45.3365179Z [0m[[0m[31merror[0m] [0m[0mError: Total 0, Failed 0, Errors 0, Passed 0[0m
2022-07-23T10:08:45.3491396Z [0m[[0m[31merror[0m] [0m[0mError during tests:[0m
2022-07-23T10:08:45.9793527Z [0m[[0m[31merror[0m] [0m[0m Running java with options -classpath /home/runner/work/spark/spark/sql/core/target/scala-2.12/test-classes:/home/runner/work/spark/spark/sql/core/target/scala-2.12/spark-sql_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/common/sketch/target/scala-2.12/spark-sketch_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/common/tags/target/scala-2.12/spark-tags_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/core/target/scala-2.12/spark-core_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/core/target/scala-2.12/test-classes:/home/runner/work/spark/spark/common/network-shuffle/target/scala-2.12/spark-network-shuffle_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/common/network-shuffle/target/scala-2.12/test-classes:/home/runner/work/spark/spark/common/network-common/target/scala-2.12/spark-network-common_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/common/network-common/target/scala-2.12/
test-classes:/home/runner/work/spark/spark/common/tags/target/scala-2.12/test-classes:/home/runner/work/spark/spark/common/unsafe/target/scala-2.12/spark-unsafe_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/launcher/target/scala-2.12/spark-launcher_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/common/kvstore/target/scala-2.12/spark-kvstore_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/launcher/target/scala-2.12/test-classes:/home/runner/work/spark/spark/sql/catalyst/target/scala-2.12/spark-catalyst_2.12-3.4.0-SNAPSHOT.jar:/home/runner/work/spark/spark/sql/catalyst/target/scala-2.12/test-classes:/home/runner/.sbt/boot/scala-2.12.16/lib/scala-library.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/rocksdb/rocksdbjni/7.3.1/rocksdbjni-7.3.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/univocity/univocity-parsers/2.9.1/univocity-parsers-2.9.1.jar:/home/runner/.
cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/orc/orc-core/1.7.5/orc-core-1.7.5.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/orc/orc-mapreduce/1.7.5/orc-mapreduce-1.7.5.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/hive/hive-storage-api/2.7.3/hive-storage-api-2.7.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-column/1.12.3/parquet-column-1.12.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-column/1.12.3/parquet-column-1.12.3-tests.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-encoding/1.12.3/parquet-encoding-1.12.3-tests.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googl
eapis.com/maven2/org/apache/parquet/parquet-common/1.12.3/parquet-common-1.12.3-tests.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-hadoop/1.12.3/parquet-hadoop-1.12.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-servlet/9.4.48.v20220622/jetty-servlet-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.3/jackson-databind-2.13.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/xbean/xbean-asm9-shaded/4.20/xbean-asm9-shaded-4.20.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalacheck/scalacheck_2.12/1.15.4/scalacheck_2.12-1.15.4.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2
/com/h2database/h2/2.1.210/h2-2.1.210.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/mariadb/jdbc/mariadb-java-client/2.7.4/mariadb-java-client-2.7.4.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/ibm/db2/jcc/11.5.6.0/jcc-11.5.6.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/microsoft/sqlserver/mssql-jdbc/9.4.0.jre8/mssql-jdbc-9.4.0.jre8.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/oracle/database/jdbc/ojdbc8/21.3.0.0/ojdbc8-21.3.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-avro/1.12.3/parquet-avro-1.12.3.jar:/home/runner/.cache/coursier/v1/https/maven
-central.storage-download.googleapis.com/maven2/org/mockito/mockito-core/4.5.1/mockito-core-4.5.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-java/3.141.59/selenium-java-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/htmlunit-driver/2.62.0/htmlunit-driver-2.62.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest_2.12/3.3.0-SNAP3/scalatest_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatestplus/scalacheck-1-15_2.12/3.3.0.0-SNAP3/scalacheck-1-15_2.12-3.3.0.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googl
eapis.com/maven2/org/scalatestplus/mockito-4-5_2.12/3.2.12.0/mockito-4-5_2.12-3.2.12.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatestplus/selenium-3-141_2.12/3.3.0.0-SNAP3/selenium-3-141_2.12-3.3.0.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/junit/junit/4.13.2/junit-4.13.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/sbt/junit-interface/0.13.3/junit-interface-0.13.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scala-lang/modules/scala-collection-compat_2.12/2.2.0/scala-collection-compat_2.12-2.2.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/ghik/silencer-lib_2.12.16/1.7.9/silencer-lib_2.12.16-1.7.9.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven
2/org/apache/avro/avro/1.11.0/avro-1.11.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/avro/avro-mapred/1.11.0/avro-mapred-1.11.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/twitter/chill_2.12/0.10.0/chill_2.12-0.10.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/twitter/chill-java/0.10.0/chill-java-0.10.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/jnr/jnr-posix/3.1.15/jnr-posix-3.1.15.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/hadoop/hadoop-client-api/3.3.3/hadoop-client-api-3.3.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.3/hadoop-client-runtime-3.3.3.jar:/home/runner/.cache/coursier/v1/https/mave
n-central.storage-download.googleapis.com/maven2/javax/activation/activation/1.1.1/activation-1.1.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/curator/curator-recipes/2.13.0/curator-recipes-2.13.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/zookeeper/zookeeper/3.6.2/zookeeper-3.6.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-plus/9.4.48.v20220622/jetty-plus-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-security/9.4.48.v20220622/jetty-security-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-util/9.4.48.v20220622/jetty-util-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.g
oogleapis.com/maven2/org/eclipse/jetty/jetty-server/9.4.48.v20220622/jetty-server-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-http/9.4.48.v20220622/jetty-http-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-continuation/9.4.48.v20220622/jetty-continuation-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-proxy/9.4.48.v20220622/jetty-proxy-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-client/9.4.48.v20220622/jetty-client-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-servlets/9.4.48.v20220622/jetty-servlets-9.4.48.v20220622.jar:/home/runner/.cache
/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/jakarta/servlet/jakarta.servlet-api/4.0.3/jakarta.servlet-api-4.0.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/commons-codec/commons-codec/1.15/commons-codec-1.15.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/commons/commons-text/1.9/commons-text-1.9.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/commons-collections/commons-collections/3.2.2
/commons-collections-3.2.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/commons/commons-collections4/4.4/commons-collections4-4.4.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/slf4j/jul-to-slf4j/1.7.32/jul-to-slf4j-1.7.32.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/slf4j/jcl-over-slf4j/1.7.32/jcl-over-slf4j-1.7.32.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-slf4j-impl/2.18.0/log4j-slf4j-impl-2.18.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/m
aven2/org/apache/logging/log4j/log4j-api/2.18.0/log4j-api-2.18.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-core/2.18.0/log4j-core-2.18.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/logging/log4j/log4j-1.2-api/2.18.0/log4j-1.2-api-2.18.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/ning/compress-lzf/1.1/compress-lzf-1.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/xerial/snappy/snappy-java/1.1.8.4/snappy-java-1.1.8.4.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/luben/zstd-jni/1.5.2-3/zstd-jni-1.5.2-3.jar:/home/runner/.cache/coursier/v1/https/ma
ven-central.storage-download.googleapis.com/maven2/org/roaringbitmap/RoaringBitmap/0.9.30/RoaringBitmap-0.9.30.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scala-lang/modules/scala-xml_2.12/1.2.0/scala-xml_2.12-1.2.0.jar:/home/runner/.sbt/boot/scala-2.12.16/lib/scala-reflect.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/json4s/json4s-jackson_2.12/3.7.0-M11/json4s-jackson_2.12-3.7.0-M11.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/core/jersey-client/2.35/jersey-client-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/core/jersey-common/2.35/jersey-common-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/core/jersey-server/2.35/jersey-server-2.35.jar:/home/runner/.cache/
coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/containers/jersey-container-servlet/2.35/jersey-container-servlet-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/containers/jersey-container-servlet-core/2.35/jersey-container-servlet-core-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/inject/jersey-hk2/2.35/jersey-hk2-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/test-framework/providers/jersey-test-framework-provider-simple/2.35/jersey-test-framework-provider-simple-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-all/4.1.77.Final/netty-all-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/n
etty-transport-native-epoll/4.1.77.Final/netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport-native-epoll/4.1.77.Final/netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport-native-kqueue/4.1.77.Final/netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport-native-kqueue/4.1.77.Final/netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-tcnative-classes/2.0.52.Final/netty-tcnative-classes-2.0.52.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/clearspring/analyti
cs/stream/2.9.6/stream-2.9.6.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/dropwizard/metrics/metrics-core/4.2.7/metrics-core-4.2.7.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/dropwizard/metrics/metrics-jvm/4.2.7/metrics-jvm-4.2.7.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/dropwizard/metrics/metrics-json/4.2.7/metrics-json-4.2.7.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/dropwizard/metrics/metrics-graphite/4.2.7/metrics-graphite-4.2.7.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/dropwizard/metrics/metrics-jmx/4.2.7/metrics-jmx-4.2.7.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/fasterxml/jackson/module/jackson-module-scala_2.12/2.13.3/jackson-module-scala_2.12-2.13.3
.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/derby/derby/10.14.2.0/derby-10.14.2.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/ivy/ivy/2.5.0/ivy-2.5.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/oro/oro/2.0.8/oro-2.0.8.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/sourceforge/htmlunit/htmlunit/2.62.0/htmlunit-2.62.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/sourceforge/htmlunit/htmlunit-core-js/2.62.0/htmlunit-core-js-2.62.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/httpcomponents/httpm
ime/4.5.13/httpmime-4.5.13.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/httpcomponents/httpcore/4.4.14/httpcore-4.4.14.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/curator/curator-test/2.13.0/curator-test-2.13.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/hadoop/hadoop-minikdc/3.3.3/hadoop-minikdc-3.3.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/razorvine/pickle/1.2/pickle-1.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/sf/py4j/py4j/0.10.9.5/py4j-0.10.9.5.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/
hadoop/hadoop-aws/3.3.3/hadoop-aws-3.3.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/commons/commons-crypto/1.1.0/commons-crypto-1.1.0.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.12/1.1.2/scala-parser-combinators_2.12-1.1.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/codehaus/janino/janino/3.1.7/janino-3.1.7.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/codehaus/janino/commons-compiler/3.1.7/commons-compiler-3.1.7.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/antlr/antlr4-runtime/4.8/antlr4-runtime-4.8.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/arrow/arrow-vector/8.0.0/arrow-vector-8.0.0.jar:/home/runner/.cache/coursier/v1/https/mav
en-central.storage-download.googleapis.com/maven2/org/apache/arrow/arrow-memory-netty/8.0.0/arrow-memory-netty-8.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/orc/orc-shims/1.7.5/orc-shims-1.7.5.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/google/protobuf/protobuf-java/3.14.0/protobuf-java-3.14.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/airlift/aircompressor/0.21/aircompressor-0.21.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/jetbrains/annotations/17.0.0/annotations-17.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/threeten/threeten-extra/1.5.0/threeten-extra-1.5.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-common/1.12.3/parq
uet-common-1.12.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-encoding/1.12.3/parquet-encoding-1.12.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/yetus/audience-annotations/0.13.0/audience-annotations-0.13.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/carrotsearch/junit-benchmarks/0.7.2/junit-benchmarks-0.7.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/slf4j/slf4j-simple/1.7.22/slf4j-simple-1.7.22.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/mockito/mockito-all/1.10.19/mockito-all-1.10.19.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.stora
ge-download.googleapis.com/maven2/org/easymock/easymock/3.4/easymock-3.4.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-format-structures/1.12.3/parquet-format-structures-1.12.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/semver/api/0.9.33/api-0.9.33.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/parquet/parquet-jackson/1.12.3/parquet-jackson-1.12.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-util-ajax/9.4.48.v20220622/jetty-util-ajax-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-annotations/2.13.3/jackson-annotations-2.13.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/fa
sterxml/jackson/core/jackson-core/2.13.3/jackson-core-2.13.3.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/test-interface/1.0/test-interface-1.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/checkerframework/checker-qual/3.5.0/checker-qual-3.5.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy/1.12.9/byte-buddy-1.12.9.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy-agent/1.12.9/byte-buddy-agent-1.12.9.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/objenesis/objenesis/3.2/objenesis-3.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-api/3.141.59/selenium-api-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage
-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-chrome-driver/3.141.59/selenium-chrome-driver-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-edge-driver/3.141.59/selenium-edge-driver-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-firefox-driver/3.141.59/selenium-firefox-driver-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-ie-driver/3.141.59/selenium-ie-driver-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-opera-driver/3.141.59/selenium-opera-driver-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-remote-driver/3.141.59/seleni
um-remote-driver-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-safari-driver/3.141.59/selenium-safari-driver-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/seleniumhq/selenium/selenium-support/3.141.59/selenium-support-3.141.59.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/squareup/okhttp3/okhttp/3.11.0/okhttp-3.11.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/squareup/okio/okio/1.14.0/okio-1.14.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-core_2.12/3.3.0-SNAP3/scalatest-core_2.12-3.3.0-SNAP3.jar:/home/runne
r/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-featurespec_2.12/3.3.0-SNAP3/scalatest-featurespec_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-flatspec_2.12/3.3.0-SNAP3/scalatest-flatspec_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-freespec_2.12/3.3.0-SNAP3/scalatest-freespec_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-funsuite_2.12/3.3.0-SNAP3/scalatest-funsuite_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-funspec_2.12/3.3.0-SNAP3/scalatest-funspec_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scal
atest/scalatest-propspec_2.12/3.3.0-SNAP3/scalatest-propspec_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-refspec_2.12/3.3.0-SNAP3/scalatest-refspec_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-wordspec_2.12/3.3.0-SNAP3/scalatest-wordspec_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-diagrams_2.12/3.3.0-SNAP3/scalatest-diagrams_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-matchers-core_2.12/3.3.0-SNAP3/scalatest-matchers-core_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-shouldmatchers_2.12/3.3.0-SNAP3/scalatest-shouldmatchers_2.12-3.3.0-
SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-mustmatchers_2.12/3.3.0-SNAP3/scalatest-mustmatchers_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/google/crypto/tink/tink/1.6.1/tink-1.6.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/commons/commons-compress/1.21/commons-compress-1.21.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/avro/avro-ipc/1.11.0/avro-ipc-1.11.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.st
orage-download.googleapis.com/maven2/com/esotericsoftware/kryo-shaded/4.0.2/kryo-shaded-4.0.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/jnr/jnr-ffi/2.2.11/jnr-ffi-2.2.11.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/jnr/jnr-constants/0.10.3/jnr-constants-0.10.3.jar:/home/runner/.cache/coursier/v1/https/repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/curator/curator-framework/2.13.0/curator-framework-2.13.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/zookeeper/zookeeper-jute/3.6.2/zookeeper-jute-3.6.2.jar:/home/runner/.cache/co
ursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-webapp/9.4.48.v20220622/jetty-webapp-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-jndi/9.4.48.v20220622/jetty-jndi-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-io/9.4.48.v20220622/jetty-io-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/roaringbitmap/shims/0.9.30/shims-0.9.30.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/json4s/json4s-core_2.12/3.7.0-M11/json4s-core_2.12-3.7.0-M11.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/jakarta/ws/rs/jakarta.ws.rs-api/2.1.6/jakarta.ws.rs-api-2.1.6.jar:/home/runner/.cache/coursier/v1/https/maven-central.stor
age-download.googleapis.com/maven2/org/glassfish/hk2/external/jakarta.inject/2.6.1/jakarta.inject-2.6.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/jakarta/annotation/jakarta.annotation-api/1.3.5/jakarta.annotation-api-1.3.5.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/hk2/osgi-resource-locator/1.0.3/osgi-resource-locator-1.0.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/jakarta/validation/jakarta.validation-api/2.0.2/jakarta.validation-api-2.0.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/hk2/hk2-locator/2.6.1/hk2-locator-2.6.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/javassist/javassist/3.25.0-GA/javassist-3.25.0-GA.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapi
s.com/maven2/org/glassfish/jersey/test-framework/jersey-test-framework-core/2.35/jersey-test-framework-core-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/containers/jersey-container-simple-http/2.35/jersey-container-simple-http-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-buffer/4.1.77.Final/netty-buffer-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-codec/4.1.77.Final/netty-codec-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-common/4.1.77.Final/netty-common-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-handler/4.1.77.Final/netty-handler-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-d
ownload.googleapis.com/maven2/io/netty/netty-resolver/4.1.77.Final/netty-resolver-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport/4.1.77.Final/netty-transport-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport-classes-epoll/4.1.77.Final/netty-transport-classes-epoll-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport-native-unix-common/4.1.77.Final/netty-transport-native-unix-common-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport-classes-kqueue/4.1.77.Final/netty-transport-classes-kqueue-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/thoughtworks/paranamer/paranamer/2.8/paranamer-2.8.jar:/hom
e/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/xalan/xalan/2.7.2/xalan-2.7.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/sourceforge/htmlunit/neko-htmlunit/2.62.0/neko-htmlunit-2.62.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/sourceforge/htmlunit/htmlunit-cssparser/1.12.0/htmlunit-cssparser-1.12.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/commons-net/commons-net/3.8.0/commons-net-3.8.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/brotli/dec/0.1.2/dec-0.1.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/shapesecurity/salvation2/3.0.0/salvation2-3.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/websocket/websock
et-client/9.4.46.v20220331/websocket-client-9.4.46.v20220331.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-simplekdc/1.0.1/kerb-simplekdc-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.1026/aws-java-sdk-bundle-1.11.1026.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/wildfly/openssl/wildfly-openssl/1.0.7.Final/wildfly-openssl-1.0.7.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/arrow/arrow-format/8.0.0/arrow-format-8.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/arrow/arrow-memory-core/8.0.0/arrow-memory-core-8.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/fasterxml/jackson/datatype/jackso
n-datatype-jsr310/2.13.2/jackson-datatype-jsr310-2.13.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/google/flatbuffers/flatbuffers-java/1.12.0/flatbuffers-java-1.12.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/ow2/asm/asm/9.2/asm-9.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/ow2/asm/asm-commons/9.2/asm-commons-9.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/de/tototec/de.tototec.cmdoption/0.2.0/de.tototec.cmdoption-0.2.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/commons-pool/commons-pool/1.6/commons-pool-1.6.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/scalatest/scalatest-compatible/3.3.0-SNAP3/scalatest-compatible-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/
https/maven-central.storage-download.googleapis.com/maven2/org/scalactic/scalactic_2.12/3.3.0-SNAP3/scalactic_2.12-3.3.0-SNAP3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/google/code/gson/gson/2.8.6/gson-2.8.6.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/tukaani/xz/1.9/xz-1.9.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/jnr/jffi/1.3.9/jffi-1.3.9.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/jnr/jffi/1.3.9/jffi-1.3.9-native.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/ow2/asm/asm-analysis/9.2/asm-analysis-9.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.s
torage-download.googleapis.com/maven2/org/ow2/asm/asm-tree/9.2/asm-tree-9.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/ow2/asm/asm-util/9.2/asm-util-9.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/jnr/jnr-a64asm/1.0.0/jnr-a64asm-1.0.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/github/jnr/jnr-x86asm/1.0.2/jnr-x86asm-1.0.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/curator/curator-client/2.13.0/curator-client-2.13.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/jetty-xml/9.4.48.v20220622/jetty-xml-9.4.48.v20220622.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/json4s/json4s-ast_2.12/3.7.0-M11/json4s-ast_2.12-3.7.0-M11.jar:/home/runner/
.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/json4s/json4s-scalap_2.12/3.7.0-M11/json4s-scalap_2.12-3.7.0-M11.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/hk2/external/aopalliance-repackaged/2.6.1/aopalliance-repackaged-2.6.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/hk2/hk2-api/2.6.1/hk2-api-2.6.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/hk2/hk2-utils/2.6.1/hk2-utils-2.6.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/jakarta/xml/bind/jakarta.xml.bind-api/2.3.3/jakarta.xml.bind-api-2.3.3.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/glassfish/jersey/media/jersey-media-jaxb/2.35/jersey-media-jaxb-2.35.jar:/home/runner/.cache/coursier/v1/https/maven-ce
ntral.storage-download.googleapis.com/maven2/org/simpleframework/simple-http/6.0.1/simple-http-6.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/simpleframework/simple-transport/6.0.1/simple-transport-6.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/simpleframework/simple-common/6.0.1/simple-common-6.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/xalan/serializer/2.7.2/serializer-2.7.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/xerces/xercesImpl/2.12.0/xercesImpl-2.12.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/websocket/websocket-common/9.4.46.v20220331/websocket-common-9.4.46.v20220331.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transp
ort-native-epoll/4.1.77.Final/netty-transport-native-epoll-4.1.77.Final.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-client/1.0.1/kerb-client-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-admin/1.0.1/kerb-admin-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/jakarta/activation/jakarta.activation-api/1.2.2/jakarta.activation-api-1.2.2.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/eclipse/jetty/websocket/websocket-api/9.4.46.v20220331/websocket-api-9.4.46.v20220331.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerby-config/1.0.1/kerby-config-1.0
.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-core/1.0.1/kerb-core-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-common/1.0.1/kerb-common-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-util/1.0.1/kerb-util-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/token-provider/1.0.1/token-provider-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-server/1.0.1/kerb-server-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerby-xdr/1.0.1/kerby-xdr-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/
kerby-pkix/1.0.1/kerby-pkix-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-crypto/1.0.1/kerb-crypto-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/com/nimbusds/nimbus-jose-jwt/3.10/nimbus-jose-jwt-3.10.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerb-identity/1.0.1/kerb-identity-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerby-asn1/1.0.1/kerby-asn1-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/apache/kerby/kerby-util/1.0.1/kerby-util-1.0.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.go
ogleapis.com/maven2/net/minidev/json-smart/1.3.1/json-smart-1.3.1.jar:/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/org/bouncycastle/bcprov-jdk15on/1.52/bcprov-jdk15on-1.52.jar:/home/runner/.sbt/boot/scala-2.12.16/org.scala-sbt/sbt/1.7.1/test-agent-1.7.1.jar:/home/runner/.sbt/boot/scala-2.12.16/org.scala-sbt/sbt/1.7.1/test-interface-1.0.jar sbt.ForkMain 40811 failed with exit code 137[0m
2022-07-23T10:09:10.9930558Z [0m[[0m[33mwarn[0m] [0m[0mIn the last 5006 seconds, 20.534 (0.4%) were spent in GC. [Heap: 2.69GB free of 3.33GB, max 3.56GB] Consider increasing the JVM heap using `-Xmx` or try a different collector, e.g. `-XX:+UseG1GC`, for better performance.[0m
2022-07-23T10:09:12.6699449Z [0m[[0m[31merror[0m] [0m[0m(sql / Test / [31mtest[0m) sbt.TestsFailedException: Tests unsuccessful[0m
2022-07-23T10:09:12.6842264Z [0m[[0m[31merror[0m] [0m[0mTotal time: 5008 s (01:23:28), completed Jul 23, 2022 10:09:12 AM[0m
2022-07-23T10:09:13.4527622Z [0J[error] running /home/runner/work/spark/spark/build/sbt -Phadoop-3 -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest sql/test ; received return code 1
2022-07-23T10:09:14.0520245Z ##[error]Process completed with exit code 18.
```
sbt.ForkMain 40811 failed with exit code 137
This seems to denote the memory issue on the process - killed by OS/container. Not sure whether it hits the limit of Github Action container or not, and if it is, we can increase the memory or not.
@HyukjinKwon Do you have any idea of this? Is it known issue we have to tolerate as of now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #37248:
URL: https://github.com/apache/spark/pull/37248#discussion_r927321253
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala:
##########
@@ -2053,8 +2054,8 @@ class DataFrameSuite extends QueryTest
val df = Dataset.ofRows(spark, statsPlan)
val logicalRDD = LogicalRDD(
- df.logicalPlan.output, spark.sparkContext.emptyRDD, Some(df.queryExecution.analyzed),
- isStreaming = true)(spark)
+ df.logicalPlan.output, spark.sparkContext.emptyRDD[InternalRow], isStreaming = true)(
+ spark, Some(df.queryExecution.optimizedPlan.stats), None)
Review Comment:
Unfortunately we can't, as OutputListAwareStatsTestPlan and OutputListAwareConstraintsTestPlan do not have corresponding physical plan. (output partitioning is picked up from physical plan, as same as we do in Dataset.checkpoint)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37248:
URL: https://github.com/apache/spark/pull/37248#issuecomment-1193085763
Just rebased to master branch. Let's see whether the failure keeps happening...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #37248:
URL: https://github.com/apache/spark/pull/37248#discussion_r927314233
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala:
##########
@@ -116,37 +119,78 @@ case class LogicalRDD(
case e: Attribute => rewrite.getOrElse(e, e)
}.asInstanceOf[SortOrder])
- val rewrittenOriginLogicalPlan = originLogicalPlan.map { plan =>
- assert(output == plan.output, "The output columns are expected to the same for output " +
- s"and originLogicalPlan. output: $output / output in originLogicalPlan: ${plan.output}")
+ val rewrittenStatistics = originStats.map { s =>
+ Statistics(
+ s.sizeInBytes,
+ s.rowCount,
+ AttributeMap[ColumnStat](s.attributeStats.map {
+ case (attr, v) => (rewrite.getOrElse(attr, attr), v)
+ }),
+ s.isRuntime
+ )
+ }
- val projectList = output.map { attr =>
- Alias(attr, attr.name)(exprId = rewrite(attr).exprId)
- }
- Project(projectList, plan)
+ val rewrittenConstraints = originConstraints.map { c =>
+ c.map(_.transform {
+ case e: Attribute => rewrite.getOrElse(e, e)
+ })
}
LogicalRDD(
output.map(rewrite),
rdd,
- rewrittenOriginLogicalPlan,
rewrittenPartitioning,
rewrittenOrdering,
isStreaming
- )(session).asInstanceOf[this.type]
+ )(session, rewrittenStatistics, rewrittenConstraints).asInstanceOf[this.type]
}
override protected def stringArgs: Iterator[Any] = Iterator(output, isStreaming)
override def computeStats(): Statistics = {
- originLogicalPlan.map(_.stats).getOrElse {
+ originStats.getOrElse {
Statistics(
// TODO: Instead of returning a default value here, find a way to return a meaningful size
// estimate for RDDs. See PR 1238 for more discussions.
sizeInBytes = BigInt(session.sessionState.conf.defaultSizeInBytes)
)
}
}
+
+ override lazy val constraints: ExpressionSet = originConstraints.getOrElse(ExpressionSet())
+}
+
+object LogicalRDD {
+ /**
+ * Create a new LogicalRDD based on existing Dataset. stats and constraints are inherited from
Review Comment:
nit:
```suggestion
* Create a new LogicalRDD based on existing Dataset. Stats and constraints are inherited from
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
URL: https://github.com/apache/spark/pull/37248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37248:
URL: https://github.com/apache/spark/pull/37248#issuecomment-1193081390
Well, I see failures in `sql - other tests` fairly consistent which seems to also happen in other PR.
This PR:
https://pipelines.actions.githubusercontent.com/serviceHosts/d1c0934f-d388-49ac-ba5e-1283a65d8b30/_apis/pipelines/1/runs/11623/signedlogcontent/15?urlExpires=2022-07-23T07%3A46%3A17.9651606Z&urlSigningMethod=HMACV1&urlSignature=24fDCI%2B2p5XmFp%2BbkTkrcc8HdFZP9CgJMMemm%2FFNiYc%3D
Other PR (https://github.com/apache/spark/pull/37252):
https://pipelines.actions.githubusercontent.com/serviceHosts/532009b4-88e9-492d-8155-6ec67e62ada4/_apis/pipelines/1/runs/35/signedlogcontent/17?urlExpires=2022-07-23T07%3A46%3A46.5904528Z&urlSigningMethod=HMACV1&urlSignature=Xl94VVDoxI0GGcAS5fO4DjMvZu0uI2l%2FXgBXNF39s6w%3D
Seems like the build became unstable recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a diff in pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
viirya commented on code in PR #37248:
URL: https://github.com/apache/spark/pull/37248#discussion_r927316150
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala:
##########
@@ -2053,8 +2054,8 @@ class DataFrameSuite extends QueryTest
val df = Dataset.ofRows(spark, statsPlan)
val logicalRDD = LogicalRDD(
- df.logicalPlan.output, spark.sparkContext.emptyRDD, Some(df.queryExecution.analyzed),
- isStreaming = true)(spark)
+ df.logicalPlan.output, spark.sparkContext.emptyRDD[InternalRow], isStreaming = true)(
+ spark, Some(df.queryExecution.optimizedPlan.stats), None)
Review Comment:
Maybe we can call `fromDataset` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a diff in pull request #37248: [SPARK-39834][SQL][SS] Include the origin stats and constraints for LogicalRDD if it comes from DataFrame
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #37248:
URL: https://github.com/apache/spark/pull/37248#discussion_r927300383
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/ForeachBatchSinkSuite.scala:
##########
@@ -186,48 +185,6 @@ class ForeachBatchSinkSuite extends StreamTest {
assertPlan(mem2, dsUntyped)
}
- test("Leaf node of Dataset in foreachBatch should carry over origin logical plan") {
Review Comment:
This test is removed as we no longer carry over logical plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org