You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wayang.apache.org by "novatechflow (via GitHub)" <gi...@apache.org> on 2025/12/16 08:20:16 UTC
[PR] #634 [incubator-wayang]
novatechflow opened a new pull request, #647:
URL: https://github.com/apache/incubator-wayang/pull/647
this should fix the non-deterministic enumeration. Main issue from my point of view was that we relied completely on plain `HashSet` all over the place => the order changed every time the JVM ran, therefore a unpredictable hash-set ordering was the root cause.
Wrote a regression test, too:
`mvn -pl wayang-commons/wayang-core -Dtest=PlanEnumerationDeterminismTest,ChannelConversionGraphDeterminismTest test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "novatechflow (via GitHub)" <gi...@apache.org>.
novatechflow closed pull request #647: #634
URL: https://github.com/apache/wayang/pull/647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671364892
##########
wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/mapping/Mappings.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/mapping/ParquetSinkMapping.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/operators/SparkDatasetToRddOperator.java:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671328787
##########
guides/spark-datasets.md:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671327509
##########
README.md:
##########
Review Comment:
This file should probably not be in this PR right?
I think a PR should only ever contain changes relevant for the issue it is trying to solve.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671369772
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/operators/SparkParquetSinkTest.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/operators/SparkParquetSourceDatasetOutputTest.java:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671333083
##########
wayang-commons/wayang-basic/src/main/java/org/apache/wayang/basic/operators/ParquetSink.java:
##########
Review Comment:
See comment above.
##########
wayang-commons/wayang-basic/src/main/java/org/apache/wayang/basic/operators/ParquetSource.java:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671331299
##########
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/DataQuanta.scala:
##########
@@ -1027,6 +1027,12 @@ class DataQuanta[Out: ClassTag](val operator: ElementaryOperator, outputIndex: I
writeTextFileJava(url, toSerializableFunction(formatterUdf), udfLoad)
}
+ def writeParquet(url: String, overwrite: Boolean = false)(implicit ev: Out =:= Record): Unit =
Review Comment:
See comment above.
##########
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/JavaPlanBuilder.scala:
##########
@@ -72,6 +72,16 @@ class JavaPlanBuilder(wayangCtx: WayangContext, jobName: String) {
def readParquet(url: String, projection: Array[String] = null): UnarySourceDataQuantaBuilder[UnarySourceDataQuantaBuilder[_, Record], Record] =
createSourceBuilder(ParquetSource.create(url, projection))(ClassTag(classOf[Record]))
+ /**
+ * Read a parquet file and provide it as a dataset of [[Record]]s backed by Spark Datasets.
+ *
+ * @param url the URL of the Parquet file
+ * @param projection the projection, if any
+ * @return [[DataQuantaBuilder]] for the file
+ */
+ def readParquetAsDataset(url: String, projection: Array[String] = null): UnarySourceDataQuantaBuilder[UnarySourceDataQuantaBuilder[_, Record], Record] =
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [incubator-wayang]
Posted by "novatechflow (via GitHub)" <gi...@apache.org>.
novatechflow commented on PR #647:
URL: https://github.com/apache/incubator-wayang/pull/647#issuecomment-3659347890
Since #634 popped up in the main PR I thought bring them together would make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671369062
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/operators/DatasetTestUtils.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/operators/SparkDatasetToRddOperatorTest.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/operators/SparkOperatorTestBase.java:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671367843
##########
wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/util/DatasetConverters.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/main/resources/wayang-spark-defaults.properties:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/operators/DatasetChannelTest.java:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by Alexander Alten <al...@gmail.com>.
Hey Juri,
I don’t see your comment in the PR, weird. Anyhow, yes, we should merge the work. You can use the PR and add your code, and we rebase? Better ideas?
Best,
—Alex
> On Jan 7, 2026, at 08:21, juripetersen (via GitHub) <gi...@apache.org> wrote:
>
>
> juripetersen commented on PR #647:
> URL: https://github.com/apache/wayang/pull/647#issuecomment-3717633943
>
> @novatechflow we have a different fix for the inconsistency in a fork of ours, which involved `DefaultEstimatableCost`.
> I will create a PR to introduce before we continue with this one, okay?
>
>
> --
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
>
> For queries about this service, please contact Infrastructure at:
> users@infra.apache.org
>
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on PR #647:
URL: https://github.com/apache/wayang/pull/647#issuecomment-3717633943
@novatechflow we have a different fix for the inconsistency in a fork of ours, which involved `DefaultEstimatableCost`.
I will create a PR to introduce before we continue with this one, okay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671332306
##########
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/PlanBuilder.scala:
##########
@@ -140,6 +140,16 @@ class PlanBuilder(private[api] val wayangContext: WayangContext, private var job
*/
def readParquet(url: String, projection: Array[String] = null): DataQuanta[Record] = load(ParquetSource.create(url, projection))
+ /**
+ * Read a parquet file and keep it backed by a Spark Dataset throughout execution.
+ *
+ * @param url the URL of the Parquet file
+ * @param projection the projection, if any
+ * @return [[DataQuanta]] of [[Record]]s backed by a Spark Dataset when executed on Spark
+ */
+ def readParquetAsDataset(url: String, projection: Array[String] = null): DataQuanta[Record] =
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671365646
##########
wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/operators/SparkParquetSink.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/operators/SparkParquetSource.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/operators/SparkRddToDatasetOperator.java:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #647:
URL: https://github.com/apache/wayang/pull/647#discussion_r2671370972
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/operators/SparkRddToDatasetOperatorTest.java:
##########
Review Comment:
See comment above.
##########
wayang-platforms/wayang-spark/src/test/java/org/apache/wayang/spark/test/ChannelFactory.java:
##########
Review Comment:
See comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #634 [wayang]
Posted by "novatechflow (via GitHub)" <gi...@apache.org>.
novatechflow commented on PR #647:
URL: https://github.com/apache/wayang/pull/647#issuecomment-3767246572
Closing in favor of #662, which cleanly separates determinism fixes/tests from the Spark DataFrames work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org