You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wayang.apache.org by zk...@apache.org on 2025/04/03 12:56:24 UTC
(incubator-wayang) branch main updated (74b91763 -> 4df21216)
This is an automated email from the ASF dual-hosted git repository.
zkaoudi pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-wayang.git
from 74b91763 Merge pull request #523 from mspruc/main
add 2193e49d Create ParquetSource
add 50934415 Update Record.java
add 17759ef8 Add readParquet to plan builders
add c97fa727 Add Java execution operator to ParquetSource
add 798ec683 Add SparkSession to executor
add 37cc8abc Add Spark execution operator to ParquetSource
add 0041654e Create WordCount example with parquet file
add dcd26d30 Update parquet word count example for projections
add 4d181516 Add setField and addField to Record
add 610aef83 Merge remote-tracking branch 'upstream/main'
add 6fa1a240 Merge latest changes
add 5c86222f Remove comment
add 0109c022 Merge branch 'main' into main
add d14fd248 Merge remote-tracking branch 'origin/main'
new 4df21216 Merge pull request #528 from aurb9/main
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/wayang/api/JavaPlanBuilder.scala | 10 +-
.../scala/org/apache/wayang/api/PlanBuilder.scala | 12 +-
.../wayang/apps/wordcount/WordCountParquet.java | 53 +++---
wayang-commons/wayang-basic/pom.xml | 5 +
.../java/org/apache/wayang/basic/data/Record.java | 27 ++++
.../wayang/basic/operators/ParquetSource.java | 178 ++++++++++++++++++++
wayang-platforms/wayang-java/pom.xml | 10 ++
.../org/apache/wayang/java/mapping/Mappings.java | 1 +
...ourceMapping.java => ParquetSourceMapping.java} | 15 +-
.../wayang/java/operators/JavaParquetSource.java | 180 +++++++++++++++++++++
.../main/resources/wayang-java-defaults.properties | 20 +++
.../wayang/spark/execution/SparkExecutor.java | 7 +
.../org/apache/wayang/spark/mapping/Mappings.java | 1 +
...ourceMapping.java => ParquetSourceMapping.java} | 11 +-
...TextFileSource.java => SparkParquetSource.java} | 58 ++++---
.../resources/wayang-spark-defaults.properties | 28 ++++
16 files changed, 558 insertions(+), 58 deletions(-)
copy guides/WordCount.java => wayang-benchmark/src/main/java/org/apache/wayang/apps/wordcount/WordCountParquet.java (57%)
create mode 100644 wayang-commons/wayang-basic/src/main/java/org/apache/wayang/basic/operators/ParquetSource.java
copy wayang-platforms/wayang-java/src/main/java/org/apache/wayang/java/mapping/{TextFileSourceMapping.java => ParquetSourceMapping.java} (80%)
create mode 100644 wayang-platforms/wayang-java/src/main/java/org/apache/wayang/java/operators/JavaParquetSource.java
copy wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/mapping/{CollectionSourceMapping.java => ParquetSourceMapping.java} (82%)
copy wayang-platforms/wayang-spark/src/main/java/org/apache/wayang/spark/operators/{SparkTextFileSource.java => SparkParquetSource.java} (66%)
(incubator-wayang) 01/01: Merge pull request #528 from aurb9/main
Posted by zk...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zkaoudi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-wayang.git
commit 4df2121612bcb53ad27210d664489bb3915d0dd0
Merge: 74b91763 d14fd248
Author: Zoi Kaoudi <zk...@users.noreply.github.com>
AuthorDate: Thu Apr 3 14:56:19 2025 +0200
Merge pull request #528 from aurb9/main
Add ParquetSource operator with Java and Spark implementations
.../org/apache/wayang/api/JavaPlanBuilder.scala | 10 +-
.../scala/org/apache/wayang/api/PlanBuilder.scala | 12 +-
.../wayang/apps/wordcount/WordCountParquet.java | 94 +++++++++++
wayang-commons/wayang-basic/pom.xml | 5 +
.../java/org/apache/wayang/basic/data/Record.java | 27 ++++
.../wayang/basic/operators/ParquetSource.java | 178 ++++++++++++++++++++
wayang-platforms/wayang-java/pom.xml | 10 ++
.../org/apache/wayang/java/mapping/Mappings.java | 1 +
.../wayang/java/mapping/ParquetSourceMapping.java | 58 +++++++
.../wayang/java/operators/JavaParquetSource.java | 180 +++++++++++++++++++++
.../main/resources/wayang-java-defaults.properties | 20 +++
.../wayang/spark/execution/SparkExecutor.java | 7 +
.../org/apache/wayang/spark/mapping/Mappings.java | 1 +
.../wayang/spark/mapping/ParquetSourceMapping.java | 61 +++++++
.../wayang/spark/operators/SparkParquetSource.java | 122 ++++++++++++++
.../resources/wayang-spark-defaults.properties | 28 ++++
16 files changed, 811 insertions(+), 3 deletions(-)