You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wayang.apache.org by "kamir (via GitHub)" <gi...@apache.org> on 2024/09/04 05:12:02 UTC
[PR] Kamir patch 2 [incubator-wayang]
kamir opened a new pull request, #468:
URL: https://github.com/apache/incubator-wayang/pull/468
I have added the word_count.sh and word_count_kafka.sh example applications in the new module named wayang-applications.
This PR allows me to close this contribution with a partial result:
**A working example of an application using Kafka Source and Kafka Sink in on JavaPlan in Wayang.**
Moving forward, I will start with a new feature branch in a forked repository, to keep the main repository clean in the future.
The branch can be deleted after the merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2330920353
I'd say if that is the pragmatic solution, let's go for it. But knowing that the restructure won't happen for a long time, let's try to create an issue addressing it so we will resolve that technical debt eventually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2331321341
> My understanding was, that all Kafka related functionality should be "grouped" in a specific module, and there, the individual implementations for Java / Spark coexist, but no other place in the project Wayang project has a Kafka dependency.
That is essentially what I meant, yes. But we will push that change into the future as part of #469.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2331285637
The SparkPlatform handles the Kafka Source differently, using its own Spark-Kafka-Source.
Our JavaPlatform works directly with Consumer, and with Producer API. Hence we will have different implementations for this particular Problem.
My understanding was, that all Kafka related functionality should be "grouped" in a specific module, and there, the individual implementations for Java / Spark coexist, but no other place in the project Wayang project has a Kafka dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1751639925
##########
wayang-applications/bin/bootstrap.sh:
##########
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#brew install confluentinc/tap/cli
+#brew install jq
+#brew install git-lfs
+
+export topic_l1_a=region_emea_counts
+export topic_l1_b=region_apac_counts
+export topic_l1_c=region_uswest_counts
+export topic_l2_a=global_contribution
+export topic_l2_b=global_averages
+
+confluent kafka topic delete topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_b --cluster lkc-m2kpj2
+
+confluent kafka topic create topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_b --cluster lkc-m2kpj2
+
+######################################################################################################
+# https://docs.confluent.io/cloud/current/sr/schema_registry_ccloud_tutorial.html
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO="ZZUZ3HASNNFGE2DF:EQNB0QpzDd868qlW0Nz49anodp7JjeDkoCaZelCJiUhfTX7BhuRPhNlDA/swx/Fa"
Review Comment:
That's a private API key, I'd suggest to use the Confluent notation:
export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=<SR API KEY>:<SR API SECRET>
export SCHEMA_REGISTRY_URL=<SR ENDPOINT>
and let the user add parameters to the script, like API_TOKEN= and SCHEMA_URI=
##########
wayang-applications/bin/bootstrap.sh:
##########
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#brew install confluentinc/tap/cli
+#brew install jq
+#brew install git-lfs
+
+export topic_l1_a=region_emea_counts
+export topic_l1_b=region_apac_counts
+export topic_l1_c=region_uswest_counts
+export topic_l2_a=global_contribution
+export topic_l2_b=global_averages
+
+confluent kafka topic delete topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_b --cluster lkc-m2kpj2
+
+confluent kafka topic create topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_b --cluster lkc-m2kpj2
+
+######################################################################################################
+# https://docs.confluent.io/cloud/current/sr/schema_registry_ccloud_tutorial.html
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO="ZZUZ3HASNNFGE2DF:EQNB0QpzDd868qlW0Nz49anodp7JjeDkoCaZelCJiUhfTX7BhuRPhNlDA/swx/Fa"
+export SCHEMA_REGISTRY_URL="https://psrc-lo5k9.eu-central-1.aws.confluent.cloud"
+
+confluent kafka topic list --cluster lkc-m2kpj2
Review Comment:
Maybe that's your cluster name, can we make it also more dynamic?
##########
wayang-applications/bin/cleanup.sh:
##########
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+confluent kafka topic delete topic_l1_a --cluster lkc-m2kpj2
Review Comment:
that includes than everything tied to the specific cluster
##########
wayang-applications/bin/run_wordcount_main.sh:
##########
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+######
+# to adjust variables to your own environment, please configure them in env.sh
+#
+
+cd ../..
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
Review Comment:
hardcoded?
##########
wayang-applications/data/case-study/README.md:
##########
@@ -0,0 +1,3 @@
+# Blossom Sky Setup on the "central node"
Review Comment:
Please change into Wayang, Blossom Sy is the old name of Scalytics.
##########
wayang-applications/bin/run_wordcount_main.sh:
##########
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+######
+# to adjust variables to your own environment, please configure them in env.sh
+#
+
+cd ../..
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
+export WAYANG_APP_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-applications/target/*"
Review Comment:
same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2336664569
Build runs in cloud environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749177036
##########
wayang-applications/src/main/java/org/apache/wayang/applications/OutputSerializer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.wayang.applications;
+
+import org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction;
+import org.apache.wayang.basic.data.Tuple2;
+
+public class OutputSerializer implements SerializableFunction<Tuple2<String, Integer>, String> {
+
+ @Override
+ public String apply(Tuple2<String, Integer> tuple) {
+ return tuple.getField0() + ": " + tuple.getField1();
+ }
+
+ // Example usage within a main method or similar test environment
+ public static void main(String[] args) {
Review Comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2346135767
The OSX specific paths have been removed. We will provide some specific scripts for WIN, OSX, Linux soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752175365
##########
wayang-applications/bin/run_wordcount_main.sh:
##########
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+######
+# to adjust variables to your own environment, please configure them in env.sh
+#
+
+cd ../..
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
+export WAYANG_APP_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-applications/target/*"
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752330143
##########
env_template.sh:
##########
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER=...
+export CLUSTER_API_KEY=...
+export CLUSTER_API_SECRET=...
+export SR_ENDPOINT=...
+export SR_API_KEY=...
+export SR_API_SECRET=...
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
Review Comment:
Right. I see. I can change this.
Will replace it with a hint which says which requirements are needed in the README.md
.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1753301992
##########
env_template.sh:
##########
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER=...
+export CLUSTER_API_KEY=...
+export CLUSTER_API_SECRET=...
+export SR_ENDPOINT=...
+export SR_API_KEY=...
+export SR_API_SECRET=...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO="...:...."
+export SCHEMA_REGISTRY_URL="https://...."
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
Review Comment:
please remove "/Users/kamir/GITHUB.merge/"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752150527
##########
wayang-applications/bin/bootstrap.sh:
##########
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#brew install confluentinc/tap/cli
+#brew install jq
+#brew install git-lfs
+
+export topic_l1_a=region_emea_counts
+export topic_l1_b=region_apac_counts
+export topic_l1_c=region_uswest_counts
+export topic_l2_a=global_contribution
+export topic_l2_b=global_averages
+
+confluent kafka topic delete topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_b --cluster lkc-m2kpj2
+
+confluent kafka topic create topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_b --cluster lkc-m2kpj2
+
+######################################################################################################
+# https://docs.confluent.io/cloud/current/sr/schema_registry_ccloud_tutorial.html
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO="ZZUZ3HASNNFGE2DF:EQNB0QpzDd868qlW0Nz49anodp7JjeDkoCaZelCJiUhfTX7BhuRPhNlDA/swx/Fa"
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2338785458
The debugging line has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749176810
##########
wayang-applications/pom.xml:
##########
@@ -0,0 +1,139 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang</artifactId>
+ <version>0.7.1</version>
+ </parent>
+
+ <artifactId>wayang-applications</artifactId>
+ <packaging>jar</packaging>
+
+ <name>wayang-applications</name>
+ <url>http://maven.apache.org</url>
+
+
+ <properties>
+ <DATA_REPO_001>file://${project.basedir}/wayang-applications/data/case-study/DATA_REPO_001</DATA_REPO_001>
+ <DATA_REPO_002>https://kamir.solidcommunity.net/public/ecolytiq-sustainability-profile</DATA_REPO_002>
+ <DIST_WAYANG_HOME>${project.basedir}/../wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1</DIST_WAYANG_HOME>
+ <WAYANG_VERSION>0.7.1</WAYANG_VERSION>
Review Comment:
Will adjust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749176953
##########
wayang-applications/src/main/java/org/apache/wayang/applications/App.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.wayang.applications;
+
+public class App
+{
+ public static void main( String[] args )
+ {
+ System.out.println( "Hello Apache Wayang friends!" );
Review Comment:
Will delete it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752244768
##########
env_template.sh:
##########
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER=...
+export CLUSTER_API_KEY=...
+export CLUSTER_API_SECRET=...
+export SR_ENDPOINT=...
+export SR_API_KEY=...
+export SR_API_SECRET=...
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
Review Comment:
... /Users/kamir/GITHUB.merge/ ...
Can we use PWD for finding homes of Spark, Hadoop? Or put a warning when developer using it that it's made for Apple environments, Linux hasn't homebrew.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1756300743
##########
env_template.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER= ...
+export CLUSTER_API_KEY= ...
+export CLUSTER_API_SECRET= ...
+export SR_ENDPOINT= ...
+export SR_API_KEY= ...
+export SR_API_SECRET= ...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
+export SCHEMA_REGISTRY_URL="https://.... "
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
Review Comment:
since that's an template we can add a note that's for OSX environments. Linux / Windows would be different. Or @kamir renames the file to osx_env_template.sh ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749878068
##########
bin/wayang-submit:
##########
@@ -120,5 +119,11 @@ do
ARGS="$ARGS \"${arg}\""
done
+WAYANG_CLASSPATH="${WAYANG_CLASSPATH}:${WAYANG_APP_HOME}"
+
+echo $WAYANG_CLASSPATH
Review Comment:
@kamir this doesn't seem resolved tom me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1744826909
##########
wayang-applications/pom.xml:
##########
@@ -0,0 +1,145 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang</artifactId>
+ <version>0.7.1</version>
+ </parent>
+
+ <artifactId>wayang-applications</artifactId>
+ <packaging>jar</packaging>
+
+ <name>wayang-applications</name>
+ <url>http://maven.apache.org</url>
+
+
+ <properties>
+ <DATA_REPO_001>file://${project.basedir}/wayang-applications/data/case-study/DATA_REPO_001</DATA_REPO_001>
+ <DATA_REPO_002>https://kamir.solidcommunity.net/public/ecolytiq-sustainability-profile</DATA_REPO_002>
+ <DIST_WAYANG_HOME>${project.basedir}/../wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1</DIST_WAYANG_HOME>
+ <WAYANG_VERSION>0.7.1</WAYANG_VERSION>
+
+ <spark.version>3.5.0</spark.version>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+
+ <!-- We also specify the file encoding of our source files, to avoid a warning -->
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+
+ <assertj.version>3.17.2</assertj.version>
+ <commons-io.version>2.5</commons-io.version>
+ <guava.version>19.0</guava.version>
+ <hamcrest.version>1.3</hamcrest.version>
+ <jackson.version>2.10.2</jackson.version>
+ <jacoco.version>0.8.5</jacoco.version>
+ <jodatime.version>2.10.6</jodatime.version>
+ <jsonpath.version>2.4.0</jsonpath.version>
+ <junit5.version>5.6.1</junit5.version>
+ <mockito.version>3.5.10</mockito.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-core</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-basic</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-java</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-spark_2.12</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-api-scala-java_2.12</artifactId>
Review Comment:
@kamir the modules with ${name}_2.12 don't exist anymore. We got rid of the 2.12 as we no longer support Scala 2.11, so this would load a very old snapshot of this module. It is very likely, that this is why you can't use your newly build Kafka operators in the JavaPlanBuilder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1743081678
##########
wayang-applications/bin/bootstrap.sh:
##########
@@ -0,0 +1,52 @@
+
Review Comment:
```suggestion
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
```
<!-- license-eye hidden identification -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752174577
##########
wayang-applications/bin/run_wordcount_main.sh:
##########
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+######
+# to adjust variables to your own environment, please configure them in env.sh
+#
+
+cd ../..
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
Review Comment:
This was only a temporary test for replication the error reported by Christoffer.
Removed it, since it did not reproduce the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1756300743
##########
env_template.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER= ...
+export CLUSTER_API_KEY= ...
+export CLUSTER_API_SECRET= ...
+export SR_ENDPOINT= ...
+export SR_API_KEY= ...
+export SR_API_SECRET= ...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
+export SCHEMA_REGISTRY_URL="https://.... "
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
Review Comment:
since that's an template we can add a note that's for OSX environments. Linux / Windows would be different. Or @kamir renames the file to osx_env_template.sh ;) :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1743083103
##########
wayang-applications/bin/bootstrap.sh:
##########
@@ -0,0 +1,52 @@
+
Review Comment:
Fixed the bootstrap.sh file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1745025709
##########
wayang-applications/pom.xml:
##########
@@ -0,0 +1,145 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang</artifactId>
+ <version>0.7.1</version>
+ </parent>
+
+ <artifactId>wayang-applications</artifactId>
+ <packaging>jar</packaging>
+
+ <name>wayang-applications</name>
+ <url>http://maven.apache.org</url>
+
+
+ <properties>
+ <DATA_REPO_001>file://${project.basedir}/wayang-applications/data/case-study/DATA_REPO_001</DATA_REPO_001>
+ <DATA_REPO_002>https://kamir.solidcommunity.net/public/ecolytiq-sustainability-profile</DATA_REPO_002>
+ <DIST_WAYANG_HOME>${project.basedir}/../wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1</DIST_WAYANG_HOME>
+ <WAYANG_VERSION>0.7.1</WAYANG_VERSION>
+
+ <spark.version>3.5.0</spark.version>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+
+ <!-- We also specify the file encoding of our source files, to avoid a warning -->
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+
+ <assertj.version>3.17.2</assertj.version>
+ <commons-io.version>2.5</commons-io.version>
+ <guava.version>19.0</guava.version>
+ <hamcrest.version>1.3</hamcrest.version>
+ <jackson.version>2.10.2</jackson.version>
+ <jacoco.version>0.8.5</jacoco.version>
+ <jodatime.version>2.10.6</jodatime.version>
+ <jsonpath.version>2.4.0</jsonpath.version>
+ <junit5.version>5.6.1</junit5.version>
+ <mockito.version>3.5.10</mockito.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-core</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-basic</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-java</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-spark_2.12</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-api-scala-java_2.12</artifactId>
Review Comment:
This fixed the issue I struggled with.
Thanks Juri.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2337666651
@kamir there is still one debug print in the wayang-submit. If you resolve that, I will approve the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2336642221
All issues solved.
Test Scripts still work on my system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749176444
##########
wayang-applications/bin/run_wordcount.sh:
##########
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
Review Comment:
Yes, currently it is hard coded and must be generalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749176578
##########
wayang-applications/bin/run_wordcount_kafka.sh:
##########
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
Review Comment:
Will make it more generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752152298
##########
wayang-applications/bin/bootstrap.sh:
##########
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#brew install confluentinc/tap/cli
+#brew install jq
+#brew install git-lfs
+
+export topic_l1_a=region_emea_counts
+export topic_l1_b=region_apac_counts
+export topic_l1_c=region_uswest_counts
+export topic_l2_a=global_contribution
+export topic_l2_b=global_averages
+
+confluent kafka topic delete topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic delete topic_l2_b --cluster lkc-m2kpj2
+
+confluent kafka topic create topic_l1_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_b --cluster lkc-m2kpj2
+confluent kafka topic create topic_l1_c --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_a --cluster lkc-m2kpj2
+confluent kafka topic create topic_l2_b --cluster lkc-m2kpj2
+
+######################################################################################################
+# https://docs.confluent.io/cloud/current/sr/schema_registry_ccloud_tutorial.html
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO="ZZUZ3HASNNFGE2DF:EQNB0QpzDd868qlW0Nz49anodp7JjeDkoCaZelCJiUhfTX7BhuRPhNlDA/swx/Fa"
+export SCHEMA_REGISTRY_URL="https://psrc-lo5k9.eu-central-1.aws.confluent.cloud"
+
+confluent kafka topic list --cluster lkc-m2kpj2
Review Comment:
Will be added to .env.sh script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752151575
##########
wayang-applications/bin/cleanup.sh:
##########
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+confluent kafka topic delete topic_l1_a --cluster lkc-m2kpj2
Review Comment:
Removed.
Will be defined in an .env.sh file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1753917734
##########
env_template.sh:
##########
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER=...
+export CLUSTER_API_KEY=...
+export CLUSTER_API_SECRET=...
+export SR_ENDPOINT=...
+export SR_API_KEY=...
+export SR_API_SECRET=...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO="...:...."
+export SCHEMA_REGISTRY_URL="https://...."
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
+export PATH=$PATH:$HADOOP_HOME/bin
+export WAYANG_HOME="/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1"
Review Comment:
env_template.sh in root folder has been removed. It is now in wayang-applications because it is only relevant for the demo-apps we create there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1756734185
##########
env_template.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER= ...
+export CLUSTER_API_KEY= ...
+export CLUSTER_API_SECRET= ...
+export SR_ENDPOINT= ...
+export SR_API_KEY= ...
+export SR_API_SECRET= ...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
+export SCHEMA_REGISTRY_URL="https://.... "
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
Review Comment:
Same thought for this:
> That would be the case, if it was like this for all OSX users. I think this shouldn't be versioned at all, as it makes a lot of assumptions on ones setup. This can easily be excluded and we could prepare a .env.example file that holds values like this as "guardrails". What do you think @kamir @2pk03?
##########
env_template.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER= ...
+export CLUSTER_API_KEY= ...
+export CLUSTER_API_SECRET= ...
+export SR_ENDPOINT= ...
+export SR_API_KEY= ...
+export SR_API_SECRET= ...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
+export SCHEMA_REGISTRY_URL="https://.... "
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
Review Comment:
That would be the case, if it was like this for all OSX users. I think this shouldn't be versioned at all, as it makes a lot of assumptions on ones setup. This can easily be excluded and we could prepare a .env.example file that holds values like this as "guardrails". What do you think @kamir @2pk03?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2330913106
> @kamir do you think that creating a `wayang-kafka` module inside of `wayang-platforms` would be a cleaner approach to add these operators? Right now it seems like you edit both `wayang-java` and `wayang-spark`, where both of them contain versions of operators from Kafka. I think conceptually, having all implementations for the specific apis in `wayang-kafka` would be neat. Then you could have `wayang-kafka/src/main/java/JavaKafkaOperator` `wayang-kafka/src/main/java/SparkKafkaOperator` etc and the respective platforms themselves don't have to concern about Kafka.
>
> What do you think?
Yes, this is a great idea. I was thinking about the issue of "double implementation" but did not make so far towards a solution.
Yes, lets do this.
To get ready for this task I suggest, that I first finish the demo, and the blog.
We all together nail the release 1.0.
And after we reached the 1.0 level, we make this restructuring and release it as 1.1.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2331261539
But just to make sure we are all on the same page: Is the code for a Kafka source operator and a Java source operator the same? For example, for the TextFileSource it's not the same code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1745450491
##########
guides/develop-with-Wayang.md:
##########
@@ -71,7 +71,7 @@ This tutorial shows users how to import Wayang in their Java project using the m
```
A sample pom file can be found [here](pom-example.xml).
-# Test WordCount
+# Test org.apache.wayang.examples.WordCount
Review Comment:
Same here, this doesn't really add anything.
##########
bin/wayang-submit:
##########
@@ -120,5 +119,11 @@ do
ARGS="$ARGS \"${arg}\""
done
+WAYANG_CLASSPATH="${WAYANG_CLASSPATH}:${WAYANG_APP_HOME}"
+
+echo $WAYANG_CLASSPATH
Review Comment:
Are these just for debugging? I don't think they are needed as these echos are only run after submitting a job.
##########
guides/wayang-examples.md:
##########
@@ -51,7 +51,7 @@ public class WordcountJava {
.withPlugin(Java.basicPlugin())
.withPlugin(Spark.basicPlugin());
JavaPlanBuilder planBuilder = new JavaPlanBuilder(wayangContext)
- .withJobName(String.format("WordCount (%s)", inputUrl))
+ .withJobName(String.format("org.apache.wayang.examples.WordCount (%s)", inputUrl))
Review Comment:
Same here
##########
guides/wayang-examples.md:
##########
@@ -17,13 +17,13 @@
-->
This page contains examples to be executed using Wayang.
-- [WordCount](#wordcount)
+- [org.apache.wayang.examples.WordCount](#wordcount)
* [Java scala-like API](#java-scala-like-api)
* [Scala API](#scala-api)
- [k-means](#k-means)
* [Scala API](#scala-api-1)
-## WordCount
+## org.apache.wayang.examples.WordCount
Review Comment:
Same here
##########
wayang-api/wayang-api-scala-java/README.md:
##########
@@ -87,12 +87,12 @@ import org.apache.wayang.core.api.Configuration
import org.apache.wayang.java.Java
import org.apache.wayang.spark.Spark
-class WordCount {}
+class org.apache.wayang.examples.WordCount {}
Review Comment:
This is an invalid class name in this case.
##########
wayang-api/wayang-api-scala-java/README.md:
##########
@@ -87,12 +87,12 @@ import org.apache.wayang.core.api.Configuration
import org.apache.wayang.java.Java
import org.apache.wayang.spark.Spark
-class WordCount {}
+class org.apache.wayang.examples.WordCount {}
-object WordCount {
+object org.apache.wayang.examples.WordCount {
Review Comment:
This is an invalid object name in this case.
##########
wayang-applications/pom.xml:
##########
@@ -0,0 +1,139 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang</artifactId>
+ <version>0.7.1</version>
+ </parent>
+
+ <artifactId>wayang-applications</artifactId>
+ <packaging>jar</packaging>
+
+ <name>wayang-applications</name>
+ <url>http://maven.apache.org</url>
+
+
+ <properties>
+ <DATA_REPO_001>file://${project.basedir}/wayang-applications/data/case-study/DATA_REPO_001</DATA_REPO_001>
+ <DATA_REPO_002>https://kamir.solidcommunity.net/public/ecolytiq-sustainability-profile</DATA_REPO_002>
+ <DIST_WAYANG_HOME>${project.basedir}/../wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1</DIST_WAYANG_HOME>
+ <WAYANG_VERSION>0.7.1</WAYANG_VERSION>
+
+ <spark.version>3.5.0</spark.version>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+
+ <!-- We also specify the file encoding of our source files, to avoid a warning -->
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+
+ <assertj.version>3.17.2</assertj.version>
+ <commons-io.version>2.5</commons-io.version>
+ <guava.version>19.0</guava.version>
+ <hamcrest.version>1.3</hamcrest.version>
+ <jackson.version>2.10.2</jackson.version>
+ <jacoco.version>0.8.5</jacoco.version>
+ <jodatime.version>2.10.6</jodatime.version>
+ <jsonpath.version>2.4.0</jsonpath.version>
+ <junit5.version>5.6.1</junit5.version>
+ <mockito.version>3.5.10</mockito.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-core</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-basic</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-java</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-spark</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-api-scala-java</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ <!--scope>system</scope-->
+ <!--systemPath>/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1/jars/wayang-api-scala-java-0.7.1.jar</systemPath-->
Review Comment:
This is local to your machine and can be removed.
##########
wayang-applications/src/main/java/org/apache/wayang/applications/App.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.wayang.applications;
+
+public class App
+{
+ public static void main( String[] args )
+ {
+ System.out.println( "Hello Apache Wayang friends!" );
Review Comment:
This class seems to be obsolete
##########
wayang-docs/src/main/resources/index.md:
##########
@@ -172,7 +172,7 @@ This tool will attempt to determine suitable values for the question marks (`?`)
For some executable examples, have a look at [this repository](https://github.com/sekruse/rheem-examples).
-### WordCount
+### org.apache.wayang.examples.WordCount
Review Comment:
Same here
##########
wayang-docs/src/main/resources/index.md:
##########
@@ -198,7 +198,7 @@ public class WordcountJava {
.withPlugin(Java.basicPlugin())
.withPlugin(Spark.basicPlugin());
JavaPlanBuilder planBuilder = new JavaPlanBuilder(wayangContext)
- .withJobName(String.format("WordCount (%s)", inputUrl))
+ .withJobName(String.format("org.apache.wayang.examples.WordCount (%s)", inputUrl))
Review Comment:
Same here
##########
guides/ml-in-Wayang.md:
##########
@@ -28,7 +28,7 @@ public class CustomEstimatableCost implements EstimatableCost {
* by implementing the interface in this class.
*/
}
-public class WordCount {
+public class org.apache.wayang.examples.WordCount {
Review Comment:
I think you replaced everything in a recursive search, this is not valid.
##########
guides/tutorial.md:
##########
@@ -16,7 +16,7 @@
limitations under the License.
-->
-This tutorial will show users how to run the WordCount example locally with Wayang.
+This tutorial will show users how to run the org.apache.wayang.examples.WordCount example locally with Wayang.
Review Comment:
Same here
##########
wayang-applications/bin/run_wordcount_kafka.sh:
##########
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
Review Comment:
This seems to be hardcoded to your machine?
##########
guides/wayang-examples.md:
##########
@@ -17,13 +17,13 @@
-->
This page contains examples to be executed using Wayang.
-- [WordCount](#wordcount)
+- [org.apache.wayang.examples.WordCount](#wordcount)
Review Comment:
Same here.
##########
guides/tutorial.md:
##########
@@ -57,7 +57,7 @@ source ~/.zshrc
# Run the program
-To execute the WordCount example with Apache Wayang, you need to execute your program with the 'wayang-submit' command:
+To execute the org.apache.wayang.examples.WordCount example with Apache Wayang, you need to execute your program with the 'wayang-submit' command:
Review Comment:
Same here
##########
wayang-applications/bin/run_wordcount.sh:
##########
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+export JAVA_HOME=/Users/kamir/.sdkman/candidates/java/current
Review Comment:
This seems to be hardcoded to your machine?
##########
wayang-api/wayang-api-scala-java/README.md:
##########
@@ -87,12 +87,12 @@ import org.apache.wayang.core.api.Configuration
import org.apache.wayang.java.Java
import org.apache.wayang.spark.Spark
-class WordCount {}
+class org.apache.wayang.examples.WordCount {}
-object WordCount {
+object org.apache.wayang.examples.WordCount {
def main(args: Array[String]): Unit = {
- println("WordCount")
+ println("org.apache.wayang.examples.WordCount")
Review Comment:
Same here
##########
guides/wayang-examples.md:
##########
@@ -107,7 +107,7 @@ object WordcountScala {
.withPlugin(Java.basicPlugin)
.withPlugin(Spark.basicPlugin)
val planBuilder = new PlanBuilder(wayangContext)
- .withJobName(s"WordCount ($inputUrl)")
+ .withJobName(s"org.apache.wayang.examples.WordCount ($inputUrl)")
Review Comment:
Same here
##########
wayang-applications/src/main/java/org/apache/wayang/applications/WordCount.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.wayang.applications;
+
+import org.apache.wayang.api.JavaPlanBuilder;
+import org.apache.wayang.basic.data.Tuple2;
+import org.apache.wayang.core.api.Configuration;
+import org.apache.wayang.core.api.WayangContext;
+import org.apache.wayang.core.optimizer.cardinality.DefaultCardinalityEstimator;
+import org.apache.wayang.java.Java;
+import org.apache.wayang.spark.Spark;
+import java.util.Collection;
+import java.util.Arrays;
+
+public class WordCount {
Review Comment:
This class just seems to be another ordinary WordCount example that we have in multiple other modules like wayang-benchmark. Why do we need another one?
##########
wayang-applications/src/main/java/org/apache/wayang/applications/OutputSerializer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.wayang.applications;
+
+import org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction;
+import org.apache.wayang.basic.data.Tuple2;
+
+public class OutputSerializer implements SerializableFunction<Tuple2<String, Integer>, String> {
+
+ @Override
+ public String apply(Tuple2<String, Integer> tuple) {
+ return tuple.getField0() + ": " + tuple.getField1();
+ }
+
+ // Example usage within a main method or similar test environment
+ public static void main(String[] args) {
Review Comment:
A "utility" class like this should probably not contain a main.
##########
wayang-docs/src/main/resources/index.md:
##########
@@ -254,7 +254,7 @@ object WordcountScala {
.withPlugin(Java.basicPlugin)
.withPlugin(Spark.basicPlugin)
val planBuilder = new PlanBuilder(wayangContext)
- .withJobName(s"WordCount ($inputUrl)")
+ .withJobName(s"org.apache.wayang.examples.WordCount ($inputUrl)")
Review Comment:
Same here
##########
wayang-applications/pom.xml:
##########
@@ -0,0 +1,139 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang</artifactId>
+ <version>0.7.1</version>
+ </parent>
+
+ <artifactId>wayang-applications</artifactId>
+ <packaging>jar</packaging>
+
+ <name>wayang-applications</name>
+ <url>http://maven.apache.org</url>
+
+
+ <properties>
+ <DATA_REPO_001>file://${project.basedir}/wayang-applications/data/case-study/DATA_REPO_001</DATA_REPO_001>
+ <DATA_REPO_002>https://kamir.solidcommunity.net/public/ecolytiq-sustainability-profile</DATA_REPO_002>
+ <DIST_WAYANG_HOME>${project.basedir}/../wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1</DIST_WAYANG_HOME>
+ <WAYANG_VERSION>0.7.1</WAYANG_VERSION>
Review Comment:
In other pom.xml's we use the naming convention that would result in this property being named `wayang.version`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749176863
##########
wayang-applications/pom.xml:
##########
@@ -0,0 +1,139 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang</artifactId>
+ <version>0.7.1</version>
+ </parent>
+
+ <artifactId>wayang-applications</artifactId>
+ <packaging>jar</packaging>
+
+ <name>wayang-applications</name>
+ <url>http://maven.apache.org</url>
+
+
+ <properties>
+ <DATA_REPO_001>file://${project.basedir}/wayang-applications/data/case-study/DATA_REPO_001</DATA_REPO_001>
+ <DATA_REPO_002>https://kamir.solidcommunity.net/public/ecolytiq-sustainability-profile</DATA_REPO_002>
+ <DIST_WAYANG_HOME>${project.basedir}/../wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1</DIST_WAYANG_HOME>
+ <WAYANG_VERSION>0.7.1</WAYANG_VERSION>
+
+ <spark.version>3.5.0</spark.version>
+ <maven.compiler.source>11</maven.compiler.source>
+ <maven.compiler.target>11</maven.compiler.target>
+
+ <!-- We also specify the file encoding of our source files, to avoid a warning -->
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+
+ <assertj.version>3.17.2</assertj.version>
+ <commons-io.version>2.5</commons-io.version>
+ <guava.version>19.0</guava.version>
+ <hamcrest.version>1.3</hamcrest.version>
+ <jackson.version>2.10.2</jackson.version>
+ <jacoco.version>0.8.5</jacoco.version>
+ <jodatime.version>2.10.6</jodatime.version>
+ <jsonpath.version>2.4.0</jsonpath.version>
+ <junit5.version>5.6.1</junit5.version>
+ <mockito.version>3.5.10</mockito.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-core</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-basic</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-java</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-spark</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.wayang</groupId>
+ <artifactId>wayang-api-scala-java</artifactId>
+ <version>${WAYANG_VERSION}</version>
+ <!--scope>system</scope-->
+ <!--systemPath>/Users/kamir/GITHUB.merge/incubator-wayang/wayang-assembly/target/apache-wayang-assembly-0.7.1-incubating-dist/wayang-0.7.1/jars/wayang-api-scala-java-0.7.1.jar</systemPath-->
Review Comment:
Ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1745446547
##########
README.md:
##########
@@ -48,7 +48,7 @@ Apache Wayang (incubating) can be used via the following APIs:
## Quick Guide for Running Wayang
-For a quick guide on how to run WordCount see [here](guides/tutorial.md).
+For a quick guide on how to run org.apache.wayang.examples.WordCount see [here](guides/tutorial.md).
Review Comment:
Is this change intended? If so, does it really benefit someone reading that guide?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2346142257
@2pk03 @kamir I approved the review. I am not sure why this still requires one more review by someone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 merged PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2330763603
@kamir do you think that creating a `wayang-kafka` module inside of `wayang-platforms` would be a cleaner approach to add these operators? Right now it seems like you edit both `wayang-java` and `wayang-spark`, where both of them contain versions of operators from Kafka. I think conceptually, having all implementations for the specific apis in `wayang-kafka` would be neat. Then you could have `wayang-kafka/src/main/java/JavaKafkaOperator` `wayang-kafka/src/main/java/SparkKafkaOperator` etc and the respective platforms themselves don't have to concern about Kafka.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "zkaoudi (via GitHub)" <gi...@apache.org>.
zkaoudi commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2330903445
That's an interesting idea @juripetersen. If we go that direction, we should do for all data sources, right? maybe a parent module wayang-datasources?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#issuecomment-2331029340
Issue is created:
https://github.com/apache/incubator-wayang/issues/469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749177721
##########
wayang-applications/src/main/java/org/apache/wayang/applications/WordCount.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.wayang.applications;
+
+import org.apache.wayang.api.JavaPlanBuilder;
+import org.apache.wayang.basic.data.Tuple2;
+import org.apache.wayang.core.api.Configuration;
+import org.apache.wayang.core.api.WayangContext;
+import org.apache.wayang.core.optimizer.cardinality.DefaultCardinalityEstimator;
+import org.apache.wayang.java.Java;
+import org.apache.wayang.spark.Spark;
+import java.util.Collection;
+import java.util.Arrays;
+
+public class WordCount {
Review Comment:
In this case, I use it as a starting point for building a set of example apps for a tutorial series. The replication is due to the idea of having all relevant example in one place, and I think it can be optimized later on, for now it helps me to show things faster, e.g. when I go to a Meetup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1749187454
##########
env_template.sh:
##########
@@ -0,0 +1,19 @@
+export BOOTSTRAP_SERVER=pkc-75m1o.europe-west3.gcp.confluent.cloud:9092
Review Comment:
```suggestion
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
export BOOTSTRAP_SERVER=pkc-75m1o.europe-west3.gcp.confluent.cloud:9092
```
<!-- license-eye hidden identification -->
##########
wayang-applications/bin/env_template.sh:
##########
@@ -0,0 +1 @@
+export JAVA_HOME=....
Review Comment:
```suggestion
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
export JAVA_HOME=....
```
<!-- license-eye hidden identification -->
##########
env_template.sh:
##########
@@ -0,0 +1,19 @@
+export BOOTSTRAP_SERVER=pkc-75m1o.europe-west3.gcp.confluent.cloud:9092
Review Comment:
```suggestion
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
export BOOTSTRAP_SERVER=pkc-75m1o.europe-west3.gcp.confluent.cloud:9092
```
<!-- license-eye hidden identification -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "juripetersen (via GitHub)" <gi...@apache.org>.
juripetersen commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1754314686
##########
env_template.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER= ...
+export CLUSTER_API_KEY= ...
+export CLUSTER_API_SECRET= ...
+export SR_ENDPOINT= ...
+export SR_API_KEY= ...
+export SR_API_SECRET= ...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
+export SCHEMA_REGISTRY_URL="https://.... "
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
Review Comment:
This seems to be mac os specific. Maybe also make it an argument or `...`
##########
env_template.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER= ...
+export CLUSTER_API_KEY= ...
+export CLUSTER_API_SECRET= ...
+export SR_ENDPOINT= ...
+export SR_API_KEY= ...
+export SR_API_SECRET= ...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
+export SCHEMA_REGISTRY_URL="https://.... "
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
+export HADOOP_HOME=/opt/homebrew/Cellar/hadoop/3.4.0/libexec
Review Comment:
This seems to be mac os specific. Maybe also make it an argument or ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "kamir (via GitHub)" <gi...@apache.org>.
kamir commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1752174990
##########
wayang-applications/data/case-study/README.md:
##########
@@ -0,0 +1,3 @@
+# Blossom Sky Setup on the "central node"
Review Comment:
Removed - was an old artifact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Kamir patch 2 [incubator-wayang]
Posted by "2pk03 (via GitHub)" <gi...@apache.org>.
2pk03 commented on code in PR #468:
URL: https://github.com/apache/incubator-wayang/pull/468#discussion_r1756300743
##########
env_template.sh:
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+export BOOTSTRAP_SERVER= ...
+export CLUSTER_API_KEY= ...
+export CLUSTER_API_SECRET= ...
+export SR_ENDPOINT= ...
+export SR_API_KEY= ...
+export SR_API_SECRET= ...
+export SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO=" ... : .... "
+export SCHEMA_REGISTRY_URL="https://.... "
+
+export SPARK_HOME=/opt/homebrew/Cellar/apache-spark/3.5.2/libexec
Review Comment:
since that's an template we can add a note that's for OSX environments. Linux / Windows would be different. Or @kamir renames the file to osx_env_template.sh ;) 4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@wayang.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org