You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/01/30 16:06:33 UTC
[GitHub] [incubator-seatunnel] TyrantLucifer opened a new pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
TyrantLucifer opened a new pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[SeaTunnel #XXXX] [component] Title of the pull request", where *SeaTunnel #XXXX* should be replaced by the actual issue number.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [x] Code changed are covered with tests, or it does not need tests for reason:
I have done compilation tests and unit tests, please review.
* [ ] If any new Jar binary package adding in you PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/developement/NewLicenseGuide.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800184080
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchExecution.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.config.ConfigRuntimeException;
+import org.apache.seatunnel.env.Execution;
+import org.apache.seatunnel.spark.BaseSparkSink;
+import org.apache.seatunnel.spark.BaseSparkSource;
+import org.apache.seatunnel.spark.BaseSparkTransform;
+import org.apache.seatunnel.spark.SparkEnvironment;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import java.util.List;
+
+public class SparkBatchExecution implements Execution<SparkBatchSource, BaseSparkTransform, SparkBatchSink> {
+
+ public static final String SOURCE_TABLE_NAME = "source_table_name";
+ public static final String RESULT_TABLE_NAME = "result_table_name";
+
+ public static void registerTempView(String tableName, Dataset<Row> ds) {
+ ds.createOrReplaceGlobalTempView(tableName);
+ }
+
+ public static void registerInputTempView(BaseSparkSource<Dataset<Row>> source, SparkEnvironment environment) {
+ Config config = source.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ String tableName = config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(tableName, source.getData(environment));
+ } else {
+ throw new ConfigRuntimeException(
+ "Plugin[" + source.getClass().getName() + "] must be registered as dataset/table, please set \"result_table_name\" config");
+ }
+ }
+
+ public static Dataset<Row> transformProcess(SparkEnvironment environment, BaseSparkTransform transform, Dataset<Row> ds) {
+ Dataset<Row> fromDs;
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.SOURCE_TABLE_NAME)) {
+ String sourceTableName = config.getString(SparkBatchExecution.SOURCE_TABLE_NAME);
+ fromDs = environment.getSparkSession().read().table(sourceTableName);
+ } else {
+ fromDs = ds;
+ }
+ return transform.process(fromDs, environment);
+ }
+
+ public static void registerTransformTempView(BaseSparkTransform transform, Dataset<Row> ds) {
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(RESULT_TABLE_NAME, ds);
+ }
+ }
+
+ public static void sinkProcess(SparkEnvironment environment, BaseSparkSink<?> sink, Dataset<Row> ds) {
+ Dataset<Row> fromDs;
+ Config config = sink.getConfig();
+ if (config.hasPath(SparkBatchExecution.SOURCE_TABLE_NAME)) {
+ String sourceTableName = config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ fromDs = environment.getSparkSession().read().table(sourceTableName);
+ } else {
+ fromDs = ds;
+ }
+ sink.output(fromDs, environment);
+ }
+
+ private final SparkEnvironment environment;
+
+ private Config config = ConfigFactory.empty();
+
+ public SparkBatchExecution(SparkEnvironment environment) {
+ this.environment = environment;
+ }
Review comment:
[7402da4](https://github.com/apache/incubator-seatunnel/pull/1188/commits/7402da4b0359459912acdd72bf7714491bf89f99) had been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
yx91490 commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800131606
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/SparkEnvironment.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.env.RuntimeEnv;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.streaming.Seconds;
+import org.apache.spark.streaming.StreamingContext;
+
+public class SparkEnvironment implements RuntimeEnv {
+
+ private final long sparkStreamingDuration = 5;
Review comment:
Idea warned with `Field can be converted to a local variable`, or can add a `static` modifier.
besides, better to renamed to `DEFAULT_SPARK_STREAMING_DURATION`
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/BaseSparkSink.java
##########
@@ -14,23 +14,38 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.seatunnel.spark
-import org.apache.seatunnel.apis.BaseSink
-import org.apache.seatunnel.shade.com.typesafe.config.{Config, ConfigFactory}
-import org.apache.spark.sql.{Dataset, Row}
+package org.apache.seatunnel.spark;
+
+import org.apache.seatunnel.apis.BaseSink;
+import org.apache.seatunnel.common.config.CheckResult;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
/**
* a base interface indicates a sink plugin running on Spark.
*/
-trait BaseSparkSink[OUT] extends BaseSink[SparkEnvironment] {
+public abstract class BaseSparkSink<OUT> implements BaseSink<SparkEnvironment> {
+
+ protected Config config = ConfigFactory.empty();
- protected var config: Config = ConfigFactory.empty()
+ @Override
+ public void setConfig(Config config) {
+ this.config = config;
+ }
- override def setConfig(config: Config): Unit = this.config = config
+ @Override
+ public Config getConfig() {
+ return config;
+ }
- override def getConfig: Config = config
+ public abstract CheckResult checkConfig();
Review comment:
seems no diffrerent with parent class's method, can be removed.
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchExecution.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.config.ConfigRuntimeException;
+import org.apache.seatunnel.env.Execution;
+import org.apache.seatunnel.spark.BaseSparkSink;
+import org.apache.seatunnel.spark.BaseSparkSource;
+import org.apache.seatunnel.spark.BaseSparkTransform;
+import org.apache.seatunnel.spark.SparkEnvironment;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import java.util.List;
+
+public class SparkBatchExecution implements Execution<SparkBatchSource, BaseSparkTransform, SparkBatchSink> {
+
+ public static final String SOURCE_TABLE_NAME = "source_table_name";
+ public static final String RESULT_TABLE_NAME = "result_table_name";
+
+ public static void registerTempView(String tableName, Dataset<Row> ds) {
Review comment:
keep same to change to private modifier
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchExecution.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.config.ConfigRuntimeException;
+import org.apache.seatunnel.env.Execution;
+import org.apache.seatunnel.spark.BaseSparkSink;
+import org.apache.seatunnel.spark.BaseSparkSource;
+import org.apache.seatunnel.spark.BaseSparkTransform;
+import org.apache.seatunnel.spark.SparkEnvironment;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import java.util.List;
+
+public class SparkBatchExecution implements Execution<SparkBatchSource, BaseSparkTransform, SparkBatchSink> {
+
+ public static final String SOURCE_TABLE_NAME = "source_table_name";
+ public static final String RESULT_TABLE_NAME = "result_table_name";
+
+ public static void registerTempView(String tableName, Dataset<Row> ds) {
+ ds.createOrReplaceGlobalTempView(tableName);
+ }
+
+ public static void registerInputTempView(BaseSparkSource<Dataset<Row>> source, SparkEnvironment environment) {
+ Config config = source.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ String tableName = config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(tableName, source.getData(environment));
+ } else {
+ throw new ConfigRuntimeException(
+ "Plugin[" + source.getClass().getName() + "] must be registered as dataset/table, please set \"result_table_name\" config");
+ }
+ }
+
+ public static Dataset<Row> transformProcess(SparkEnvironment environment, BaseSparkTransform transform, Dataset<Row> ds) {
+ Dataset<Row> fromDs;
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.SOURCE_TABLE_NAME)) {
+ String sourceTableName = config.getString(SparkBatchExecution.SOURCE_TABLE_NAME);
+ fromDs = environment.getSparkSession().read().table(sourceTableName);
+ } else {
+ fromDs = ds;
+ }
+ return transform.process(fromDs, environment);
+ }
+
+ public static void registerTransformTempView(BaseSparkTransform transform, Dataset<Row> ds) {
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(RESULT_TABLE_NAME, ds);
Review comment:
there is a bug about tableName.
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchSink.java
##########
@@ -14,12 +14,17 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.seatunnel.spark.batch
-import org.apache.seatunnel.spark.BaseSparkSink
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.spark.BaseSparkSink;
+
+import scala.Unit;
/**
* a SparkBatchSink plugin will write data to other system
* using Spark DataSet API.
*/
-trait SparkBatchSink extends BaseSparkSink[Unit] {}
+@SuppressWarnings("PMD.AbstractClassShouldStartWithAbstractNamingRule")
+public abstract class SparkBatchSink extends BaseSparkSink<Unit> {
Review comment:
better to change `<Unit>` to `<Void>`
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchExecution.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.config.ConfigRuntimeException;
+import org.apache.seatunnel.env.Execution;
+import org.apache.seatunnel.spark.BaseSparkSink;
+import org.apache.seatunnel.spark.BaseSparkSource;
+import org.apache.seatunnel.spark.BaseSparkTransform;
+import org.apache.seatunnel.spark.SparkEnvironment;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import java.util.List;
+
+public class SparkBatchExecution implements Execution<SparkBatchSource, BaseSparkTransform, SparkBatchSink> {
+
+ public static final String SOURCE_TABLE_NAME = "source_table_name";
+ public static final String RESULT_TABLE_NAME = "result_table_name";
+
+ public static void registerTempView(String tableName, Dataset<Row> ds) {
+ ds.createOrReplaceGlobalTempView(tableName);
+ }
+
+ public static void registerInputTempView(BaseSparkSource<Dataset<Row>> source, SparkEnvironment environment) {
+ Config config = source.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ String tableName = config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(tableName, source.getData(environment));
+ } else {
+ throw new ConfigRuntimeException(
+ "Plugin[" + source.getClass().getName() + "] must be registered as dataset/table, please set \"result_table_name\" config");
+ }
+ }
+
+ public static Dataset<Row> transformProcess(SparkEnvironment environment, BaseSparkTransform transform, Dataset<Row> ds) {
+ Dataset<Row> fromDs;
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.SOURCE_TABLE_NAME)) {
+ String sourceTableName = config.getString(SparkBatchExecution.SOURCE_TABLE_NAME);
+ fromDs = environment.getSparkSession().read().table(sourceTableName);
+ } else {
+ fromDs = ds;
+ }
+ return transform.process(fromDs, environment);
+ }
+
+ public static void registerTransformTempView(BaseSparkTransform transform, Dataset<Row> ds) {
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(RESULT_TABLE_NAME, ds);
+ }
+ }
+
+ public static void sinkProcess(SparkEnvironment environment, BaseSparkSink<?> sink, Dataset<Row> ds) {
+ Dataset<Row> fromDs;
+ Config config = sink.getConfig();
+ if (config.hasPath(SparkBatchExecution.SOURCE_TABLE_NAME)) {
+ String sourceTableName = config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ fromDs = environment.getSparkSession().read().table(sourceTableName);
+ } else {
+ fromDs = ds;
+ }
+ sink.output(fromDs, environment);
+ }
+
+ private final SparkEnvironment environment;
+
+ private Config config = ConfigFactory.empty();
+
+ public SparkBatchExecution(SparkEnvironment environment) {
+ this.environment = environment;
+ }
Review comment:
better to moved to the top of the class.
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/SparkEnvironment.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.env.RuntimeEnv;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.streaming.Seconds;
+import org.apache.spark.streaming.StreamingContext;
+
+public class SparkEnvironment implements RuntimeEnv {
+
+ private final long sparkStreamingDuration = 5;
+
+ private SparkSession sparkSession;
+
+ private StreamingContext streamingContext;
+
+ private Config config;
Review comment:
different with origin scala version, which initialized with `ConfigFactory.empty()`
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchExecution.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.config.ConfigRuntimeException;
+import org.apache.seatunnel.env.Execution;
+import org.apache.seatunnel.spark.BaseSparkSink;
+import org.apache.seatunnel.spark.BaseSparkSource;
+import org.apache.seatunnel.spark.BaseSparkTransform;
+import org.apache.seatunnel.spark.SparkEnvironment;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import java.util.List;
+
+public class SparkBatchExecution implements Execution<SparkBatchSource, BaseSparkTransform, SparkBatchSink> {
+
+ public static final String SOURCE_TABLE_NAME = "source_table_name";
+ public static final String RESULT_TABLE_NAME = "result_table_name";
Review comment:
keep same to change to private modifier, or use Plugin.RESULT_TABLE_NAME.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030667153
> @TyrantLucifer IMO, you can use the `git merge dev` command to resolve code conflicts, no need to create a new PR
OK, I will try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800156258
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchExecution.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.config.ConfigRuntimeException;
+import org.apache.seatunnel.env.Execution;
+import org.apache.seatunnel.spark.BaseSparkSink;
+import org.apache.seatunnel.spark.BaseSparkSource;
+import org.apache.seatunnel.spark.BaseSparkTransform;
+import org.apache.seatunnel.spark.SparkEnvironment;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import java.util.List;
+
+public class SparkBatchExecution implements Execution<SparkBatchSource, BaseSparkTransform, SparkBatchSink> {
+
+ public static final String SOURCE_TABLE_NAME = "source_table_name";
+ public static final String RESULT_TABLE_NAME = "result_table_name";
+
+ public static void registerTempView(String tableName, Dataset<Row> ds) {
Review comment:
In the Scala version, these variables and methods are accessible to classes in the same parent package. If change them to private in Java, other classes will not be able to access them, which is why I chose to change them to public


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800184050
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchExecution.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.common.config.ConfigRuntimeException;
+import org.apache.seatunnel.env.Execution;
+import org.apache.seatunnel.spark.BaseSparkSink;
+import org.apache.seatunnel.spark.BaseSparkSource;
+import org.apache.seatunnel.spark.BaseSparkTransform;
+import org.apache.seatunnel.spark.SparkEnvironment;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import java.util.List;
+
+public class SparkBatchExecution implements Execution<SparkBatchSource, BaseSparkTransform, SparkBatchSink> {
+
+ public static final String SOURCE_TABLE_NAME = "source_table_name";
+ public static final String RESULT_TABLE_NAME = "result_table_name";
+
+ public static void registerTempView(String tableName, Dataset<Row> ds) {
+ ds.createOrReplaceGlobalTempView(tableName);
+ }
+
+ public static void registerInputTempView(BaseSparkSource<Dataset<Row>> source, SparkEnvironment environment) {
+ Config config = source.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ String tableName = config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(tableName, source.getData(environment));
+ } else {
+ throw new ConfigRuntimeException(
+ "Plugin[" + source.getClass().getName() + "] must be registered as dataset/table, please set \"result_table_name\" config");
+ }
+ }
+
+ public static Dataset<Row> transformProcess(SparkEnvironment environment, BaseSparkTransform transform, Dataset<Row> ds) {
+ Dataset<Row> fromDs;
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.SOURCE_TABLE_NAME)) {
+ String sourceTableName = config.getString(SparkBatchExecution.SOURCE_TABLE_NAME);
+ fromDs = environment.getSparkSession().read().table(sourceTableName);
+ } else {
+ fromDs = ds;
+ }
+ return transform.process(fromDs, environment);
+ }
+
+ public static void registerTransformTempView(BaseSparkTransform transform, Dataset<Row> ds) {
+ Config config = transform.getConfig();
+ if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
+ config.getString(SparkBatchExecution.RESULT_TABLE_NAME);
+ registerTempView(RESULT_TABLE_NAME, ds);
Review comment:
[7402da4](https://github.com/apache/incubator-seatunnel/pull/1188/commits/7402da4b0359459912acdd72bf7714491bf89f99) had been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030797538
> @TyrantLucifer It seems that you omitted some scala files, like SparkStreamingExecution, SparkStreamingSink, SparkStreamingSource.
Well, I found a compilation error when I tried to convert the stream-related API, as shown below

The specific code is as follows:
`SparkStreamingSource.java`
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.spark.stream;
import org.apache.seatunnel.spark.BaseSparkSource;
import org.apache.seatunnel.spark.SparkEnvironment;
import org.apache.spark.rdd.RDD;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.streaming.dstream.DStream;
import scala.runtime.BoxedUnit;
/**
* a SparkStreamingSource plugin will read data from other system
* using Spark Streaming API.
*/
@SuppressWarnings("PMD.AbstractClassShouldStartWithAbstractNamingRule")
public abstract class SparkStreamingSource<T> extends BaseSparkSource<DStream<T>> {
public void beforeOutput() {
}
public void afterOutput() {
}
public abstract Dataset<Row> rdd2dataset(SparkSession sparkSession, RDD<T> rdd);
public void start(SparkEnvironment sparkEnvironment, SparkStreamingDataHandler handler) {
getData(sparkEnvironment).foreachRDD(rdd -> {
Dataset<Row> dataset = rdd2dataset(sparkEnvironment.getSparkSession(), rdd);
handler.handle(dataset);
return BoxedUnit.UNIT;
});
}
}
```
`SparkStreamingSink.java`
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.spark.stream;
import org.apache.seatunnel.spark.BaseSparkSink;
/**
* a SparkStreamingSink plugin will write data to other system
* using Spark Streaming API.
*/
@SuppressWarnings("PMD.AbstractClassShouldStartWithAbstractNamingRule")
public abstract class SparkStreamingSink extends BaseSparkSink<Void> {
}
```
`SparkStreamingDataHandler.java`
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.spark.stream;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
@FunctionalInterface
public interface SparkStreamingDataHandler {
void handle(Dataset<Row> dataset);
}
```
`SparkStreamingExecution.java`
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.spark.stream;
import org.apache.seatunnel.common.config.CheckResult;
import org.apache.seatunnel.env.Execution;
import org.apache.seatunnel.spark.BaseSparkSink;
import org.apache.seatunnel.spark.BaseSparkSource;
import org.apache.seatunnel.spark.BaseSparkTransform;
import org.apache.seatunnel.spark.SparkEnvironment;
import org.apache.seatunnel.spark.batch.SparkBatchExecution;
import org.apache.seatunnel.shade.com.typesafe.config.Config;
import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.streaming.StreamingContext;
import java.util.List;
public class SparkStreamingExecution implements Execution<BaseSparkSource<?>, BaseSparkTransform, BaseSparkSink<?>> {
private final SparkEnvironment sparkEnvironment;
private Config config = ConfigFactory.empty();
public SparkStreamingExecution(SparkEnvironment sparkEnvironment) {
this.sparkEnvironment = sparkEnvironment;
}
@Override
public void start(List<BaseSparkSource<?>> sources, List<BaseSparkTransform> transforms, List<BaseSparkSink<?>> sinks) {
SparkStreamingSource<?> source = (SparkStreamingSource<?>) sources.get(0);
List<BaseSparkSource<?>> subSources = sources.subList(1, sources.size());
sources.subList(1, sources.size()).forEach(s -> {
SparkBatchExecution.registerInputTempView((BaseSparkSource<Dataset<Row>>) s, sparkEnvironment);
});
source.start(sparkEnvironment, dataset -> {
Config config = source.getConfig();
if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
SparkBatchExecution.registerTempView(config.getString(SparkBatchExecution.RESULT_TABLE_NAME), dataset);
}
Dataset<Row> ds = dataset;
for (BaseSparkTransform transform : transforms) {
if (ds.head().size() > 0) {
ds = SparkBatchExecution.transformProcess(sparkEnvironment, transform, ds);
SparkBatchExecution.registerTransformTempView(transform, ds);
}
}
source.beforeOutput();
if (ds.head().length() > 0) {
for (BaseSparkSink<?> sink : sinks) {
SparkBatchExecution.sinkProcess(sparkEnvironment, sink, ds);
}
}
source.afterOutput();
StreamingContext streamingContext = sparkEnvironment.getStreamingContext();
streamingContext.start();
streamingContext.awaitTermination();
});
}
@Override
public void setConfig(Config config) {
this.config = config;
}
@Override
public Config getConfig() {
return this.config;
}
@Override
public CheckResult checkConfig() {
return CheckResult.success();
}
@Override
public void prepare(Void prepareEnv) {
}
}
```
I think it should be compilation error caused by using spark-streaming API of Scala version in Java. I have tried to modify RDD and DStream to Java version, but the subsequent data sources connected in stream form are all developed in Scala. I don't know how to solve this problem. In order not to affect other functions, I did not submit this part of the code. Do you have any other good solutions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
yx91490 commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030830449
It is quiet difficult, seems need to convert scala API to java API(for example, DStream to JavaDStream ...), may cause too mach problems when scala call java in plugins. I have no other better suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs merged pull request #1188: [Feature] [api] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
CalvinKirs merged pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
yx91490 commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1031399341
good job, LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030663353
@asdf2014 @yx91490 In the process of this pr review, #1190 had been changed some base api codes, so my changes comes after #1190. If necessary, I may need to reopen a PR to resolve these conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800183737
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/BaseSparkSink.java
##########
@@ -14,23 +14,38 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.seatunnel.spark
-import org.apache.seatunnel.apis.BaseSink
-import org.apache.seatunnel.shade.com.typesafe.config.{Config, ConfigFactory}
-import org.apache.spark.sql.{Dataset, Row}
+package org.apache.seatunnel.spark;
+
+import org.apache.seatunnel.apis.BaseSink;
+import org.apache.seatunnel.common.config.CheckResult;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
/**
* a base interface indicates a sink plugin running on Spark.
*/
-trait BaseSparkSink[OUT] extends BaseSink[SparkEnvironment] {
+public abstract class BaseSparkSink<OUT> implements BaseSink<SparkEnvironment> {
+
+ protected Config config = ConfigFactory.empty();
- protected var config: Config = ConfigFactory.empty()
+ @Override
+ public void setConfig(Config config) {
+ this.config = config;
+ }
- override def setConfig(config: Config): Unit = this.config = config
+ @Override
+ public Config getConfig() {
+ return config;
+ }
- override def getConfig: Config = config
+ public abstract CheckResult checkConfig();
Review comment:
[4d57c66](https://github.com/apache/incubator-seatunnel/pull/1188/commits/4d57c66251bd3e72f5c8f6e0748c20fab6a72a92) had beeb resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
yx91490 commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030768564
@TyrantLucifer It seems that you omitted some scala files, like SparkStreamingExecution, SparkStreamingSink, SparkStreamingSource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030832411
> It is quiet difficult, seems need to convert scala API to java API(for example, DStream to JavaDStream ...), may cause too much problems when scala call java in plugins. I have no other better suggestions.
OK, then I will only submit the question mentioned above, the version of spark streaming api is temporarily using the Scala version, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800183854
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/SparkEnvironment.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.env.RuntimeEnv;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.streaming.Seconds;
+import org.apache.spark.streaming.StreamingContext;
+
+public class SparkEnvironment implements RuntimeEnv {
+
+ private final long sparkStreamingDuration = 5;
+
+ private SparkSession sparkSession;
+
+ private StreamingContext streamingContext;
+
+ private Config config;
Review comment:
[728a64a](https://github.com/apache/incubator-seatunnel/pull/1188/commits/728a64a120fb6f9ed800b228bb223fbead4246a0) had been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800183737
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/BaseSparkSink.java
##########
@@ -14,23 +14,38 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.seatunnel.spark
-import org.apache.seatunnel.apis.BaseSink
-import org.apache.seatunnel.shade.com.typesafe.config.{Config, ConfigFactory}
-import org.apache.spark.sql.{Dataset, Row}
+package org.apache.seatunnel.spark;
+
+import org.apache.seatunnel.apis.BaseSink;
+import org.apache.seatunnel.common.config.CheckResult;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
/**
* a base interface indicates a sink plugin running on Spark.
*/
-trait BaseSparkSink[OUT] extends BaseSink[SparkEnvironment] {
+public abstract class BaseSparkSink<OUT> implements BaseSink<SparkEnvironment> {
+
+ protected Config config = ConfigFactory.empty();
- protected var config: Config = ConfigFactory.empty()
+ @Override
+ public void setConfig(Config config) {
+ this.config = config;
+ }
- override def setConfig(config: Config): Unit = this.config = config
+ @Override
+ public Config getConfig() {
+ return config;
+ }
- override def getConfig: Config = config
+ public abstract CheckResult checkConfig();
Review comment:
[4d57c66](https://github.com/apache/incubator-seatunnel/pull/1188/commits/4d57c66251bd3e72f5c8f6e0748c20fab6a72a92) had been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #1188: [Feature] [api] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1033366784
> Sorry I don't have permission to merge, you can ask @CalvinKirs for help.
Let's we waiting CI. thx your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on pull request #1188: [Feature] [api] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
yx91490 commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1033362657
Sorry I don't have permission to merge, you can ask @CalvinKirs for help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [Feature] [api] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1034258511
> > Sorry I don't have permission to merge, you can ask @CalvinKirs for help.
>
> Let's we waiting CI. thx your review.
Hi @CalvinKirs CI had been done, pls review, thx you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [Feature] [api] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1033257261
> good job, LGTM!
Hi @yx91490, If you have no problem here, what we should do next? I see this pr had not been updated for two days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 edited a comment on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
yx91490 edited a comment on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030830449
It is quiet difficult, seems need to convert scala API to java API(for example, DStream to JavaDStream ...), may cause too much problems when scala call java in plugins. I have no other better suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] asdf2014 commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
asdf2014 commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030666763
@TyrantLucifer IMO, you can use the `git merge dev` command to resolve code conflicts, no need to create a new PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800183902
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/SparkEnvironment.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.spark;
+
+import org.apache.seatunnel.common.config.CheckResult;
+import org.apache.seatunnel.env.RuntimeEnv;
+
+import org.apache.seatunnel.shade.com.typesafe.config.Config;
+
+import org.apache.spark.SparkConf;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.streaming.Seconds;
+import org.apache.spark.streaming.StreamingContext;
+
+public class SparkEnvironment implements RuntimeEnv {
+
+ private final long sparkStreamingDuration = 5;
Review comment:
[728a64a](https://github.com/apache/incubator-seatunnel/pull/1188/commits/728a64a120fb6f9ed800b228bb223fbead4246a0) had been resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a change in pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on a change in pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#discussion_r800182317
##########
File path: seatunnel-apis/seatunnel-api-spark/src/main/java/org/apache/seatunnel/spark/batch/SparkBatchSink.java
##########
@@ -14,12 +14,17 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.seatunnel.spark.batch
-import org.apache.seatunnel.spark.BaseSparkSink
+package org.apache.seatunnel.spark.batch;
+
+import org.apache.seatunnel.spark.BaseSparkSink;
+
+import scala.Unit;
/**
* a SparkBatchSink plugin will write data to other system
* using Spark DataSet API.
*/
-trait SparkBatchSink extends BaseSparkSink[Unit] {}
+@SuppressWarnings("PMD.AbstractClassShouldStartWithAbstractNamingRule")
+public abstract class SparkBatchSink extends BaseSparkSink<Unit> {
Review comment:
Because of `org.apache.seatunnel.spark.sink.Kafka` is developed by scala, so the return type of output method is Unit, if I change it to Void it will be compiled error as below:

So I did not change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030838740
> It is quiet difficult, seems need to convert scala API to java API(for example, DStream to JavaDStream ...), may cause too much problems when scala call java in plugins. I have no other better suggestions.
Hi @yx91490, I had been resolved all the problems that you raised, please review, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer edited a comment on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer edited a comment on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030672781
> Hi @TyrantLucifer, thanks for your contribution, however there are some code conflicts here, would you please fix them first, thanks a lot
@asdf2014 @bobqiu I had been resolved the conflicts of this pr, please review. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer edited a comment on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer edited a comment on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030797538
> @TyrantLucifer It seems that you omitted some scala files, like SparkStreamingExecution, SparkStreamingSink, SparkStreamingSource.
Well, I found a compilation error when I tried to convert the stream-related API, as shown below

The specific code is as follows:
`SparkStreamingSource.java`
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.spark.stream;
import org.apache.seatunnel.spark.BaseSparkSource;
import org.apache.seatunnel.spark.SparkEnvironment;
import org.apache.spark.rdd.RDD;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.streaming.dstream.DStream;
import java.util.function.Consumer;
import scala.runtime.BoxedUnit;
/**
* a SparkStreamingSource plugin will read data from other system
* using Spark Streaming API.
*/
@SuppressWarnings("PMD.AbstractClassShouldStartWithAbstractNamingRule")
public abstract class SparkStreamingSource<T> extends BaseSparkSource<DStream<T>> {
public void beforeOutput() {
}
public void afterOutput() {
}
public abstract Dataset<Row> rdd2dataset(SparkSession sparkSession, RDD<T> rdd);
public void start(SparkEnvironment sparkEnvironment, Consumer<Dataset<Row>> handler) {
getData(sparkEnvironment).foreachRDD(rdd -> {
Dataset<Row> dataset = rdd2dataset(sparkEnvironment.getSparkSession(), rdd);
handler.accept(dataset);
return BoxedUnit.UNIT;
});
}
}
```
`SparkStreamingSink.java`
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.spark.stream;
import org.apache.seatunnel.spark.BaseSparkSink;
/**
* a SparkStreamingSink plugin will write data to other system
* using Spark Streaming API.
*/
@SuppressWarnings("PMD.AbstractClassShouldStartWithAbstractNamingRule")
public abstract class SparkStreamingSink extends BaseSparkSink<Void> {
}
```
`SparkStreamingExecution.java`
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.spark.stream;
import org.apache.seatunnel.common.config.CheckResult;
import org.apache.seatunnel.env.Execution;
import org.apache.seatunnel.spark.BaseSparkSink;
import org.apache.seatunnel.spark.BaseSparkSource;
import org.apache.seatunnel.spark.BaseSparkTransform;
import org.apache.seatunnel.spark.SparkEnvironment;
import org.apache.seatunnel.spark.batch.SparkBatchExecution;
import org.apache.seatunnel.shade.com.typesafe.config.Config;
import org.apache.seatunnel.shade.com.typesafe.config.ConfigFactory;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.streaming.StreamingContext;
import java.util.List;
public class SparkStreamingExecution implements Execution<BaseSparkSource<?>, BaseSparkTransform, BaseSparkSink<?>> {
private final SparkEnvironment sparkEnvironment;
private Config config = ConfigFactory.empty();
public SparkStreamingExecution(SparkEnvironment sparkEnvironment) {
this.sparkEnvironment = sparkEnvironment;
}
@Override
public void start(List<BaseSparkSource<?>> sources, List<BaseSparkTransform> transforms, List<BaseSparkSink<?>> sinks) {
SparkStreamingSource<?> source = (SparkStreamingSource<?>) sources.get(0);
List<BaseSparkSource<?>> subSources = sources.subList(1, sources.size());
sources.subList(1, sources.size()).forEach(s -> {
SparkBatchExecution.registerInputTempView((BaseSparkSource<Dataset<Row>>) s, sparkEnvironment);
});
source.start(sparkEnvironment, dataset -> {
Config config = source.getConfig();
if (config.hasPath(SparkBatchExecution.RESULT_TABLE_NAME)) {
SparkBatchExecution.registerTempView(config.getString(SparkBatchExecution.RESULT_TABLE_NAME), dataset);
}
Dataset<Row> ds = dataset;
for (BaseSparkTransform transform : transforms) {
if (ds.head().size() > 0) {
ds = SparkBatchExecution.transformProcess(sparkEnvironment, transform, ds);
SparkBatchExecution.registerTransformTempView(transform, ds);
}
}
source.beforeOutput();
if (ds.head().length() > 0) {
for (BaseSparkSink<?> sink : sinks) {
SparkBatchExecution.sinkProcess(sparkEnvironment, sink, ds);
}
}
source.afterOutput();
StreamingContext streamingContext = sparkEnvironment.getStreamingContext();
streamingContext.start();
streamingContext.awaitTermination();
});
}
@Override
public void setConfig(Config config) {
this.config = config;
}
@Override
public Config getConfig() {
return this.config;
}
@Override
public CheckResult checkConfig() {
return CheckResult.success();
}
@Override
public void prepare(Void prepareEnv) {
}
}
```
I think it should be compilation error caused by using spark-streaming API of Scala version in Java. I have tried to modify RDD and DStream to Java version, but the subsequent data sources connected in stream form are all developed in Scala. I don't know how to solve this problem. In order not to affect other functions, I did not submit this part of the code. Do you have any other good solutions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #1188: [SeaTunnel#1142] Translate spark apis from scala to java
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on pull request #1188:
URL: https://github.com/apache/incubator-seatunnel/pull/1188#issuecomment-1030672781
> Hi @TyrantLucifer, thanks for your contribution, however there are some code conflicts here, would you please fix them first, thanks a lot
@asdf2014 I had been resolved the conflicts of this pr, please review. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org