You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/01/06 09:00:48 UTC

[GitHub] [incubator-seatunnel] felix-thinkingdata opened a new pull request #960: [Feature][seatunnel-test] flink local test

felix-thinkingdata opened a new pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960


    #955
   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
   
     - Name the pull request in the form "[SeaTunnel #XXXX] [component] Title of the pull request", where *SeaTunnel #XXXX* should be replaced by the actual issue number.
   
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   
   -->
   
   ## Purpose of this pull request
   
   <!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for reason:
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] leo65535 commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
leo65535 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006395315


   +1 this is a good idea to instroduce a new module `seatunnel-examples` to help users to run the demo quickly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] leo65535 commented on pull request #960: [Feature][seatunnel-examples] flink local environment run quickly and debug locally developed code easily

Posted by GitBox <gi...@apache.org>.
leo65535 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007111681


   > done ,Is the current title ok?
   
   💯 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006537743


   > > > Why not use docker to implement local test #815 ?
   > > 
   > > 
   > > We always hope that users can quickly test in various environments. For developers, this is a good experience.
   > 
   > It's too expensive to create a new module, maybe add as a unit test instead?
   
   
   
   > > > Why not use docker to implement local test #815 ?
   > > 
   > > 
   > > We always hope that users can quickly test in various environments. For developers, this is a good experience.
   > 
   > It's too expensive to create a new module, maybe add as a unit test instead?
   
   Sounds good~ WDYT ?@felix-thinkingdata 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] leo65535 edited a comment on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
leo65535 edited a comment on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006393543


   hi @felix-thinkingdata how about rename `seatunnel-test` to `seatunnel-examples`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] leo65535 commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
leo65535 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006393543


   Seems we can rename `seatunnel-test` to `seatunnel-examples`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779445866



##########
File path: seatunnel-examples/seatunnel-flink-examples/pom.xml
##########
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>seatunnel-examples</artifactId>
+        <groupId>org.apache.seatunnel</groupId>
+        <version>2.0.5-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>seatunnel-flink-examples</artifactId>
+    <properties>
+        <flink.scope>compile</flink.scope>
+    </properties>
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.seatunnel</groupId>
+            <artifactId>seatunnel-core-flink</artifactId>
+            <version>${parent.version}</version>
+            <exclusions>
+                <exclusion>
+                    <artifactId>slf4j-log4j12</artifactId>
+                    <groupId>org.slf4j</groupId>
+                </exclusion>
+                <exclusion>
+                    <artifactId>log4j</artifactId>
+                    <groupId>log4j</groupId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <!--flink-->
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-java</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-simple</artifactId>
+            <version>1.7.25</version>
+        </dependency>

Review comment:
       If not added, the following exceptions will occur:
   `SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
   SLF4J: Defaulting to no-operation (NOP) logger implementation`
   
   
   
   I'll see if there's a better way




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006394087


   ![image](https://user-images.githubusercontent.com/59079269/148357350-7f4be232-c579-4b5f-b6c0-f8fbe05b70ac.png)
   
   
   > Seems we can rename `seatunnel-test` to `seatunnel-examples`
   
   ok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006523074


   @wuchunfu It's been fixed ,Can you review it for me? Thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] yx91490 commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
yx91490 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007091737


   > from my side, an independent may better, like other projects https://github.com/apache/spark/tree/master/examples https://github.com/apache/flink/tree/master/flink-examples https://github.com/apache/hudi/tree/master/hudi-examples
   
   It seems that these modules don't aim at local test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007101799


   > 
   
   The main solution is to allow developers to run quickly and debug locally developed code easily.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] yx91490 commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
yx91490 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006523195


   Why not use docker to implement local test https://github.com/apache/incubator-seatunnel/pull/815 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779463301



##########
File path: seatunnel-examples/seatunnel-flink-examples/pom.xml
##########
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>seatunnel-examples</artifactId>
+        <groupId>org.apache.seatunnel</groupId>
+        <version>2.0.5-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>seatunnel-flink-examples</artifactId>
+    <properties>
+        <flink.scope>compile</flink.scope>
+    </properties>
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.seatunnel</groupId>
+            <artifactId>seatunnel-core-flink</artifactId>
+            <version>${parent.version}</version>
+            <exclusions>
+                <exclusion>
+                    <artifactId>slf4j-log4j12</artifactId>
+                    <groupId>org.slf4j</groupId>
+                </exclusion>
+                <exclusion>
+                    <artifactId>log4j</artifactId>
+                    <groupId>log4j</groupId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <!--flink-->
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-java</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-simple</artifactId>
+            <version>1.7.25</version>
+        </dependency>

Review comment:
       Resolved ,the addition of log4j.properties




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006413981


   > > `SeaTunnel` supports `flink` and `spark` engine, it's better add two sub module under `seatunnel-examples`.
   > 
   > > `SeaTunnel` supports `flink` and `spark` engine, it's better add two sub module under `seatunnel-examples`.
   > 
   > Spark is not very familiar, SO I will look into it
   
   ![image](https://user-images.githubusercontent.com/59079269/148361833-22f3d237-6208-4e13-b59b-c7037740fef2.png)
   ![image](https://user-images.githubusercontent.com/59079269/148361878-dd682b7b-7865-49ea-81df-0c56cab71fd8.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] leo65535 commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
leo65535 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006399337


   `SeaTunnel` supports `flink` and `spark` engine, it's better add two sub module under `seatunnel-examples`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] leo65535 commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
leo65535 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007059697


   from my side, an independent may better, like other projects
   https://github.com/apache/spark/tree/master/examples
   https://github.com/apache/flink/tree/master/flink-examples
   https://github.com/apache/hudi/tree/master/hudi-examples
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007097321


   > > from my side, an independent may better, like other projects https://github.com/apache/spark/tree/master/examples https://github.com/apache/flink/tree/master/flink-examples https://github.com/apache/hudi/tree/master/hudi-examples
   > 
   > It seems that these modules don't aim at local test.
   
   Just a reference suggestion, let us return to the PR itself. What are your thoughts on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006525121


   > Why not use docker to implement local test #815 ?
   
   We always hope that users can quickly test in various environments. For developers, this is a good experience.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] CalvinKirs commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779982589



##########
File path: seatunnel-examples/seatunnel-flink-examples/src/main/resources/examples/fake_to_console.config
##########
@@ -0,0 +1,55 @@
+##
+## Licensed to the Apache Software Foundation (ASF) under one or more
+## contributor license agreements.  See the NOTICE file distributed with
+## this work for additional information regarding copyright ownership.
+## The ASF licenses this file to You under the Apache License, Version 2.0
+## (the "License"); you may not use this file except in compliance with
+## the License.  You may obtain a copy of the License at
+##
+##    http://www.apache.org/licenses/LICENSE-2.0

Review comment:
       Please keep the license format consistent




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007079613


   > from my side, an independent may better, like other projects https://github.com/apache/spark/tree/master/examples https://github.com/apache/flink/tree/master/flink-examples https://github.com/apache/hudi/tree/master/hudi-examples
   
   Is there anything else I need to change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007089205


   @leo65535 PTAL,thx ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779983467



##########
File path: seatunnel-examples/seatunnel-flink-examples/src/main/resources/examples/fake_to_console.config
##########
@@ -0,0 +1,55 @@
+##
+## Licensed to the Apache Software Foundation (ASF) under one or more
+## contributor license agreements.  See the NOTICE file distributed with
+## this work for additional information regarding copyright ownership.
+## The ASF licenses this file to You under the Apache License, Version 2.0
+## (the "License"); you may not use this file except in compliance with
+## the License.  You may obtain a copy of the License at
+##
+##    http://www.apache.org/licenses/LICENSE-2.0

Review comment:
       > Please keep the license format consistent
   
   done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] CalvinKirs merged pull request #960: [Feature][seatunnel-examples] flink local environment run quickly and debug locally developed code easily

Posted by GitBox <gi...@apache.org>.
CalvinKirs merged pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006400980


   > `SeaTunnel` supports `flink` and `spark` engine, it's better add two sub module under `seatunnel-examples`.
   
   
   
   > `SeaTunnel` supports `flink` and `spark` engine, it's better add two sub module under `seatunnel-examples`.
   
   Spark is not very familiar, SO I will look into it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] wuchunfu commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
wuchunfu commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779420552



##########
File path: seatunnel-examples/seatunnel-flink-examples/pom.xml
##########
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>seatunnel-examples</artifactId>
+        <groupId>org.apache.seatunnel</groupId>
+        <version>2.0.5-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>seatunnel-flink-examples</artifactId>
+    <properties>
+        <flink.scope>compile</flink.scope>
+    </properties>
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.seatunnel</groupId>
+            <artifactId>seatunnel-core-flink</artifactId>
+            <version>${parent.version}</version>
+            <exclusions>
+                <exclusion>
+                    <artifactId>slf4j-log4j12</artifactId>
+                    <groupId>org.slf4j</groupId>
+                </exclusion>
+                <exclusion>
+                    <artifactId>log4j</artifactId>
+                    <groupId>log4j</groupId>
+                </exclusion>
+            </exclusions>
+        </dependency>
+
+        <!--flink-->
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-java</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.flink</groupId>
+            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
+            <version>${flink.version}</version>
+            <scope>${flink.scope}</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-simple</artifactId>
+            <version>1.7.25</version>
+        </dependency>

Review comment:
       What is the purpose of adding this dependency separately?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779418533



##########
File path: seatunnel-examples/seatunnel-flink-examples/pom.xml
##########
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"

Review comment:
       > Please add license, thanks
   
   ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata removed a comment on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata removed a comment on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006600412


   > flink.streaming.conf.template
   
   You need to change Maven Scope to Runtime or compile to run properly.If put in unit test, need to change the scope of the entire project Maven, not very suitable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] yx91490 commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
yx91490 commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006528076


   > > Why not use docker to implement local test #815 ?
   > 
   > We always hope that users can quickly test in various environments. For developers, this is a good experience.
   
   It's too expensive to create a new module, maybe add as a unit test instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata edited a comment on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata edited a comment on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006394087


   
   > Seems we can rename `seatunnel-test` to `seatunnel-examples`
   
   ok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] leo65535 edited a comment on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
leo65535 edited a comment on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006395315


   +1 this is a good idea to introduce a new module `seatunnel-examples` to help users to run the demo quickly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] wuchunfu commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
wuchunfu commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779419948



##########
File path: seatunnel-examples/pom.xml
##########
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>seatunnel</artifactId>
+        <groupId>org.apache.seatunnel</groupId>
+        <version>2.0.5-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>seatunnel-examples</artifactId>
+    <packaging>pom</packaging>
+    <modules>
+        <module>seatunnel-flink-examples</module>
+    </modules>
+
+
+

Review comment:
       Please check it as a whole and remove useless spaces, thanks




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] wuchunfu commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
wuchunfu commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779418282



##########
File path: seatunnel-examples/seatunnel-flink-examples/pom.xml
##########
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"

Review comment:
       Please add license, thanks




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-examples] flink local environment run quickly and debug locally developed code easily

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1007103814


   > LGTM, please update the title of the patch
   
   done ,Is the current title ok?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] wuchunfu commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
wuchunfu commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779421669



##########
File path: seatunnel-examples/seatunnel-flink-examples/src/main/resources/examples/fake_to_console.config
##########
@@ -0,0 +1,39 @@
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+  # You can set flink configuration here
+  execution.parallelism = 1
+  #execution.checkpoint.interval = 10000
+  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+  # This is a example source plugin **only for test and demonstrate the feature source plugin**
+    FakeSourceStream {
+      result_table_name = "fake"
+      field_name = "name,age"
+    }
+
+  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+  # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/configuration/base
+}
+
+transform {
+    sql {
+      sql = "select name,age from fake"
+    }
+
+  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+  # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/configuration/base
+}
+
+sink {
+  ConsoleSink {}
+
+
+  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
+  # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/configuration/base
+}

Review comment:
       Do you want to enable the new official document address for the document address here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006600942


   > > > > Why not use docker to implement local test #815 ?
   > > > 
   > > > 
   > > > We always hope that users can quickly test in various environments. For developers, this is a good experience.
   > > 
   > > 
   > > It's too expensive to create a new module, maybe add as a unit test instead?
   > 
   > > > > Why not use docker to implement local test #815 ?
   > > > 
   > > > 
   > > > We always hope that users can quickly test in various environments. For developers, this is a good experience.
   > > 
   > > 
   > > It's too expensive to create a new module, maybe add as a unit test instead?
   > 
   > Sounds good~ WDYT ?@felix-thinkingdata
   You need to change Maven Scope to Runtime or compile to run properly.If put in unit test, need to change the scope of the entire project Maven, not very suitable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006600412


   > flink.streaming.conf.template
   
   You need to change Maven Scope to Runtime or compile to run properly.If put in unit test, need to change the scope of the entire project Maven, not very suitable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] RickyHuo commented on pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
RickyHuo commented on pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#issuecomment-1006529388


   `seatunnel-flink-examples/src/main/resources/examples/fake_to_console.config`  can replaced with `config/flink.streaming.conf.template`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779446912



##########
File path: seatunnel-examples/seatunnel-flink-examples/src/main/resources/examples/fake_to_console.config
##########
@@ -0,0 +1,39 @@
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+  # You can set flink configuration here
+  execution.parallelism = 1
+  #execution.checkpoint.interval = 10000
+  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+  # This is a example source plugin **only for test and demonstrate the feature source plugin**
+    FakeSourceStream {
+      result_table_name = "fake"
+      field_name = "name,age"
+    }
+
+  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+  # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/configuration/base
+}
+
+transform {
+    sql {
+      sql = "select name,age from fake"
+    }
+
+  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+  # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/configuration/base
+}
+
+sink {
+  ConsoleSink {}
+
+
+  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
+  # please go to https://interestinglab.github.io/seatunnel-docs/#/zh-cn/configuration/base
+}

Review comment:
       Let me change it. I copied it from the demo. I forgot to change it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-seatunnel] felix-thinkingdata commented on a change in pull request #960: [Feature][seatunnel-test] flink local test

Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on a change in pull request #960:
URL: https://github.com/apache/incubator-seatunnel/pull/960#discussion_r779449351



##########
File path: seatunnel-examples/pom.xml
##########
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>seatunnel</artifactId>
+        <groupId>org.apache.seatunnel</groupId>
+        <version>2.0.5-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>seatunnel-examples</artifactId>
+    <packaging>pom</packaging>
+    <modules>
+        <module>seatunnel-flink-examples</module>
+    </modules>
+
+
+

Review comment:
       
   
   > Please check it as a whole and remove useless spaces, thanks
   
   fix




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org