You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by M Singh <ma...@yahoo.com.INVALID> on 2022/03/10 16:12:57 UTC
Apache Seatunnel - JDBC Source not showing results in console sink
Hi Folks:
I am trying to write a very simple seatunnel configuration which gets data from database (included below - it only has a jdbc source and console sink). However, the application terminates without any results. I am using Seatunnel 2.0.5-SNAPSHOT for my tests.
I have added the jdbc dependencies as required and have tried to add source_table_name in the console sink but still am not able to figure out what could be the reason that there are no console results.
If I use the fake source config example - it does work and shows output in the console.
Can you please let me know how to debug this and what could the issue ?
Thanks
I am using configuration for Jdbc Source mentioned at https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Jdbc
Here is the additional jdbc dependencies for the included config:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.49</version> <scope>runtime</scope> </dependency>
Here is the configuration seatunnel config (it's based on the fake example config https://seatunnel.apache.org/docs/flink/configuration/ConfigExamples/)
env { # You can set flink configuration here execution.parallelism = 1 #execution.checkpoint.interval = 10000 #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"}
source { # This is a example source plugin **only for test and demonstrate the feature source plugin** JdbcSource { driver = "com.mysql.jdbc.Driver" url = "jdbc:mysql://localhost/mydatabase?useSSL=false" username = "root" password = "" query = "select * from people" result_table_name = "people" field_name = "name,age,phone" } # If you would like to get more information about how to configure seatunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake}
transform {## # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,# # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql}
sink { ConsoleSink {# source_table_name = "people" }
# If you would like to get more information about how to configure seatunnel and see full list of sink plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console}
回复：Apache Seatunnel - JDBC Source not showing results in console sink
Posted by 冰封十字 <16...@qq.com.INVALID>.
you can import seatunnel project into IDE by run mvn clean package -DskipTests -Dfast, then try to run LocalFlinkExample located on seatunnel/seatunnel-examples/seatunnel-flink-examples/ module.
seatunnel-flink-examples provides a demo for run seatunnel job in IDE, reproduce you scene, then you can debug freely.
------------------ 原始邮件 ------------------
发件人: "dev" <mans2singh@yahoo.com.INVALID>;
发送时间: 2022年3月11日(星期五) 凌晨0:12
收件人: "dev@seatunnel.apache.org"<dev@seatunnel.apache.org>;
主题: Apache Seatunnel - JDBC Source not showing results in console sink
Hi Folks:
I am trying to write a very simple seatunnel configuration which gets data from database (included below - it only has a jdbc source and console sink). However, the application terminates without any results. I am using Seatunnel 2.0.5-SNAPSHOT for my tests.
I have added the jdbc dependencies as required and have tried to add source_table_name in the console sink but still am not able to figure out what could be the reason that there are no console results.
If I use the fake source config example - it does work and shows output in the console.
Can you please let me know how to debug this and what could the issue ?
Thanks
I am using configuration for Jdbc Source mentioned at https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Jdbc
Here is the additional jdbc dependencies for the included config:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.49</version> <scope>runtime</scope> </dependency>
Here is the configuration seatunnel config (it's based on the fake example config https://seatunnel.apache.org/docs/flink/configuration/ConfigExamples/)
env { # You can set flink configuration here execution.parallelism = 1 #execution.checkpoint.interval = 10000 #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"}
source { # This is a example source plugin **only for test and demonstrate the feature source plugin** JdbcSource { driver = "com.mysql.jdbc.Driver" url = "jdbc:mysql://localhost/mydatabase?useSSL=false" username = "root" password = "" query = "select * from people" result_table_name = "people" field_name = "name,age,phone" } # If you would like to get more information about how to configure seatunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake}
transform {## # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,# # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql}
sink { ConsoleSink {# source_table_name = "people" }
# If you would like to get more information about how to configure seatunnel and see full list of sink plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console}
Re: Apache Seatunnel - JDBC Source not showing results in console sink
Posted by M Singh <ma...@yahoo.com.INVALID>.
Thanks Wenjun for your help.
On Thursday, March 10, 2022, 08:21:43 PM EST, wenjun <we...@apache.org> wrote:
Hi,
I also find this problem, I guess this might be caused by right now
the `seatunnel-flink-jdbc` source will use the batch mode to execute
the job.
And I find in `org.apache.seatunnel.flink.sink.ConsoleSink.java` it
will not print the DataSet in outputBatch method, this might be a BUG?
```
@Override
public DataSink<Row> outputBatch(FlinkEnvironment env, DataSet<Row>
rowDataSet) {
return rowDataSet.output(this);
}
@Override
public DataStreamSink<Row> outputStream(FlinkEnvironment env,
DataStream<Row> dataStream) {
return dataStream.print();
}
```
You can simply add print function in outputBatch, then you will see
the result in console.
```
@Override
public DataSink<Row> outputBatch(FlinkEnvironment env, DataSet<Row>
rowDataSet) {
try {
rowDataSet.print();
} catch (Exception ex) {
throw new RuntimeException("Print result into console error", ex);
}
return rowDataSet.output(this);
}
```
Best,
Wenjun
On Fri, Mar 11, 2022 at 12:13 AM M Singh <ma...@yahoo.com.invalid> wrote:
>
> Hi Folks:
> I am trying to write a very simple seatunnel configuration which gets data from database (included below - it only has a jdbc source and console sink). However, the application terminates without any results. I am using Seatunnel 2.0.5-SNAPSHOT for my tests.
> I have added the jdbc dependencies as required and have tried to add source_table_name in the console sink but still am not able to figure out what could be the reason that there are no console results.
> If I use the fake source config example - it does work and shows output in the console.
> Can you please let me know how to debug this and what could the issue ?
> Thanks
>
> I am using configuration for Jdbc Source mentioned at https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Jdbc
> Here is the additional jdbc dependencies for the included config:
> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.49</version> <scope>runtime</scope> </dependency>
>
> Here is the configuration seatunnel config (it's based on the fake example config https://seatunnel.apache.org/docs/flink/configuration/ConfigExamples/)
> env { # You can set flink configuration here execution.parallelism = 1 #execution.checkpoint.interval = 10000 #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"}
> source { # This is a example source plugin **only for test and demonstrate the feature source plugin** JdbcSource { driver = "com.mysql.jdbc.Driver" url = "jdbc:mysql://localhost/mydatabase?useSSL=false" username = "root" password = "" query = "select * from people" result_table_name = "people" field_name = "name,age,phone" } # If you would like to get more information about how to configure seatunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake}
> transform {## # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,# # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql}
> sink { ConsoleSink {# source_table_name = "people" }
> # If you would like to get more information about how to configure seatunnel and see full list of sink plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console}
>
Re: Apache Seatunnel - JDBC Source not showing results in console sink
Posted by wenjun <we...@apache.org>.
Hi,
I also find this problem, I guess this might be caused by right now
the `seatunnel-flink-jdbc` source will use the batch mode to execute
the job.
And I find in `org.apache.seatunnel.flink.sink.ConsoleSink.java` it
will not print the DataSet in outputBatch method, this might be a BUG?
```
@Override
public DataSink<Row> outputBatch(FlinkEnvironment env, DataSet<Row>
rowDataSet) {
return rowDataSet.output(this);
}
@Override
public DataStreamSink<Row> outputStream(FlinkEnvironment env,
DataStream<Row> dataStream) {
return dataStream.print();
}
```
You can simply add print function in outputBatch, then you will see
the result in console.
```
@Override
public DataSink<Row> outputBatch(FlinkEnvironment env, DataSet<Row>
rowDataSet) {
try {
rowDataSet.print();
} catch (Exception ex) {
throw new RuntimeException("Print result into console error", ex);
}
return rowDataSet.output(this);
}
```
Best,
Wenjun
On Fri, Mar 11, 2022 at 12:13 AM M Singh <ma...@yahoo.com.invalid> wrote:
>
> Hi Folks:
> I am trying to write a very simple seatunnel configuration which gets data from database (included below - it only has a jdbc source and console sink). However, the application terminates without any results. I am using Seatunnel 2.0.5-SNAPSHOT for my tests.
> I have added the jdbc dependencies as required and have tried to add source_table_name in the console sink but still am not able to figure out what could be the reason that there are no console results.
> If I use the fake source config example - it does work and shows output in the console.
> Can you please let me know how to debug this and what could the issue ?
> Thanks
>
> I am using configuration for Jdbc Source mentioned at https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Jdbc
> Here is the additional jdbc dependencies for the included config:
> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.49</version> <scope>runtime</scope> </dependency>
>
> Here is the configuration seatunnel config (it's based on the fake example config https://seatunnel.apache.org/docs/flink/configuration/ConfigExamples/)
> env { # You can set flink configuration here execution.parallelism = 1 #execution.checkpoint.interval = 10000 #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"}
> source { # This is a example source plugin **only for test and demonstrate the feature source plugin** JdbcSource { driver = "com.mysql.jdbc.Driver" url = "jdbc:mysql://localhost/mydatabase?useSSL=false" username = "root" password = "" query = "select * from people" result_table_name = "people" field_name = "name,age,phone" } # If you would like to get more information about how to configure seatunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake}
> transform {## # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,# # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql}
> sink { ConsoleSink {# source_table_name = "people" }
> # If you would like to get more information about how to configure seatunnel and see full list of sink plugins, # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console}
>