You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/04/15 10:06:54 UTC
[GitHub] [incubator-seatunnel] BenJFan opened a new issue, #1702: [Bug] [Connector] Use Console sink will submit two job in flink cluster
BenJFan opened a new issue, #1702:
URL: https://github.com/apache/incubator-seatunnel/issues/1702
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When use console sink, and submit job to flink cluster, client will sumbit two job.
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
env {
execution.parallelism = 2
}
source {
JdbcSource {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://localhost:3306/test?useUnicode=true&characterEncoding=utf8&useSSL=false"
username = "root"
password = "123456"
query = "SELECT int_type,char_type,bigint_type from test_test where 1=1"
result_table_name = "user"
partition_column = "int_type"
partition_upper_bound = 50
partition_lower_bound = 0
parallelism = 4
}
}
transform {
}
sink {
ConsoleSink {}
}
```
### Running Command
```shell
./bin/start-seatunnel-flink.sh --config ./config/flink.batch.conf
```
### Error Exception
```log
-
```
### Flink or Spark Version
Flink 1.13.6
### Java or Scala Version
Java8
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1702: [Bug] [Connector] Use Console sink will submit two job in flink cluster
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1702:
URL: https://github.com/apache/incubator-seatunnel/issues/1702#issuecomment-1100518858
@BenJFan I find this may be caused by we have added two `DataSink` in `ConsoleSink`? After change to below code, this bug has been fixed.
https://github.com/apache/incubator-seatunnel/blob/3a4ef4c60d8397fbf7ee7b16a06494072d3f0245/seatunnel-connectors/seatunnel-connectors-flink/seatunnel-connector-flink-console/src/main/java/org/apache/seatunnel/flink/console/sink/ConsoleSink.java#L47-L54
After change to
```java
@Override
public DataSink<Row> outputBatch(FlinkEnvironment env, DataSet<Row> rowDataSet) {
return rowDataSet.output(this);
}
@SuppressWarnings("RegexpSingleline")
@Override
public void writeRecord(Row record) {
if (limit <= 0) {
return;
}
System.out.println(record.toString());
limit--;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun closed issue #1702: [Bug] [Connector] Use Console sink will submit two job in flink cluster
Posted by GitBox <gi...@apache.org>.
ruanwenjun closed issue #1702: [Bug] [Connector] Use Console sink will submit two job in flink cluster
URL: https://github.com/apache/incubator-seatunnel/issues/1702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1702: [Bug] [Connector] Use Console sink will submit two job in flink cluster
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1702:
URL: https://github.com/apache/incubator-seatunnel/issues/1702#issuecomment-1100510873
Yes, I also find this, and this will happen in batch job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1702: [Bug] [Connector] Use Console sink will submit two job in flink cluster
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1702:
URL: https://github.com/apache/incubator-seatunnel/issues/1702#issuecomment-1100563457
> > @BenJFan I find this may be caused by we have added two `DataSink` in `ConsoleSink`?
> > https://github.com/apache/incubator-seatunnel/blob/3a4ef4c60d8397fbf7ee7b16a06494072d3f0245/seatunnel-connectors/seatunnel-connectors-flink/seatunnel-connector-flink-console/src/main/java/org/apache/seatunnel/flink/console/sink/ConsoleSink.java#L47-L54
> >
> > After change to below code, this bug has been fixed.
> > ```java
> > @Override
> > public DataSink<Row> outputBatch(FlinkEnvironment env, DataSet<Row> rowDataSet) {
> > return rowDataSet.output(this);
> > }
> >
> > @SuppressWarnings("RegexpSingleline")
> > @Override
> > public void writeRecord(Row record) {
> > if (limit <= 0) {
> > return;
> > }
> > System.out.println(record.toString());
> > limit--;
> > }
> > ```
>
> This method will make record print on TaskManager, not client side.
Yes, you are right, we cannot easily do this change. The main reason is that Flink will do `execute` in collect method, and this will conflict with our execute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1702: [Bug] [Connector] Use Console sink will submit two job in flink cluster
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1702:
URL: https://github.com/apache/incubator-seatunnel/issues/1702#issuecomment-1100525181
> @BenJFan I find this may be caused by we have added two `DataSink` in `ConsoleSink`?
>
> https://github.com/apache/incubator-seatunnel/blob/3a4ef4c60d8397fbf7ee7b16a06494072d3f0245/seatunnel-connectors/seatunnel-connectors-flink/seatunnel-connector-flink-console/src/main/java/org/apache/seatunnel/flink/console/sink/ConsoleSink.java#L47-L54
>
> After change to below code, this bug has been fixed.
>
> ```java
> @Override
> public DataSink<Row> outputBatch(FlinkEnvironment env, DataSet<Row> rowDataSet) {
> return rowDataSet.output(this);
> }
>
> @SuppressWarnings("RegexpSingleline")
> @Override
> public void writeRecord(Row record) {
> if (limit <= 0) {
> return;
> }
> System.out.println(record.toString());
> limit--;
> }
> ```
This method will make record print on TaskManager, not client side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org