You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "reswqa (via GitHub)" <gi...@apache.org> on 2023/07/03 11:40:10 UTC
[GitHub] [flink] reswqa opened a new pull request, #22933: [FLINK-32459][connector] Force set the parallelism of SocketTableSource to 1
reswqa opened a new pull request, #22933:
URL: https://github.com/apache/flink/pull/22933
## What is the purpose of the change
*`SocketSource` can only work with parallelism of 1, It is best to force set it when load it in `DynamicTableSource`.*
## Brief change log
- *Force set the parallelism of `SocketTableSource` to `1`*
- *Check parallelism of `SocketSource` when createReader.*
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] reswqa commented on a diff in pull request #22933: [FLINK-32459][connector] Force set the parallelism of SocketTableSource to 1
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on code in PR #22933:
URL: https://github.com/apache/flink/pull/22933#discussion_r1251913361
##########
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/connectors/SocketDynamicTableSource.java:
##########
@@ -69,16 +73,28 @@ public ChangelogMode getChangelogMode() {
@Override
public ScanRuntimeProvider getScanRuntimeProvider(ScanContext runtimeProviderContext) {
+ return new DataStreamScanProvider() {
+ @Override
+ public DataStream<RowData> produceDataStream(
+ ProviderContext providerContext, StreamExecutionEnvironment execEnv) {
+ final DeserializationSchema<RowData> deserializer =
+ decodingFormat.createRuntimeDecoder(
+ runtimeProviderContext, producedDataType);
- // create runtime classes that are shipped to the cluster
+ final SocketSource socketSource =
+ new SocketSource(hostname, port, byteDelimiter, deserializer);
- final DeserializationSchema<RowData> deserializer =
- decodingFormat.createRuntimeDecoder(runtimeProviderContext, producedDataType);
+ return execEnv.fromSource(
+ socketSource, WatermarkStrategy.noWatermarks(), "SocketSource")
+ // SocketSource can only work with a parallelism of 1.
Review Comment:
Good sugesstion, added it.
##########
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/connectors/SocketDynamicTableSource.java:
##########
@@ -69,16 +73,28 @@ public ChangelogMode getChangelogMode() {
@Override
public ScanRuntimeProvider getScanRuntimeProvider(ScanContext runtimeProviderContext) {
+ return new DataStreamScanProvider() {
+ @Override
+ public DataStream<RowData> produceDataStream(
+ ProviderContext providerContext, StreamExecutionEnvironment execEnv) {
+ final DeserializationSchema<RowData> deserializer =
+ decodingFormat.createRuntimeDecoder(
+ runtimeProviderContext, producedDataType);
- // create runtime classes that are shipped to the cluster
+ final SocketSource socketSource =
+ new SocketSource(hostname, port, byteDelimiter, deserializer);
- final DeserializationSchema<RowData> deserializer =
- decodingFormat.createRuntimeDecoder(runtimeProviderContext, producedDataType);
+ return execEnv.fromSource(
+ socketSource, WatermarkStrategy.noWatermarks(), "SocketSource")
+ // SocketSource can only work with a parallelism of 1.
Review Comment:
Good suggestion, added it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a diff in pull request #22933: [FLINK-32459][connector] Force set the parallelism of SocketTableSource to 1
Posted by "TanYuxin-tyx (via GitHub)" <gi...@apache.org>.
TanYuxin-tyx commented on code in PR #22933:
URL: https://github.com/apache/flink/pull/22933#discussion_r1251627031
##########
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/connectors/SocketDynamicTableSource.java:
##########
@@ -69,16 +73,28 @@ public ChangelogMode getChangelogMode() {
@Override
public ScanRuntimeProvider getScanRuntimeProvider(ScanContext runtimeProviderContext) {
+ return new DataStreamScanProvider() {
+ @Override
+ public DataStream<RowData> produceDataStream(
+ ProviderContext providerContext, StreamExecutionEnvironment execEnv) {
+ final DeserializationSchema<RowData> deserializer =
+ decodingFormat.createRuntimeDecoder(
+ runtimeProviderContext, producedDataType);
- // create runtime classes that are shipped to the cluster
+ final SocketSource socketSource =
+ new SocketSource(hostname, port, byteDelimiter, deserializer);
- final DeserializationSchema<RowData> deserializer =
- decodingFormat.createRuntimeDecoder(runtimeProviderContext, producedDataType);
+ return execEnv.fromSource(
+ socketSource, WatermarkStrategy.noWatermarks(), "SocketSource")
+ // SocketSource can only work with a parallelism of 1.
Review Comment:
we'd better add the description of only supporting parallelism 1 to the Class Java doc explicitly. For example, Add a "Note that ..." to the class java doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] reswqa merged pull request #22933: [FLINK-32459][connector] Force set the parallelism of SocketTableSource to 1
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa merged PR #22933:
URL: https://github.com/apache/flink/pull/22933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] reswqa commented on pull request #22933: [FLINK-32459][connector] Force set the parallelism of SocketTableSource to 1
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on PR #22933:
URL: https://github.com/apache/flink/pull/22933#issuecomment-1620091128
Thanks @TanYuxin-tyx for the review, I have updated this, PTAL again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22933: [FLINK-32459][connector] Force set the parallelism of SocketTableSource to 1
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22933:
URL: https://github.com/apache/flink/pull/22933#issuecomment-1618060154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "950c193e3f244436a3646ceeffcc31cf1550f8b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "950c193e3f244436a3646ceeffcc31cf1550f8b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 950c193e3f244436a3646ceeffcc31cf1550f8b7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a diff in pull request #22933: [FLINK-32459][connector] Force set the parallelism of SocketTableSource to 1
Posted by "TanYuxin-tyx (via GitHub)" <gi...@apache.org>.
TanYuxin-tyx commented on code in PR #22933:
URL: https://github.com/apache/flink/pull/22933#discussion_r1251627031
##########
flink-examples/flink-examples-table/src/main/java/org/apache/flink/table/examples/java/connectors/SocketDynamicTableSource.java:
##########
@@ -69,16 +73,28 @@ public ChangelogMode getChangelogMode() {
@Override
public ScanRuntimeProvider getScanRuntimeProvider(ScanContext runtimeProviderContext) {
+ return new DataStreamScanProvider() {
+ @Override
+ public DataStream<RowData> produceDataStream(
+ ProviderContext providerContext, StreamExecutionEnvironment execEnv) {
+ final DeserializationSchema<RowData> deserializer =
+ decodingFormat.createRuntimeDecoder(
+ runtimeProviderContext, producedDataType);
- // create runtime classes that are shipped to the cluster
+ final SocketSource socketSource =
+ new SocketSource(hostname, port, byteDelimiter, deserializer);
- final DeserializationSchema<RowData> deserializer =
- decodingFormat.createRuntimeDecoder(runtimeProviderContext, producedDataType);
+ return execEnv.fromSource(
+ socketSource, WatermarkStrategy.noWatermarks(), "SocketSource")
+ // SocketSource can only work with a parallelism of 1.
Review Comment:
we'd better add the description of only supporting parallelism 1 to the Class Java doc explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org