You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/03/10 12:47:12 UTC
[GitHub] [flink] zhoulii opened a new pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
zhoulii opened a new pull request #19040:
URL: https://github.com/apache/flink/pull/19040
## What is the purpose of the change
As described in [FLINK-26548](https://issues.apache.org/jira/browse/FLINK-26548), when using legacy file sources with AdaptiveBatcheScheduler, the parallelism of source reader is not set correctly, this pr aims to fix it.
## Verifying this change
- covered by existed unit tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (**no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**no**)
- The serializers: (**no**)
- The runtime per-record code paths (performance sensitive): (**no**)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**no**)
- The S3 file system connector: (**no**)
## Documentation
- Does this pull request introduce a new feature? (**no**)
- If yes, how is the feature documented? (**not applicable**)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r823767307
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1774,17 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ Configuration conf = (Configuration) getExecutionEnvironment().getConfiguration();
+ int readerParallelism = getParallelism();
+ if(readerParallelism == -1
+ && ClusterOptions.getSchedulerType(conf) == JobManagerOptions.SchedulerType.AdaptiveBatch) {
+ readerParallelism = conf.get(
Review comment:
Maybe:
`if (readerParallelism == ExecutionConfig.PARALLELISM_DEFAULT && getConfig().isDynamicGraph()) {
readerParallelism = getConfiguration().get(JobManagerOptions.ADAPTIVE_BATCH_SCHEDULER_DEFAULT_SOURCE_PARALLELISM);
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884)
* 555dc6ab31ce27539cae8499161bd87ecee0b353 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r824338994
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1774,17 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ Configuration conf = (Configuration) getExecutionEnvironment().getConfiguration();
+ int readerParallelism = getParallelism();
+ if(readerParallelism == -1
+ && ClusterOptions.getSchedulerType(conf) == JobManagerOptions.SchedulerType.AdaptiveBatch) {
+ readerParallelism = conf.get(
Review comment:
I noticed that your original code logic was `getExecutionEnvironment().getConfiguration()`, I think this will lead to an empty configuration because the [StreamExecutionEnvironment#getExecutionEnvironment()](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#:~:text=public%20static%20StreamExecutionEnvironment%20getExecutionEnvironment()%20%7B) will pass an empty configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk closed pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk closed pull request #19040:
URL: https://github.com/apache/flink/pull/19040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b493e3a66e19b87d0bfd35cfec5a9c901b4ac489 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840)
* c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b493e3a66e19b87d0bfd35cfec5a9c901b4ac489 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884)
* 555dc6ab31ce27539cae8499161bd87ecee0b353 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 555dc6ab31ce27539cae8499161bd87ecee0b353 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903)
* 4f2a2980bc6f97833d227a93a4dd8f4d20866a49 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 555dc6ab31ce27539cae8499161bd87ecee0b353 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r824319476
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1774,17 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ Configuration conf = (Configuration) getExecutionEnvironment().getConfiguration();
+ int readerParallelism = getParallelism();
+ if(readerParallelism == -1
+ && ClusterOptions.getSchedulerType(conf) == JobManagerOptions.SchedulerType.AdaptiveBatch) {
+ readerParallelism = conf.get(
Review comment:
-- if (readerParallelism == ExecutionConfig.PARALLELISM_DEFAULT && getConfig().isDynamicGraph())
yes, it's more concise.
--readerParallelism = getConfiguration().get(JobManagerOptions....);
actually, we cannot get config info via getConfiguration(), the result of getConfiguration() is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r824336058
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1774,17 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ Configuration conf = (Configuration) getExecutionEnvironment().getConfiguration();
+ int readerParallelism = getParallelism();
+ if(readerParallelism == -1
+ && ClusterOptions.getSchedulerType(conf) == JobManagerOptions.SchedulerType.AdaptiveBatch) {
+ readerParallelism = conf.get(
Review comment:
-> actually, we cannot get config info via getConfiguration(), the result of getConfiguration() is empty.
Why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064814833
> >
>
> Hi @wanglijie95 Thanks for reviewing, and the code formatting helps a lot. BTW, I have tested the code on a real cluster, but don't come up with a good idea to cover the change with ut or it, do you have any suggestions ?
I think we can check the number of splits each downstream reader received. Without this pr, only one downstream reader will receive one split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064720855
>
Hi @wanglijie95 Thanks for reviewing, and the code formatting helps a lot. BTW, I have tested the code on a real cluster, but don't come up with a good idea to cover the change with ut or it, do you have any suggestions ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r824320010
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1786,7 +1796,8 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
SingleOutputStreamOperator<OUT> source =
addSource(monitoringFunction, sourceName, null, boundedness)
- .transform("Split Reader: " + sourceName, typeInfo, factory);
+ .transform("Split Reader: " + sourceName, typeInfo, factory)
+ .setParallelism(readerParallelism);
Review comment:
agree, I will add a judgment here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b493e3a66e19b87d0bfd35cfec5a9c901b4ac489 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
}, {
"hash" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000",
"triggerID" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"triggerType" : "PUSH"
}, {
"hash" : "5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33060",
"triggerID" : "5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000)
* 5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33060)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 555dc6ab31ce27539cae8499161bd87ecee0b353 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903)
* 4f2a2980bc6f97833d227a93a4dd8f4d20866a49 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r823775267
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1786,7 +1796,8 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
SingleOutputStreamOperator<OUT> source =
addSource(monitoringFunction, sourceName, null, boundedness)
- .transform("Split Reader: " + sourceName, typeInfo, factory);
+ .transform("Split Reader: " + sourceName, typeInfo, factory)
+ .setParallelism(readerParallelism);
Review comment:
I think we'd better not modify the logic when we don't use the adaptive batch scheduler, so I think we also need to add a `readerParallelism == ExecutionConfig.PARALLELISM_DEFAULT && getConfig().isDynamicGraph()` judgment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064720855
>
Thanks for reviewing, and the code formatting helps a lot. BTW, I have tested the code on a real cluster, but don't come up with a good idea to cover the change with ut or it, do you have any suggestions ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b493e3a66e19b87d0bfd35cfec5a9c901b4ac489 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840)
* c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1067723796
Thanks for all the efforts on this PR @zhoulii, but unfortunately this PR doesn't fully solve the problem and introduces some new problems.
Currently, the solution discussed in FLINK-26548 encounters the following problem: Setting the parallelism of the actual source directly will cause the parallelism of the multiInput to be configured, which will break the forward property and cause the following error:
`org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Forward partitioning does not allow change of parallelism. Upstream operation: Calc[1124]-1669 parallelism: -1, downstream operation: MultipleInput[1149]-1721 parallelism: 1 You must use another partitioning strategy, such as broadcast, rebalance, shuffle or global.`
Considering that the legacy file sources will be replaced by new source(FLIP-27), and there are some known issues: FLINK-26576, I think we should choose to support new sources only, and add this to limitations of adaptive batch scheduler. BTW, I think we should migrate the `StreamExecutionEnvironment#readFile` `StreamExecutionEnvironment#createInput` to new source as soon as possible.
WDYT @zhoulii @zhuzhurk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r825546683
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1773,23 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ int readerParallelism = getParallelism();
+
+ boolean isAdaptiveBatchSchedulerEnabled =
+ readerParallelism == ExecutionConfig.PARALLELISM_DEFAULT
+ && getConfig().isDynamicGraph();
+
+ if (isAdaptiveBatchSchedulerEnabled) {
+ readerParallelism =
+ getConfiguration()
Review comment:
yes, I found this problem too. I agree to keep this PR open before the problem is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
}, {
"hash" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2a2980bc6f97833d227a93a4dd8f4d20866a49 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982)
* ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
}, {
"hash" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000",
"triggerID" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"triggerType" : "PUSH"
}, {
"hash" : "5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000)
* 5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1067723796
Thanks for all the efforts on this PR @zhoulii, but unfortunately this PR doesn't fully solve the problem and introduces new problems.
Currently, the solution discussed in FLINK-26548 encounters the following problem: Setting the parallelism of the actual source directly will cause the parallelism of the multiInput to be configured, which will break the forward property and cause the following error:
`org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Forward partitioning does not allow change of parallelism. Upstream operation: Calc[1124]-1669 parallelism: -1, downstream operation: MultipleInput[1149]-1721 parallelism: 1 You must use another partitioning strategy, such as broadcast, rebalance, shuffle or global.`
Considering that the legacy file sources will be replaced by new source(FLIP-27), and there are some known issues: FLINK-26576, I think we should choose to support new sources only, and add this to limitations of adaptive batch scheduler. BTW, I think we should migrate the `StreamExecutionEnvironment#readFile` `StreamExecutionEnvironment#createInput` to new source as soon as possible.
WDYT @zhoulii @zhuzhurk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r824435119
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1788,6 +1800,11 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
addSource(monitoringFunction, sourceName, null, boundedness)
.transform("Split Reader: " + sourceName, typeInfo, factory);
+ if (getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT
+ && getConfig().isDynamicGraph()) {
Review comment:
I think we can introduce a boolean variable to reuse the `getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT && getConfig().isDynamicGraph()` logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b493e3a66e19b87d0bfd35cfec5a9c901b4ac489 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1068240218
+1 to deprecate this PR. Given that we me multiple problems trying to fix this problem and the temporary fix would become complex and add more risk, I think reworking the methods use new source interface later would be a better solution.
Thanks @zhoulii for the nice work on this PR and looking into the problems! And thanks @wanglijie95 for the reviewing and investigation!
I will close this PR and change the target fix of FLINK-26548 to 1.16.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
}, {
"hash" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000",
"triggerID" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2a2980bc6f97833d227a93a4dd8f4d20866a49 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982)
* ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1067738895
> Thanks for all the efforts on this PR @zhoulii, but unfortunately this PR doesn't fully solve the problem and introduces new problems.
>
> Currently, the solution discussed in [FLINK-26548](https://issues.apache.org/jira/browse/FLINK-26548) encounters the following problem: Setting the parallelism of the actual source directly will cause the parallelism of the multiInput to be configured, which will break the forward property and cause the following error: `org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Forward partitioning does not allow change of parallelism. Upstream operation: Calc[1124]-1669 parallelism: -1, downstream operation: MultipleInput[1149]-1721 parallelism: 1 You must use another partitioning strategy, such as broadcast, rebalance, shuffle or global.`
>
> Considering that the legacy file sources will be replaced by new source(FLIP-27), and there are some known issues: [FLINK-26576](https://issues.apache.org/jira/browse/FLINK-26576), I think we should choose to support new sources only, and add this to limitations of adaptive batch scheduler. BTW, I think we should migrate the `StreamExecutionEnvironment#readFile` `StreamExecutionEnvironment#createInput` to new source as soon as possible.
>
> WDYT @zhoulii @zhuzhurk ?
Thanks for replying, I found this problem too. Considering that the legacy file sources will be replaced by new source and this pr would introduce some new problems, supporting new sources only is a proper way, so I agree with your proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r825474342
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1773,23 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ int readerParallelism = getParallelism();
+
+ boolean isAdaptiveBatchSchedulerEnabled =
+ readerParallelism == ExecutionConfig.PARALLELISM_DEFAULT
+ && getConfig().isDynamicGraph();
+
+ if (isAdaptiveBatchSchedulerEnabled) {
+ readerParallelism =
+ getConfiguration()
Review comment:
`getConfiguration()` may be empty when executing sql jobs and will make this fix ineffective in that case. This problem seems to be caused by `DummyStreamExecutionEnvironment` which does not properly override `getConfiguration()`. But we are still looking into some details to figure out whether it is safe to fix it by simply do the overriding.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1773,23 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ int readerParallelism = getParallelism();
+
+ boolean isAdaptiveBatchSchedulerEnabled =
Review comment:
Maybe `useDynamicGraphSourceParallelism` is more accurate because conceptually `isAdaptiveBatchSchedulerEnabled == getConfig().isDynamicGraph()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r825545501
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1773,23 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ int readerParallelism = getParallelism();
+
+ boolean isAdaptiveBatchSchedulerEnabled =
Review comment:
yes, `useDynamicGraphSourceParallelism` is more appropriate here, I will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1068240218
+1 to deprecate this PR. Given that we me multiple problems trying to fix this problem and the temporary fix would become complex and add more risk, I think reworking the methods use new source interface later would be a better solution.
Thanks @zhoulii for the work on this PR and looking into the problems! And thanks @wanglijie95 for the reviewing and investigation!
I will close this PR and change the target fix of FLINK-26548 to 1.16.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r823767307
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1774,17 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ Configuration conf = (Configuration) getExecutionEnvironment().getConfiguration();
+ int readerParallelism = getParallelism();
+ if(readerParallelism == -1
+ && ClusterOptions.getSchedulerType(conf) == JobManagerOptions.SchedulerType.AdaptiveBatch) {
+ readerParallelism = conf.get(
Review comment:
Maybe:
`
if (readerParallelism == ExecutionConfig.PARALLELISM_DEFAULT && getConfig().isDynamicGraph()) {
readerParallelism = getConfiguration().get(JobManagerOptions....);
}
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064128995
Thanks for creating this PR @zhoulii. I have left two comments. In addition, I think at least one test is needed to cover the changes, and you also need to pay attention to the code formatting(You can see https://nightlies.apache.org/flink/flink-docs-master/docs/flinkdev/ide_setup/#code-formatting for details)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r824343392
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1772,9 +1774,17 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
+ ContinuousFileMonitoringFunction.MIN_MONITORING_INTERVAL
+ " ms.");
+ Configuration conf = (Configuration) getExecutionEnvironment().getConfiguration();
+ int readerParallelism = getParallelism();
+ if(readerParallelism == -1
+ && ClusterOptions.getSchedulerType(conf) == JobManagerOptions.SchedulerType.AdaptiveBatch) {
+ readerParallelism = conf.get(
Review comment:
[StreamExecutionEnvironment#getExecutionEnvironment()](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#:~:text=public%20static%20StreamExecutionEnvironment%20getExecutionEnvironment()%20%7B) will load conf to initialize configuration, I have debugged the code, it will not lead to an empty configuration. On the contrary, getConfiguration() get the value of [configuration](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L208), it is an empty configuration created in [StreamExecutionEnvironment()](https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L221)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19040:
URL: https://github.com/apache/flink/pull/19040#discussion_r824451310
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1788,6 +1800,11 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
addSource(monitoringFunction, sourceName, null, boundedness)
.transform("Split Reader: " + sourceName, typeInfo, factory);
+ if (getParallelism() == ExecutionConfig.PARALLELISM_DEFAULT
+ && getConfig().isDynamicGraph()) {
Review comment:
agree, it's more concise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
}, {
"hash" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000",
"triggerID" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f2a2980bc6f97833d227a93a4dd8f4d20866a49 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19040: [FLINK-26548][runtime] set the source parallelism correctly when using legacy file sources with AdaptiveBatcheScheduler
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19040:
URL: https://github.com/apache/flink/pull/19040#issuecomment-1064026493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32840",
"triggerID" : "b493e3a66e19b87d0bfd35cfec5a9c901b4ac489",
"triggerType" : "PUSH"
}, {
"hash" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32884",
"triggerID" : "c0c7dcadc7b7a8a8999b3689fb0ccae191e2d321",
"triggerType" : "PUSH"
}, {
"hash" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32903",
"triggerID" : "555dc6ab31ce27539cae8499161bd87ecee0b353",
"triggerType" : "PUSH"
}, {
"hash" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32982",
"triggerID" : "4f2a2980bc6f97833d227a93a4dd8f4d20866a49",
"triggerType" : "PUSH"
}, {
"hash" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33000",
"triggerID" : "ab5b2d1c9a5792cd1018668e9ab225d91eac8f0d",
"triggerType" : "PUSH"
}, {
"hash" : "5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33060",
"triggerID" : "5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d2fa164cd0bc5b51729c96f47e5a2bc1dbdd703 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33060)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org