You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ifndef-SleePy (via GitHub)" <gi...@apache.org> on 2023/01/30 09:26:10 UTC
[GitHub] [flink] ifndef-SleePy opened a new pull request, #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
ifndef-SleePy opened a new pull request, #21785:
URL: https://github.com/apache/flink/pull/21785
## What is the purpose of the change
*This pull request introduces a way to enable speculative execution for `OutputFormat`*
## Brief change log
- *Introduce context in `OutputFormat` and `FinalizeOnMaster` to expose necessary infos for speculative execution*
- *Speculative execution would be enabled if a `OutputFormat` inherits SupportsConcurrentExecutionAttempts*
## Verifying this change
This change added tests:
- *Add new test cases in `StreamingJobGraphGeneratorTest`*
- *Add new test cases in `SpeculativeSchedulerITCase`*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (JavaDocs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a diff in pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk commented on code in PR #21785:
URL: https://github.com/apache/flink/pull/21785#discussion_r1090524599
##########
flink-core/src/main/java/org/apache/flink/api/common/io/FinalizeOnMaster.java:
##########
@@ -66,6 +66,7 @@ interface FinalizationContext {
*
* @param subtaskIndex the subtask index.
* @return the finished attempts.
+ * @throws IllegalArgumentException Thrown, if subtaskIndex is invalid.
Review Comment:
This change should be part of a previous commit. This also applies some other change in this commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] ifndef-SleePy commented on pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "ifndef-SleePy (via GitHub)" <gi...@apache.org>.
ifndef-SleePy commented on PR #21785:
URL: https://github.com/apache/flink/pull/21785#issuecomment-1408261870
@zhuzhurk Please take a look when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] ifndef-SleePy commented on a diff in pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "ifndef-SleePy (via GitHub)" <gi...@apache.org>.
ifndef-SleePy commented on code in PR #21785:
URL: https://github.com/apache/flink/pull/21785#discussion_r1090532614
##########
flink-core/src/main/java/org/apache/flink/api/common/io/FinalizeOnMaster.java:
##########
@@ -66,6 +66,7 @@ interface FinalizationContext {
*
* @param subtaskIndex the subtask index.
* @return the finished attempts.
+ * @throws IllegalArgumentException Thrown, if subtaskIndex is invalid.
Review Comment:
Oops, you are right. I squash the wrong commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk closed pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk closed pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
URL: https://github.com/apache/flink/pull/21785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk commented on PR #21785:
URL: https://github.com/apache/flink/pull/21785#issuecomment-1409593507
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a diff in pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk commented on code in PR #21785:
URL: https://github.com/apache/flink/pull/21785#discussion_r1090489821
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobVertex.java:
##########
@@ -579,6 +579,28 @@ public interface InitializeOnMasterContext {
int getExecutionParallelism();
}
+ /** The context exposes some runtime infos for finalization. */
+ public interface FinalizationOnMasterContext {
Review Comment:
FinalizationOnMasterContext -> FinalizeOnMasterContext
To be aligned with `InitializeOnMasterContext`.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobVertex.java:
##########
@@ -579,6 +579,28 @@ public interface InitializeOnMasterContext {
int getExecutionParallelism();
}
+ /** The context exposes some runtime infos for finalization. */
+ public interface FinalizationOnMasterContext {
+ /** The class loader for user defined code. */
+ ClassLoader getClassLoader();
+
+ /**
+ * The actual parallelism this vertex will be run with. In contrast, the {@link
+ * #getParallelism()} is the original parallelism set when creating the {@link JobGraph} and
+ * might be updated e.g. by the {@link
+ * org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler}.
+ */
+ int getExecutionParallelism();
+
+ /**
+ * Get the finished attempt number of subtask.
+ *
+ * @param subtaskIndex the subtask index.
+ * @return the finished attempts.
Review Comment:
attempts -> attempt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #21785:
URL: https://github.com/apache/flink/pull/21785#issuecomment-1408275959
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9c9c6f95c524c9863dfdcfe489b9095dd4161c97",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c9c6f95c524c9863dfdcfe489b9095dd4161c97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c9c6f95c524c9863dfdcfe489b9095dd4161c97 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] ifndef-SleePy commented on a diff in pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "ifndef-SleePy (via GitHub)" <gi...@apache.org>.
ifndef-SleePy commented on code in PR #21785:
URL: https://github.com/apache/flink/pull/21785#discussion_r1090532614
##########
flink-core/src/main/java/org/apache/flink/api/common/io/FinalizeOnMaster.java:
##########
@@ -66,6 +66,7 @@ interface FinalizationContext {
*
* @param subtaskIndex the subtask index.
* @return the finished attempts.
+ * @throws IllegalArgumentException Thrown, if subtaskIndex is invalid.
Review Comment:
Oops, you are right. I squashed the wrong commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] ifndef-SleePy commented on pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "ifndef-SleePy (via GitHub)" <gi...@apache.org>.
ifndef-SleePy commented on PR #21785:
URL: https://github.com/apache/flink/pull/21785#issuecomment-1408511394
@zhuzhurk Thanks for the detailed review! I've addressed the comments. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #21785: [FLINK-30798][runtime] Make OutputFormat support speculative execution for batch jobs
Posted by "zhuzhurk (via GitHub)" <gi...@apache.org>.
zhuzhurk commented on PR #21785:
URL: https://github.com/apache/flink/pull/21785#issuecomment-1409664873
Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org