You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/11/17 15:40:47 UTC
[GitHub] [flink] godfreyhe opened a new pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
godfreyhe opened a new pull request #14103:
URL: https://github.com/apache/flink/pull/14103
## What is the purpose of the change
*Users can't specify job name for sql job when using new apis introduced in FLIP-84. As discussion in [FLINK-18545](https://issues.apache.org/jira/browse/FLINK-18545), we should introduce `pipeline.name` in `PipelineOptions` which could work for both DataStream jobs and SQL jobs.*
## Brief change log
- *Introduce `pipeline.name` to allow users to specify job name by configuration*
- *Specify job name by `pipeline.name` for flink-python*
- *Specify job name by `pipeline.name` for sql job*
## Verifying this change
This change added tests and can be verified as follows:
- *Extended StreamExecutionEnvironmentTest & PythonConfigUtilTest to verify the job name specified by pipeline.name*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (**yes** / no)
- If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #14103:
URL: https://github.com/apache/flink/pull/14103#discussion_r530846446
##########
File path: flink-java/src/main/java/org/apache/flink/api/java/ExecutionEnvironment.java
##########
@@ -1122,12 +1125,14 @@ void registerDataSink(DataSink<?> sink) {
}
/**
- * Gets a default job name, based on the timestamp when this method is invoked.
+ * Gets the job name. If user defined job name is not found in the configuration,
+ * the default name based on the timestamp when this method is invoked will return.
*
- * @return A default job name.
+ * @return A job name.
*/
- private static String getDefaultName() {
- return "Flink Java Job at " + Calendar.getInstance().getTime();
+ @VisibleForTesting
Review comment:
This method must not be public and thus part of the API. You can call `createProgramPlan().getJobName` for testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
kl0u commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-731017286
I see. If everyone agrees then feel free to go ahead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-731919518
@twalthr @dawidwys could you take a look at this pr, thanks so much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-730956823
@kl0u, different from job id, job name is not the key info of a StreamGraph, and `pipeline.name` is just used for overwriting the default job name. Just as @wuchong mentioned, we can also change the value of `pipeline.name` before submitting a job. And it's a very useful feature which could let sql client specify job name. cc @dawidwys @twalthr
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
}, {
"hash" : "96911576999869f37acb8133be917c91745e90a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96911576999869f37acb8133be917c91745e90a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cda47b16b8fa216fefb5399249026c2a54e907ed Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146)
* 96911576999869f37acb8133be917c91745e90a4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729015076
cc @twalthr @dawidwys
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
}, {
"hash" : "96911576999869f37acb8133be917c91745e90a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96911576999869f37acb8133be917c91745e90a4",
"triggerType" : "PUSH"
}, {
"hash" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10160",
"triggerID" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"triggerType" : "PUSH"
}, {
"hash" : "1858bb1abcedf787cb83f77443298baf3afe63e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10172",
"triggerID" : "1858bb1abcedf787cb83f77443298baf3afe63e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96911576999869f37acb8133be917c91745e90a4 UNKNOWN
* 84daf24842f84ae2b07fa9fccef5a328bffc6029 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10160)
* 1858bb1abcedf787cb83f77443298baf3afe63e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10172)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729014282
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 6cdf328a97a33c80d0b2275468b907e619fa63b8 (Tue Nov 17 15:42:35 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14103:
URL: https://github.com/apache/flink/pull/14103#discussion_r530346136
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -2330,4 +2330,8 @@ public void registerCachedFile(String filePath, String name, boolean executable)
}
return (T) resolvedTypeInfo;
}
+
+ private String getJobName() {
+ return configuration.getString(PipelineOptions.NAME, DEFAULT_JOB_NAME);
Review comment:
Can't we just put the default name in the pipeline option? That way, it will also be rendered in the documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 938cbb403a01e6d31cbc605a49f9baf7b3b9bf31 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cda47b16b8fa216fefb5399249026c2a54e907ed Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #14103:
URL: https://github.com/apache/flink/pull/14103#discussion_r530745246
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -2330,4 +2330,8 @@ public void registerCachedFile(String filePath, String name, boolean executable)
}
return (T) resolvedTypeInfo;
}
+
+ private String getJobName() {
+ return configuration.getString(PipelineOptions.NAME, DEFAULT_JOB_NAME);
Review comment:
`TableEnvironment` also uses a different default name for the SQL query, e.g. "insert-into_<sink-names>". So I think no default value makes more sense, that the default value is defined by different environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
twalthr commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-730959217
I agree with @jark and @godfreyhe. The name is just used for logging and easier identification in UIs. The user can always change it before the next execution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
kl0u edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-731017286
I see. If everyone agrees then feel free to go ahead, as I do not have a better solution for the moment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
kl0u commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-730338073
Hi all, I am not sure if allowing the users to specify a global name for the whole application is the application's config file would work for multi-execute applications. This is the same argument as in a related discussion here https://issues.apache.org/jira/browse/FLINK-19358?focusedCommentId=17204556&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17204556.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 938cbb403a01e6d31cbc605a49f9baf7b3b9bf31 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714)
* cda47b16b8fa216fefb5399249026c2a54e907ed UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-734531462
I will merge this pr
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
}, {
"hash" : "96911576999869f37acb8133be917c91745e90a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96911576999869f37acb8133be917c91745e90a4",
"triggerType" : "PUSH"
}, {
"hash" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10160",
"triggerID" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"triggerType" : "PUSH"
}, {
"hash" : "1858bb1abcedf787cb83f77443298baf3afe63e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1858bb1abcedf787cb83f77443298baf3afe63e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96911576999869f37acb8133be917c91745e90a4 UNKNOWN
* 84daf24842f84ae2b07fa9fccef5a328bffc6029 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10160)
* 1858bb1abcedf787cb83f77443298baf3afe63e0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
wuchong commented on a change in pull request #14103:
URL: https://github.com/apache/flink/pull/14103#discussion_r530745246
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -2330,4 +2330,8 @@ public void registerCachedFile(String filePath, String name, boolean executable)
}
return (T) resolvedTypeInfo;
}
+
+ private String getJobName() {
+ return configuration.getString(PipelineOptions.NAME, DEFAULT_JOB_NAME);
Review comment:
`TableEnvironment` also uses a different default name for the SQL query, e.g. `insert-into_<sink-names>`. So I think no default value makes more sense, that the default value is defined by different environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 938cbb403a01e6d31cbc605a49f9baf7b3b9bf31 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u removed a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
kl0u removed a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-730338073
Hi all, I am not sure if allowing the users to specify a global name for the whole application is the application's config file would work for multi-execute applications. This is the same argument as in a related discussion here https://issues.apache.org/jira/browse/FLINK-19358?focusedCommentId=17204556&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17204556.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 938cbb403a01e6d31cbc605a49f9baf7b3b9bf31 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714)
* cda47b16b8fa216fefb5399249026c2a54e907ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
}, {
"hash" : "96911576999869f37acb8133be917c91745e90a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96911576999869f37acb8133be917c91745e90a4",
"triggerType" : "PUSH"
}, {
"hash" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10160",
"triggerID" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96911576999869f37acb8133be917c91745e90a4 UNKNOWN
* 84daf24842f84ae2b07fa9fccef5a328bffc6029 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10160)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
}, {
"hash" : "96911576999869f37acb8133be917c91745e90a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96911576999869f37acb8133be917c91745e90a4",
"triggerType" : "PUSH"
}, {
"hash" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10160",
"triggerID" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"triggerType" : "PUSH"
}, {
"hash" : "1858bb1abcedf787cb83f77443298baf3afe63e0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10172",
"triggerID" : "1858bb1abcedf787cb83f77443298baf3afe63e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96911576999869f37acb8133be917c91745e90a4 UNKNOWN
* 1858bb1abcedf787cb83f77443298baf3afe63e0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10172)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #14103:
URL: https://github.com/apache/flink/pull/14103#discussion_r530365066
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -2330,4 +2330,8 @@ public void registerCachedFile(String filePath, String name, boolean executable)
}
return (T) resolvedTypeInfo;
}
+
+ private String getJobName() {
+ return configuration.getString(PipelineOptions.NAME, DEFAULT_JOB_NAME);
Review comment:
Are pipeline options also used in `ExecutionEnvironment`? The PR doesn't update this class but maybe it should? In that case `Flink Streaming Job` would be wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe commented on a change in pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
godfreyhe commented on a change in pull request #14103:
URL: https://github.com/apache/flink/pull/14103#discussion_r530734736
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -2330,4 +2330,8 @@ public void registerCachedFile(String filePath, String name, boolean executable)
}
return (T) resolvedTypeInfo;
}
+
+ private String getJobName() {
+ return configuration.getString(PipelineOptions.NAME, DEFAULT_JOB_NAME);
Review comment:
Thanks all for the suggestions. @twalthr, I have checked that the pipeline options are also used in `ExecutionEnvironment`. I also don't notice that before, I will update it. The default name of `ExecutionEnvironment` is `"Flink Java Job at " + Calendar.getInstance().getTime()`, so the `NAME` option has no default value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
wuchong commented on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-730823946
Hi @kl0u , I don't think the job name configuration conflicts with application mode. I think multi-execute applications may al so have same job name, e.g. use the default job name when submitting using `env.execute()`.
With the job name configuration, users can still change the job name before submitting, for example:
```java
Configuration conf = new Configuration();
conf.setString("pipeline.name", "job1");
env.configure(conf, Thread.currentThread().getContextClassLoader);
env.execute(); // submit job1
conf.setString("pipeline.name", "job2");
env.configure(conf, Thread.currentThread().getContextClassLoader);
env.execute(); // submit job2
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 938cbb403a01e6d31cbc605a49f9baf7b3b9bf31 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] godfreyhe merged pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
godfreyhe merged pull request #14103:
URL: https://github.com/apache/flink/pull/14103
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14103:
URL: https://github.com/apache/flink/pull/14103#issuecomment-729030820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9714",
"triggerID" : "938cbb403a01e6d31cbc605a49f9baf7b3b9bf31",
"triggerType" : "PUSH"
}, {
"hash" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146",
"triggerID" : "cda47b16b8fa216fefb5399249026c2a54e907ed",
"triggerType" : "PUSH"
}, {
"hash" : "96911576999869f37acb8133be917c91745e90a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96911576999869f37acb8133be917c91745e90a4",
"triggerType" : "PUSH"
}, {
"hash" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84daf24842f84ae2b07fa9fccef5a328bffc6029",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cda47b16b8fa216fefb5399249026c2a54e907ed Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10146)
* 96911576999869f37acb8133be917c91745e90a4 UNKNOWN
* 84daf24842f84ae2b07fa9fccef5a328bffc6029 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #14103: [FLINK-18545] Introduce `pipeline.name` to allow users to specify job name by configuration
Posted by GitBox <gi...@apache.org>.
twalthr commented on a change in pull request #14103:
URL: https://github.com/apache/flink/pull/14103#discussion_r530250712
##########
File path: docs/_includes/generated/pipeline_configuration.html
##########
@@ -86,6 +86,12 @@
<td>Integer</td>
<td>The program-wide maximum parallelism used for operators which haven't specified a maximum parallelism. The maximum parallelism specifies the upper limit for dynamic scaling and the number of key groups used for partitioned state.</td>
</tr>
+ <tr>
+ <td><h5>pipeline.name</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>The name to display for the job.</td>
Review comment:
nit: change this to `The job name used for printing and logging.`
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/StreamExecutionEnvironmentTest.java
##########
@@ -269,6 +271,27 @@ public void testGetStreamGraph() {
}
}
+ @Test
+ public void testDefaultJobName() {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+ env.fromCollection(new DummySplittableIterator<>(), BasicTypeInfo.INT_TYPE_INFO).addSink(new DiscardingSink<>());
Review comment:
nit: simplify the test here and below to just use `env.fromElements(1, 2, 3).print()`
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java
##########
@@ -1778,7 +1778,7 @@ private StateBackend loadStateBackend(ReadableConfig configuration, ClassLoader
* @throws Exception which occurs during job execution.
*/
public JobExecutionResult execute() throws Exception {
Review comment:
Update `StreamExecutionEnvironment#configure` as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org