You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/08/16 03:37:39 UTC
[GitHub] [flink] HuangXingBo opened a new pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
HuangXingBo opened a new pull request #16833:
URL: https://github.com/apache/flink/pull/16833
## What is the purpose of the change
*This pull request will support to specify python client interpreter used to compile jobs*
## Brief change log
- *Support to specify python client interpreter used to compile jobs*
## Verifying this change
This change added tests and can be verified as follows:
- *UT in `PythonEnvUtilsTest.java`*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16833:
URL: https://github.com/apache/flink/pull/16833#issuecomment-899284371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16833:
URL: https://github.com/apache/flink/pull/16833#issuecomment-899284371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22250",
"triggerID" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22250)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16833:
URL: https://github.com/apache/flink/pull/16833#issuecomment-899284371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22250",
"triggerID" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"triggerType" : "PUSH"
}, {
"hash" : "c24f8855fbb6a0429b0af82bdddc0fc28db16492",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c24f8855fbb6a0429b0af82bdddc0fc28db16492",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22250)
* c24f8855fbb6a0429b0af82bdddc0fc28db16492 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16833:
URL: https://github.com/apache/flink/pull/16833#issuecomment-899284371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22250",
"triggerID" : "b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6",
"triggerType" : "PUSH"
}, {
"hash" : "c24f8855fbb6a0429b0af82bdddc0fc28db16492",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22258",
"triggerID" : "c24f8855fbb6a0429b0af82bdddc0fc28db16492",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b2b1a7d4158a69c9f92ac69799ce9296a8fc49e6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22250)
* c24f8855fbb6a0429b0af82bdddc0fc28db16492 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=22258)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16833:
URL: https://github.com/apache/flink/pull/16833#issuecomment-899190803
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 04eafab53e83893932c11a966ebb09505b1520b4 (Mon Aug 16 03:40:41 UTC 2021)
✅no warnings
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangXingBo closed pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
Posted by GitBox <gi...@apache.org>.
HuangXingBo closed pull request #16833:
URL: https://github.com/apache/flink/pull/16833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #16833: [FLINK-22871][python] Support to specify python client interpreter used to compile jobs
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #16833:
URL: https://github.com/apache/flink/pull/16833#discussion_r689222816
##########
File path: docs/content.zh/docs/deployment/cli.md
##########
@@ -479,6 +479,14 @@ related options. Here's an overview of all the Python related options for the ac
f = open('data/data.txt', 'r').
</td>
</tr>
+ <tr>
+ <td><code class="highlighter-rouge">-pyclientexec,--pyClientExecutable</code></td>
+ <td>
+ The path of the Python interpreter used to launch the Python process when submitting
+ the Python jobs via \"flink run\" or compiling the Java/Scala jobs containing
Review comment:
Give an example?
##########
File path: flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java
##########
@@ -137,6 +143,27 @@ static PythonEnvironment preparePythonEnvironment(
.collect(Collectors.toList());
addToPythonPath(env, pythonFiles);
}
+
+ // 5. set the archives directory as the working directory, then user could access the
+ // content of the archives via relative path
+ if (config.getOptional(PYTHON_ARCHIVES).isPresent()
+ && (config.getOptional(PYTHON_CLIENT_EXECUTABLE).isPresent()
+ || !System.getenv(PYFLINK_CLIENT_EXECUTABLE).isEmpty())) {
+ env.archivesDirectory = String.join(File.separator, tmpDir, PYTHON_ARCHIVES_DIR);
+ List<Path> pythonArchives =
+ Arrays.stream(config.get(PYTHON_ARCHIVES).split(FILE_DELIMITER))
+ .map(Path::new)
+ .collect(Collectors.toList());
+
+ // extract archives to archives directory
+ for (Path pythonArchive : pythonArchives) {
+ ZipUtils.extractZipFileWithPermissions(
+ pythonArchive.getPath(),
+ String.join(
+ File.separator, env.archivesDirectory, pythonArchive.getName()));
Review comment:
if python.archives is configured as venv.zip#myenv, the extracted directory name should be myenv
##########
File path: docs/content/docs/deployment/cli.md
##########
@@ -477,6 +477,14 @@ related options. Here's an overview of all the Python related options for the ac
f = open('data/data.txt', 'r').
</td>
</tr>
+ <tr>
+ <td><code class="highlighter-rouge">-pyclientexec,--pyClientExecutable</code></td>
Review comment:
The description in PythonOptions.PYTHON_CLIENT_EXECUTABLE should also be updated.
##########
File path: flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java
##########
@@ -137,6 +143,27 @@ static PythonEnvironment preparePythonEnvironment(
.collect(Collectors.toList());
addToPythonPath(env, pythonFiles);
}
+
+ // 5. set the archives directory as the working directory, then user could access the
+ // content of the archives via relative path
+ if (config.getOptional(PYTHON_ARCHIVES).isPresent()
+ && (config.getOptional(PYTHON_CLIENT_EXECUTABLE).isPresent()
+ || !System.getenv(PYFLINK_CLIENT_EXECUTABLE).isEmpty())) {
Review comment:
it may throw NPE as System.getenv(PYFLINK_CLIENT_EXECUTABLE) may return null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org