You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/04/13 08:26:54 UTC
[GitHub] [flink] WeiZhong94 opened a new pull request #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
WeiZhong94 opened a new pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714
## What is the purpose of the change
*This pull request fixes the bug that the startup of the Python UDF worker will fail when the pyflink job runs in local mode and the command "python" points to Python 2.7.*
## Brief change log
- *Add logic to the constructor of `TableEnvironment` to set the default value of 'python.executable' to the path of the python interpreter which launches the job if the job is executed by `LocalExecutor`.*
## Verifying this change
This change is already covered by existing tests, such as *test_table_environment_api.py*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): ( no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160050547",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdff6e294f476e886a036e7cd462c0e0f79f4c58 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160050547) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e8789334295f653c075177bcec9ebb5cd1facc8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160025227) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11714: [FLINK-17114][python]
Set the default value of 'python.executable' to the path of the python
interpreter which launches the job when the job is executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e8789334295f653c075177bcec9ebb5cd1facc8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu closed pull request #11714: [FLINK-17114][python]
Set the default value of 'python.executable' to the path of the python
interpreter which launches the job when the job is executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
dianfu closed pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160050547",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e8789334295f653c075177bcec9ebb5cd1facc8 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160025227) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389)
* fdff6e294f476e886a036e7cd462c0e0f79f4c58 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160050547) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e8789334295f653c075177bcec9ebb5cd1facc8 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160025227) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on a change in pull request #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on a change in pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#discussion_r407464588
##########
File path: flink-python/pyflink/table/table_environment.py
##########
@@ -1082,6 +1086,13 @@ def _from_elements(self, elements, schema):
finally:
os.unlink(temp_file.name)
+ def _set_python_executable_for_local_executor(self):
+ jvm = get_gateway().jvm
+ j_config = get_j_env_configuration(self)
+ if not j_config.containsKey(jvm.PythonOptions.PYTHON_EXECUTABLE.key()) \
Review comment:
Yes, the configuration in `TableConfig` has priority over the configuration in `ExecutionEnvironment`. So even if we set the `python.executable` in `ExecutionEnvironment`, it will not override the user configuration in `TableConfig`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160050547",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
}, {
"hash" : "213ac14377e762efdfc3b57f100935e3a21c6690",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "213ac14377e762efdfc3b57f100935e3a21c6690",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdff6e294f476e886a036e7cd462c0e0f79f4c58 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160050547) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404)
* 213ac14377e762efdfc3b57f100935e3a21c6690 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160050547",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
}, {
"hash" : "213ac14377e762efdfc3b57f100935e3a21c6690",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160157730",
"triggerID" : "213ac14377e762efdfc3b57f100935e3a21c6690",
"triggerType" : "PUSH"
}, {
"hash" : "213ac14377e762efdfc3b57f100935e3a21c6690",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7431",
"triggerID" : "213ac14377e762efdfc3b57f100935e3a21c6690",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdff6e294f476e886a036e7cd462c0e0f79f4c58 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160050547) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7404)
* 213ac14377e762efdfc3b57f100935e3a21c6690 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160157730) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7431)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fdff6e294f476e886a036e7cd462c0e0f79f4c58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e8789334295f653c075177bcec9ebb5cd1facc8 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160025227) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389)
* fdff6e294f476e886a036e7cd462c0e0f79f4c58 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#discussion_r407458530
##########
File path: flink-python/pyflink/util/utils.py
##########
@@ -57,3 +59,48 @@ def load_java_class(class_name):
gateway = get_gateway()
context_classloader = gateway.jvm.Thread.currentThread().getContextClassLoader()
return context_classloader.loadClass(class_name)
+
+
+def is_instance_of(java_object, java_class):
+ gateway = get_gateway()
+ if isinstance(java_class, str):
+ param = java_class
+ elif isinstance(java_class, JavaClass):
+ param = get_java_class(java_class)
+ elif isinstance(java_class, JavaObject):
+ if not is_instance_of(java_class, gateway.jvm.Class):
+ param = java_class.getClass()
+ else:
+ param = java_class
+ else:
+ raise TypeError(
+ "java_class must be a string, a JavaClass, or a JavaObject")
+
+ return gateway.jvm.org.apache.flink.api.python.shaded.py4j.reflection.TypeUtil.isInstanceOf(
+ param, java_object)
+
+
+def get_j_env_configuration(t_env):
+ if is_instance_of(t_env._get_j_env(), "org.apache.flink.api.java.ExecutionEnvironment"):
+ j_configuration = t_env._get_j_env().getConfiguration()
+ else:
+ env_clazz = load_java_class(
+ "org.apache.flink.streaming.api.environment.StreamExecutionEnvironment")
+ method = env_clazz.getDeclaredMethod(
+ "getConfiguration", to_jarray(get_gateway().jvm.Class, []))
+ method.setAccessible(True)
+ j_configuration = method.invoke(t_env._get_j_env(), to_jarray(get_gateway().jvm.Object, []))
+ return j_configuration
+
+
+def is_local_executor(j_env, j_configuration):
Review comment:
Does it work to check if j_configuration.get(DeploymentOptions.TARGET) equals to "local"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11714: [FLINK-17114][python]
Set the default value of 'python.executable' to the path of the python
interpreter which launches the job when the job is executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612806248
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 9e8789334295f653c075177bcec9ebb5cd1facc8 (Mon Apr 13 08:29:52 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on issue #11714: [FLINK-17114][python]
Set the default value of 'python.executable' to the path of the python
interpreter which launches the job when the job is executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-613243489
@dianfu Thanks for your review! I have eliminate the code duplication in the latest commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#discussion_r407456184
##########
File path: flink-python/pyflink/table/table_environment.py
##########
@@ -1082,6 +1086,13 @@ def _from_elements(self, elements, schema):
finally:
os.unlink(temp_file.name)
+ def _set_python_executable_for_local_executor(self):
+ jvm = get_gateway().jvm
+ j_config = get_j_env_configuration(self)
+ if not j_config.containsKey(jvm.PythonOptions.PYTHON_EXECUTABLE.key()) \
Review comment:
Should we consider the case when a user has set the `python.executable` via `table_config.get_configuration()`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#discussion_r407460520
##########
File path: flink-python/pyflink/table/table_environment.py
##########
@@ -79,6 +81,8 @@ def __init__(self, j_tenv, serializer=PickleSerializer()):
self._j_tenv = j_tenv
self._is_blink_planner = TableEnvironment._judge_blink_planner(j_tenv)
self._serializer = serializer
+ # for local executor, use current sys.executable to launch UDF worker.
Review comment:
Update the comments as following:
`when running in MiniCluster, launch the Python UDF worker using the Python executable specified by sys.executable if users have not specified it explicitly via configuration python.executable.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#discussion_r407821291
##########
File path: flink-python/pyflink/table/tests/test_table_environment_api.py
##########
@@ -440,6 +450,12 @@ def test_table_environment_with_blink_planner(self):
self.assert_equals(results, ['2,hi,hello\n', '3,hello,hello\n'])
+ def test_set_sys_executable_for_local_mode(self):
Review comment:
We should improve the test to eliminate the code duplication. It is the same case for the other test cases in this file and there are many code duplication. You could refer to test_udf.py on how to eliminate the duplication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on a change in pull request #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#discussion_r407821291
##########
File path: flink-python/pyflink/table/tests/test_table_environment_api.py
##########
@@ -440,6 +450,12 @@ def test_table_environment_with_blink_planner(self):
self.assert_equals(results, ['2,hi,hello\n', '3,hello,hello\n'])
+ def test_set_sys_executable_for_local_mode(self):
Review comment:
We should improve the test to eliminate the duplication. It is the same case for the other test cases in this file and there are many code duplication. You could refer to test_udf.py on how to eliminate the duplication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11714:
[FLINK-17114][python] Set the default value of 'python.executable' to the
path of the python interpreter which launches the job when the job is
executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612809197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
}, {
"hash" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160025227",
"triggerID" : "9e8789334295f653c075177bcec9ebb5cd1facc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e8789334295f653c075177bcec9ebb5cd1facc8 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160025227) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7389)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] WeiZhong94 commented on issue #11714: [FLINK-17114][python]
Set the default value of 'python.executable' to the path of the python
interpreter which launches the job when the job is executed by LocalExecutor.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on issue #11714: [FLINK-17114][python] Set the default value of 'python.executable' to the path of the python interpreter which launches the job when the job is executed by LocalExecutor.
URL: https://github.com/apache/flink/pull/11714#issuecomment-612889346
@dianfu Thanks for your review! I have addressed your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services