You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gaborgsomogyi (via GitHub)" <gi...@apache.org> on 2023/08/28 11:20:34 UTC
[GitHub] [flink] gaborgsomogyi opened a new pull request, #23306: [FLINK-32981][PYTHON] Add python dynamic Flink home detection
gaborgsomogyi opened a new pull request, #23306:
URL: https://github.com/apache/flink/pull/23306
## What is the purpose of the change
During `pyflink` library compilation Flink home is calculated from the provided `pyflink` version which is normally something like: `1.19.dev0`. Such case `.dev0` is replaced to `-SNAPSHOT` which ends-up in hardcoded home directory: `../../flink-dist/target/flink-1.19-SNAPSHOT-bin/flink-1.19-SNAPSHOT`. This is fine as long as one uses the basic version types described [here](https://peps.python.org/pep-0440/#developmental-releases). In order to support any kind of `pyflink` version one can dynamically find out the Flink home directory through globbing. In this PR I've added this dynamic Flink home detection.
## Brief change log
Dynamic Flink home detection through globbing.
## Verifying this change
Manually:
```
# Change pyflink version to: 1.19.post0.dev0
# OR
# Change Flink version to: 1.19-foo-SNAPSHOT
cd flink
mvn clean install -DskipTests
conda env remove -n venv
conda create -y -n venv python=3.10
conda activate venv
pip3 install -r flink-python/dev/dev-requirements.txt
cd flink-python
python3 setup.py sdist bdist_wheel
cd apache-flink-libraries
python3 setup.py sdist
```
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a diff in pull request #23306: [FLINK-32981][PYTHON] Add python dynamic Flink home detection
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on code in PR #23306:
URL: https://github.com/apache/flink/pull/23306#discussion_r1307408172
##########
flink-python/apache-flink-libraries/setup.py:
##########
@@ -98,8 +98,12 @@ def find_file_path(pattern):
file=sys.stderr)
sys.exit(-1)
flink_version = VERSION.replace(".dev0", "-SNAPSHOT")
- FLINK_HOME = os.path.abspath(
- "../../flink-dist/target/flink-%s-bin/flink-%s" % (flink_version, flink_version))
+ flink_homes = glob.glob('../../flink-dist/target/flink-*-bin/flink-*')
+ if len(flink_homes) != 1:
+ print("More than one Flink home directory found {0}".format(flink_homes),
Review Comment:
Good point, tried it out only w/ more than one dir. Changing...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23306: [FLINK-32981][PYTHON] Add python dynamic Flink home detection
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23306:
URL: https://github.com/apache/flink/pull/23306#issuecomment-1695526812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c6a76e4dd41c6ea386e5ab640e299adf46a5fcac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6a76e4dd41c6ea386e5ab640e299adf46a5fcac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6a76e4dd41c6ea386e5ab640e299adf46a5fcac UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] ferenc-csaky commented on a diff in pull request #23306: [FLINK-32981][PYTHON] Add python dynamic Flink home detection
Posted by "ferenc-csaky (via GitHub)" <gi...@apache.org>.
ferenc-csaky commented on code in PR #23306:
URL: https://github.com/apache/flink/pull/23306#discussion_r1307401154
##########
flink-python/apache-flink-libraries/setup.py:
##########
@@ -98,8 +98,12 @@ def find_file_path(pattern):
file=sys.stderr)
sys.exit(-1)
flink_version = VERSION.replace(".dev0", "-SNAPSHOT")
- FLINK_HOME = os.path.abspath(
- "../../flink-dist/target/flink-%s-bin/flink-%s" % (flink_version, flink_version))
+ flink_homes = glob.glob('../../flink-dist/target/flink-*-bin/flink-*')
+ if len(flink_homes) != 1:
+ print("More than one Flink home directory found {0}".format(flink_homes),
Review Comment:
Is it possible to get to this line if we did not package Flink yet? Even if not, I could end up cleaning up the dist target dir, so there is a possibility that `len(flink_homes)` will be `0` and in that case the "More than one..." error msg. is misleading. I think this can be rephrased in a more general way, maybe "Ambiguous Flink home directory found: ..." or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #23306: [FLINK-32981][PYTHON] Add python dynamic Flink home detection
Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #23306:
URL: https://github.com/apache/flink/pull/23306#issuecomment-1695517850
cc @gyfora @mbalassi @morhidi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] mbalassi merged pull request #23306: [FLINK-32981][PYTHON] Add python dynamic Flink home detection
Posted by "mbalassi (via GitHub)" <gi...@apache.org>.
mbalassi merged PR #23306:
URL: https://github.com/apache/flink/pull/23306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org