You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "Imbruced (via GitHub)" <gi...@apache.org> on 2025/03/23 18:06:31 UTC
[PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Imbruced opened a new pull request, #1875:
URL: https://github.com/apache/sedona/pull/1875
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
- No, I haven't read it.
## Is this PR related to a ticket?
- Yes, and the PR name follows the format `[SEDONA-XXX] my subject`.
- Yes, and the PR name follows the format `[GH-XXX] my subject`.
- No:
- this is a documentation update. The PR name follows the format `[DOCS] my subject`
- this is a CI update. The PR name follows the format `[CI] my subject`
## What changes were proposed in this PR?
## How was this patch tested?
## Did this PR include necessary documentation updates?
- Yes, I am adding a new API. I am using the [current SNAPSHOT version number](https://github.com/apache/sedona/blob/99239524f17389fc4ae9548ea88756f8ea538bb9/pom.xml#L29) in `vX.Y.Z` format.
- Yes, I have updated the documentation.
- No, this PR does not affect any public API so no need to change the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-725] Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2899788732
@Imbruced 2 minor comments. Please also fix the failed lint action
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2864500896
@jiayuasu what do we do with the Apache Flink MR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-725] Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #1875:
URL: https://github.com/apache/sedona/pull/1875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2839770625
> Overall LGTM. Can you break this PR into 2 PRs? The first PR moves all Sedona PySpark stuff to Sedona Spark folder. The second PR adds the PyFlink. Does it make sense?
Yeah, sure. I'm sorry for not getting back to you sooner. Can we merge this one first? https://github.com/apache/sedona/pull/1924 It's helping to not spawn a lot of jobs in the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on code in PR #1875:
URL: https://github.com/apache/sedona/pull/1875#discussion_r2021694676
##########
.github/workflows/pyflink.yml:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Sedona Pyflink Test
+
+on:
+ push:
+ branches:
+ - master
+ paths:
+ - 'common/**'
+ - 'flink/**'
+ - 'flink-shaded/**'
+ - 'pom.xml'
+ - 'python/**'
+ - '.github/workflows/pyflink.yml'
+ pull_request:
+ branches:
+ - '*'
+ paths:
+ - 'common/**'
+ - 'flink/**'
+ - 'flink-shaded/**'
+ - 'pom.xml'
+ - 'python/**'
+ - '.github/workflows/pyflink.yml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ test:
+ runs-on: ubuntu-22.04
+ strategy:
+ matrix:
+ include:
+ - python: '3.10'
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-java@v4
+ with:
+ distribution: 'zulu'
+ java-version: '8'
+ - uses: actions/setup-python@v5
+ with:
+ python-version: ${{ matrix.python }}
+ - run: sudo apt-get -y install python3-pip python-dev-is-python3
+ - run: mvn package -pl "org.apache.sedona:sedona-flink-shaded_2.12" -am -DskipTests
+ - run: sudo pip3 install -U setuptools
+ - run: sudo pip3 install -U wheel
+ - run: sudo pip3 install -U virtualenvwrapper
+ - run: python3 -m pip install uv
Review Comment:
should we keep the tool we use consistent across the project? we are using pipenv in the rest of the project AFAIK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2829523760
Overall LGTM. Can you break this PR into 2 PRs? The first PR moves all Sedona PySpark stuff to Sedona Spark folder. The second PR adds the PyFlink. Does it make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-725] Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2865142527
@Imbruced LGTM. Can you add documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on code in PR #1875:
URL: https://github.com/apache/sedona/pull/1875#discussion_r2021707267
##########
.github/workflows/pyflink.yml:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Sedona Pyflink Test
+
+on:
+ push:
+ branches:
+ - master
+ paths:
+ - 'common/**'
+ - 'flink/**'
+ - 'flink-shaded/**'
+ - 'pom.xml'
+ - 'python/**'
+ - '.github/workflows/pyflink.yml'
+ pull_request:
+ branches:
+ - '*'
+ paths:
+ - 'common/**'
+ - 'flink/**'
+ - 'flink-shaded/**'
+ - 'pom.xml'
+ - 'python/**'
+ - '.github/workflows/pyflink.yml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ test:
+ runs-on: ubuntu-22.04
+ strategy:
+ matrix:
+ include:
+ - python: '3.10'
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-java@v4
+ with:
+ distribution: 'zulu'
+ java-version: '8'
+ - uses: actions/setup-python@v5
+ with:
+ python-version: ${{ matrix.python }}
+ - run: sudo apt-get -y install python3-pip python-dev-is-python3
+ - run: mvn package -pl "org.apache.sedona:sedona-flink-shaded_2.12" -am -DskipTests
+ - run: sudo pip3 install -U setuptools
+ - run: sudo pip3 install -U wheel
+ - run: sudo pip3 install -U virtualenvwrapper
+ - run: python3 -m pip install uv
Review Comment:
I plan to propose uv as a base tool for Python dependencies in Sedona. I think uv is the future https://github.com/astral-sh/uv
It's way more popular now and faster than, pipenv and poetry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-725] Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1875:
URL: https://github.com/apache/sedona/pull/1875#discussion_r2101546185
##########
docs/setup/flink/install-python.md:
##########
@@ -0,0 +1,34 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+To install Apache Sedona Python, you need to install the following steps:
+
+Install the required Python packages.
+
+```
+pip install apache-sedona[flink] shapely attr
+```
+
+Download the required JAR files from Maven Central:
+
+* sedona-flink-shaded_2.12:jar:1.7.1
+* geotools-wrapper-1.7.1-28.5.jar
Review Comment:
```suggestion
* geotools-wrapper-{{ sedona.current_geotools }}.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-725] Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1875:
URL: https://github.com/apache/sedona/pull/1875#discussion_r2101545897
##########
docs/setup/flink/install-python.md:
##########
@@ -0,0 +1,34 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+To install Apache Sedona Python, you need to install the following steps:
+
+Install the required Python packages.
+
+```
+pip install apache-sedona[flink] shapely attr
+```
+
+Download the required JAR files from Maven Central:
+
+* sedona-flink-shaded_2.12:jar:1.7.1
Review Comment:
```suggestion
* sedona-flink-shaded_2.12:jar:{{ sedona.current_version }}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2753099908
this is so cool. Do you think we'd better put pyflink as a new Python module to avoid conflicts with pyspark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "jbampton (via GitHub)" <gi...@apache.org>.
jbampton commented on code in PR #1875:
URL: https://github.com/apache/sedona/pull/1875#discussion_r2021972564
##########
.github/workflows/pyflink.yml:
##########
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Sedona Pyflink Test
+
+on:
+ push:
+ branches:
+ - master
+ paths:
+ - 'common/**'
+ - 'flink/**'
+ - 'flink-shaded/**'
+ - 'pom.xml'
+ - 'python/**'
+ - '.github/workflows/pyflink.yml'
+ pull_request:
+ branches:
+ - '*'
+ paths:
+ - 'common/**'
+ - 'flink/**'
+ - 'flink-shaded/**'
+ - 'pom.xml'
+ - 'python/**'
+ - '.github/workflows/pyflink.yml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ test:
+ runs-on: ubuntu-22.04
+ strategy:
+ matrix:
+ include:
+ - python: '3.10'
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-java@v4
+ with:
+ distribution: 'zulu'
+ java-version: '8'
+ - uses: actions/setup-python@v5
+ with:
+ python-version: ${{ matrix.python }}
+ - run: sudo apt-get -y install python3-pip python-dev-is-python3
+ - run: mvn package -pl "org.apache.sedona:sedona-flink-shaded_2.12" -am -DskipTests
+ - run: sudo pip3 install -U setuptools
+ - run: sudo pip3 install -U wheel
+ - run: sudo pip3 install -U virtualenvwrapper
+ - run: python3 -m pip install uv
Review Comment:
We were using `Ruff` - An extremely fast Python linter and code formatter, written in Rust, in our pre-commit framework but we changed to `black`.
It seems that Astral is modern popular fast tooling.
https://astral.sh/
https://github.com/astral-sh/ruff
https://github.com/apache/sedona/pull/1368
https://github.com/psf/black
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-725] Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2896484223
> @Imbruced LGTM. Can you add documentation?
@Imbruced gentle ping...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SEDONA-725] Add pyflink to Sedona. [sedona]
Posted by "Imbruced (via GitHub)" <gi...@apache.org>.
Imbruced commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2899401759
@jiayuasu docs added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SEDONA-725 Add pyflink to Sedona. [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #1875:
URL: https://github.com/apache/sedona/pull/1875#issuecomment-2839787007
@Imbruced Just merge that PR 😎
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org