You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "zhangfengcdt (via GitHub)" <gi...@apache.org> on 2025/09/16 17:23:43 UTC
[PR] [GH-2351] [CI] Fix R CI flakiness with Spark download retry logic and timeout [sedona]
zhangfengcdt opened a new pull request, #2352:
URL: https://github.com/apache/sedona/pull/2352
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
## Is this PR related to a ticket?
- Yes, and the PR name follows the format `[GH-XXX] my subject`. Closes #<issue_number>
## What changes were proposed in this PR?
Add retry logic with exponential backoff for Spark downloads to make the workflow resilient to temporary network issues.
## How was this patch tested?
test on ci run
## Did this PR include necessary documentation updates?
- No, this PR does not affect any public API so no need to change the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2351] [CI] Fix R CI flakiness with Spark download from PySpark [sedona]
Posted by "zhangfengcdt (via GitHub)" <gi...@apache.org>.
zhangfengcdt commented on code in PR #2352:
URL: https://github.com/apache/sedona/pull/2352#discussion_r2353892384
##########
.github/workflows/r.yml:
##########
@@ -109,24 +112,34 @@ jobs:
distribution: 'temurin'
java-version: '11'
cache: 'maven'
- - name: Get OS name
- id: os-name
- run: |
- # `os_name` will be like "Ubuntu-20.04.1-LTS"
- OS_NAME=$(lsb_release -ds | sed 's/\s/-/g')
- echo "os-name=$OS_NAME" >> $GITHUB_OUTPUT
- - name: Cache Spark installations
- if: runner.os != 'Windows'
- uses: actions/cache@master
+ - uses: actions/setup-python@v5
with:
- path: ~/spark
- key: apache.sedona-apache-spark-${{ steps.os-name.outputs.os-name }}-${{ env.SPARK_VERSION }}
+ python-version: '3.11'
+ - name: Install PySpark
+ run: |
+ pip3 install pyspark==${SPARK_VERSION}
Review Comment:
It is defined in line 64:
SPARK_VERSION: ${{ matrix.spark }}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2351] [CI] Fix R CI flakiness with Spark download from PySpark [sedona]
Posted by "zhangfengcdt (via GitHub)" <gi...@apache.org>.
zhangfengcdt commented on PR #2352:
URL: https://github.com/apache/sedona/pull/2352#issuecomment-3300699639
> Maybe we should see if we can download the full distribution via pyspark? see how we do it in PySpark to avoid touching archive.apache.org: https://github.com/apache/sedona/blob/master/.github/workflows/python.yml#L157
Great suggestion! I have implemented the same way as python.yml and it works well without timeout. Now R tests finishes around 10 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2351] [CI] Fix R CI flakiness with Spark download from PySpark [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #2352:
URL: https://github.com/apache/sedona/pull/2352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2351] [CI] Fix R CI flakiness with Spark download retry logic and timeout [sedona]
Posted by "zhangfengcdt (via GitHub)" <gi...@apache.org>.
zhangfengcdt commented on PR #2352:
URL: https://github.com/apache/sedona/pull/2352#issuecomment-3299855236
Currently Apache Archive downloading is very slow.
Tested locally:
- Speed: ~44 KB/s (43,968 bytes/sec)
- 10MB would take: ~4 minutes
- 400MB Spark would take: ~2.5 hours!
Also, the spark package cache can NOT be found on github:
gh api repos/apache/sedona/actions/caches --jq '.actions_caches[].key'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2351] [CI] Fix R CI flakiness with Spark download from PySpark [sedona]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2352:
URL: https://github.com/apache/sedona/pull/2352#discussion_r2353890039
##########
.github/workflows/r.yml:
##########
@@ -109,24 +112,34 @@ jobs:
distribution: 'temurin'
java-version: '11'
cache: 'maven'
- - name: Get OS name
- id: os-name
- run: |
- # `os_name` will be like "Ubuntu-20.04.1-LTS"
- OS_NAME=$(lsb_release -ds | sed 's/\s/-/g')
- echo "os-name=$OS_NAME" >> $GITHUB_OUTPUT
- - name: Cache Spark installations
- if: runner.os != 'Windows'
- uses: actions/cache@master
+ - uses: actions/setup-python@v5
with:
- path: ~/spark
- key: apache.sedona-apache-spark-${{ steps.os-name.outputs.os-name }}-${{ env.SPARK_VERSION }}
+ python-version: '3.11'
+ - name: Install PySpark
+ run: |
+ pip3 install pyspark==${SPARK_VERSION}
+ - name: Download JAI libraries
+ run: |
+ PYSPARK_PATH=$(python3 -c "import pyspark; print(pyspark.__path__[0])")
+ wget --retry-connrefused --waitretry=10 --read-timeout=20 --timeout=15 --tries=5 https://repo.osgeo.org/repository/release/javax/media/jai_core/${JAI_CORE_VERSION}/jai_core-${JAI_CORE_VERSION}.jar
+ wget --retry-connrefused --waitretry=10 --read-timeout=20 --timeout=15 --tries=5 https://repo.osgeo.org/repository/release/javax/media/jai_codec/${JAI_CODEC_VERSION}/jai_codec-${JAI_CODEC_VERSION}.jar
+ wget --retry-connrefused --waitretry=10 --read-timeout=20 --timeout=15 --tries=5 https://repo.osgeo.org/repository/release/javax/media/jai_imageio/${JAI_IMAGEIO_VERSION}/jai_imageio-${JAI_IMAGEIO_VERSION}.jar
Review Comment:
[nitpick] The wget retry configuration is duplicated across three lines. Consider extracting the common retry parameters to a variable to reduce duplication and make maintenance easier.
```suggestion
WGET_RETRY_FLAGS="--retry-connrefused --waitretry=10 --read-timeout=20 --timeout=15 --tries=5"
wget $WGET_RETRY_FLAGS https://repo.osgeo.org/repository/release/javax/media/jai_core/${JAI_CORE_VERSION}/jai_core-${JAI_CORE_VERSION}.jar
wget $WGET_RETRY_FLAGS https://repo.osgeo.org/repository/release/javax/media/jai_codec/${JAI_CODEC_VERSION}/jai_codec-${JAI_CODEC_VERSION}.jar
wget $WGET_RETRY_FLAGS https://repo.osgeo.org/repository/release/javax/media/jai_imageio/${JAI_IMAGEIO_VERSION}/jai_imageio-${JAI_IMAGEIO_VERSION}.jar
```
##########
.github/workflows/r.yml:
##########
@@ -109,24 +112,34 @@ jobs:
distribution: 'temurin'
java-version: '11'
cache: 'maven'
- - name: Get OS name
- id: os-name
- run: |
- # `os_name` will be like "Ubuntu-20.04.1-LTS"
- OS_NAME=$(lsb_release -ds | sed 's/\s/-/g')
- echo "os-name=$OS_NAME" >> $GITHUB_OUTPUT
- - name: Cache Spark installations
- if: runner.os != 'Windows'
- uses: actions/cache@master
+ - uses: actions/setup-python@v5
with:
- path: ~/spark
- key: apache.sedona-apache-spark-${{ steps.os-name.outputs.os-name }}-${{ env.SPARK_VERSION }}
+ python-version: '3.11'
+ - name: Install PySpark
+ run: |
+ pip3 install pyspark==${SPARK_VERSION}
Review Comment:
The SPARK_VERSION environment variable is not defined in the workflow's env section. This will cause the pip install command to fail with an undefined variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2351] [CI] Fix R CI flakiness with Spark download from PySpark [sedona]
Posted by "zhangfengcdt (via GitHub)" <gi...@apache.org>.
zhangfengcdt commented on code in PR #2352:
URL: https://github.com/apache/sedona/pull/2352#discussion_r2353892757
##########
.github/workflows/r.yml:
##########
@@ -109,24 +112,34 @@ jobs:
distribution: 'temurin'
java-version: '11'
cache: 'maven'
- - name: Get OS name
- id: os-name
- run: |
- # `os_name` will be like "Ubuntu-20.04.1-LTS"
- OS_NAME=$(lsb_release -ds | sed 's/\s/-/g')
- echo "os-name=$OS_NAME" >> $GITHUB_OUTPUT
- - name: Cache Spark installations
- if: runner.os != 'Windows'
- uses: actions/cache@master
+ - uses: actions/setup-python@v5
with:
- path: ~/spark
- key: apache.sedona-apache-spark-${{ steps.os-name.outputs.os-name }}-${{ env.SPARK_VERSION }}
+ python-version: '3.11'
+ - name: Install PySpark
+ run: |
+ pip3 install pyspark==${SPARK_VERSION}
+ - name: Download JAI libraries
+ run: |
+ PYSPARK_PATH=$(python3 -c "import pyspark; print(pyspark.__path__[0])")
+ wget --retry-connrefused --waitretry=10 --read-timeout=20 --timeout=15 --tries=5 https://repo.osgeo.org/repository/release/javax/media/jai_core/${JAI_CORE_VERSION}/jai_core-${JAI_CORE_VERSION}.jar
+ wget --retry-connrefused --waitretry=10 --read-timeout=20 --timeout=15 --tries=5 https://repo.osgeo.org/repository/release/javax/media/jai_codec/${JAI_CODEC_VERSION}/jai_codec-${JAI_CODEC_VERSION}.jar
+ wget --retry-connrefused --waitretry=10 --read-timeout=20 --timeout=15 --tries=5 https://repo.osgeo.org/repository/release/javax/media/jai_imageio/${JAI_IMAGEIO_VERSION}/jai_imageio-${JAI_IMAGEIO_VERSION}.jar
Review Comment:
we will follow the same way as python.yml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [GH-2351] [CI] Fix R CI flakiness with Spark download retry logic and timeout [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #2352:
URL: https://github.com/apache/sedona/pull/2352#issuecomment-3300484254
Maybe we should see if we can download the full distribution via pyspark? see how we do it in PySpark to avoid touching archive.apache.org: https://github.com/apache/sedona/blob/master/.github/workflows/python.yml#L157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org