You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2026/08/05 15:25:24 UTC
[PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
raulcd opened a new pull request, #50814:
URL: https://github.com/apache/arrow/pull/50814
### Rationale for this change
The [test-conda-python-3.10-hdfs-2.9.2](https://github.com/ursacomputing/crossbow/actions/runs/22944679479/job/66593987920) failed sporadically on this PR when upgrading some dependencies due to time out rebuilding the docker image.
A big chunk of time was spent on the job just downloading HDFS 2.9.2.
HDFS 2.9.2 was released on 2018. We test 2.9.2 and 3.2.1, released on 2019.
### What changes are included in this PR?
Bump to newer versions, latests are 3.5.0 (2026) and from the v2 series, 2.10.2 (2022)
### Are these changes tested?
Yes via the archery jobs
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5678835814
I will merge this if CI is successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50814:
URL: https://github.com/apache/arrow/pull/50814#discussion_r3721956840
##########
dev/tasks/tasks.yml:
##########
@@ -771,7 +771,7 @@ tasks:
image: conda-python-dask
{% endfor %}
-{% for hdfs_version in ["2.9.2", "3.2.1"] %}
+{% for hdfs_version in ["2.10.2", "3.5.0"] %}
test-conda-python-3.11-hdfs-{{ hdfs_version }}:
ci: github
Review Comment:
The PR description references sporadic failures in `test-conda-python-3.10-hdfs-2.9.2`, but this change updates a `test-conda-python-3.11-hdfs-*` task matrix. If the intent is to bump the versions for the 3.10 HDFS jobs, consider updating the PR description to match the affected job(s), or update the corresponding 3.10 task definitions if they exist elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5194007100
:warning: GitHub issue #49524 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5204608280
Revision: 34191f1f60e69407324c9ea7167432284bc24503
Submitted crossbow builds: [ursacomputing/crossbow @ actions-b83945da96](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b83945da96)
|Task|Status|
|----|------|
|test-conda-python-3.11-hdfs-2.10.2|[](https://github.com/ursacomputing/crossbow/actions/runs/31101321470/job/92615313721)|
|test-conda-python-3.11-hdfs-3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/31101321679/job/92615314719)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/31101322321/job/92615317039)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd merged PR #50814:
URL: https://github.com/apache/arrow/pull/50814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5204583076
@github-actions crossbow submit test-*hdfs* test*spark*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5678858545
Revision: 51fd13d87fb6e82f94fe3a9e7e4f885221b3aa56
Submitted crossbow builds: [ursacomputing/crossbow @ actions-a22f77e929](https://github.com/ursacomputing/crossbow/branches/all?query=actions-a22f77e929)
|Task|Status|
|----|------|
|test-conda-python-3.11-hdfs-2.10.2|[](https://github.com/ursacomputing/crossbow/actions/runs/34959265933/job/104348687501)|
|test-conda-python-3.11-hdfs-3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/34959266476/job/104348688753)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/34959265463/job/104348684736)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5193748756
@github-actions crossbow submit test-*hdfs*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5203916676
@github-actions crossbow submit test-*hdfs*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5194596019
ok, 3.4.3 worked locally but 3.5.0 doesn't, I'll investigate requirements as it just seems like a JDK version issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5203950474
Revision: 36535404ee49e5d9f45f9a08b257fddff04b930e
Submitted crossbow builds: [ursacomputing/crossbow @ actions-551ab1d4c1](https://github.com/ursacomputing/crossbow/branches/all?query=actions-551ab1d4c1)
|Task|Status|
|----|------|
|test-conda-python-3.11-hdfs-2.10.2|[](https://github.com/ursacomputing/crossbow/actions/runs/31096590965/job/92599973574)|
|test-conda-python-3.11-hdfs-3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/31096592318/job/92599978692)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5194038219
Revision: a7c35cbbf260b26e7c55c41ca645cb5d707cb9c2
Submitted crossbow builds: [ursacomputing/crossbow @ actions-f937a83a40](https://github.com/ursacomputing/crossbow/branches/all?query=actions-f937a83a40)
|Task|Status|
|----|------|
|test-conda-python-3.11-hdfs-2.10.2|[](https://github.com/ursacomputing/crossbow/actions/runs/31022245126/job/92361652783)|
|test-conda-python-3.11-hdfs-3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/31022245330/job/92361653403)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5244435108
Looks like a clean bump!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50814:
URL: https://github.com/apache/arrow/pull/50814#discussion_r4014696308
##########
dev/tasks/tasks.yml:
##########
@@ -765,7 +765,7 @@ tasks:
image: conda-python-dask
{% endfor %}
-{% for hdfs_version in ["2.9.2", "3.2.1"] %}
+{% for hdfs_version in ["2.10.2", "3.5.0"] %}
Review Comment:
The `2.10.2` job inherits the global `JDK=17` from `.env` because this matrix only overrides `HDFS` and `PYTHON`. Hadoop 2.10.2's release build requirements specify JDK 1.7/1.8, while JDK 17 is required by the 3.5.0 line, so the new v2 job is running an unsupported JVM combination. Please parameterize the JDK per HDFS version and use JDK 8 for 2.10.2 while retaining 17 for 3.5.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50814:
URL: https://github.com/apache/arrow/pull/50814#issuecomment-5678831156
@github-actions crossbow submit test-*hdfs* test*spark*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org