You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Kontinuation (via GitHub)" <gi...@apache.org> on 2025/05/22 09:25:10 UTC
[PR] Bump arrow to 18.3.0 [datafusion-comet]
Kontinuation opened a new pull request, #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773
## Which issue does this PR close?
Closes #1615 .
## Rationale for this change
Issue #1615 is caused by a bug in arrow-java. The latest release 18.3.0 has shipped the [fix for that bug](https://github.com/apache/arrow-java/pull/705) so we can simply bump the version of arrow to fix the issue.
I have also found that https://github.com/apache/arrow-java/pull/707 is also shipped with 18.3.0, it has fixed the sliced string array importing issue, so we can remove the workaround in our code.
## What changes are included in this PR?
* Bump version of arrow from 16.0.0 to 18.3.0
* Remove our workaround for importing sliced string/binary arrays
## How are these changes tested?
1. Passing existing tests
2. Manually run TPC-DS SF=100 to verify that #1615 was fixed, and #540 does not revive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Bump arrow to 18.3.0 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773#issuecomment-2906822147
> I have rebased this branch onto the latest main. The [ubuntu-latest/java 17-spark-4.0/java](https://github.com/apache/datafusion-comet/actions/runs/15222051464/job/42819140331?pr=1773#logs) test slowed down then failed. I'm not sure if it is transient or not.
>
> ```
> Sat, 24 May 2025 02:45:30 GMT - columnar shuffle on map [float] (3 seconds, 751 milliseconds)
> Sat, 24 May 2025 02:45:34 GMT - columnar shuffle on map [double] (3 seconds, 776 milliseconds)
> Sat, 24 May 2025 02:45:42 GMT - columnar shuffle on map [date] (7 seconds, 957 milliseconds)
> Sat, 24 May 2025 02:54:54 GMT - columnar shuffle on map [timestamp] (9 minutes, 11 seconds)
> Sat, 24 May 2025 03:05:03 GMT [INFO] ------------------------------------------------------------------------
> Sat, 24 May 2025 03:05:03 GMT [INFO] Reactor Summary for Comet Project Parent POM 0.9.0-SNAPSHOT:
> ```
Yes, this is a transient issue and we have not figured out a root cause. It did start happening just after we changed the DataFusion dependency to be a git dependency on that latest code, soi that is one clue. We do not understand why this only happens on the Spark 4 builds.
Here are the timings for the above test from the Spark 3.5 run:
```
2025-05-24T02:13:42.8902668Z - columnar shuffle on map [bool] (4 seconds, 787 milliseconds)
2025-05-24T02:13:47.7676309Z - columnar shuffle on map [byte] (4 seconds, 877 milliseconds)
2025-05-24T02:13:52.3613358Z - columnar shuffle on map [short] (4 seconds, 593 milliseconds)
2025-05-24T02:13:56.8987095Z - columnar shuffle on map [int] (4 seconds, 537 milliseconds)
2025-05-24T02:14:01.6694624Z - columnar shuffle on map [long] (4 seconds, 770 milliseconds)
2025-05-24T02:14:06.2930589Z - columnar shuffle on map [float] (4 seconds, 623 milliseconds)
2025-05-24T02:14:10.9593270Z - columnar shuffle on map [double] (4 seconds, 666 milliseconds)
2025-05-24T02:14:15.4833575Z - columnar shuffle on map [date] (4 seconds, 524 milliseconds)
2025-05-24T02:14:20.0771823Z - columnar shuffle on map [timestamp] (4 seconds, 594 milliseconds)
2025-05-24T02:14:24.7802633Z - columnar shuffle on map [decimal] (4 seconds, 703 milliseconds)
2025-05-24T02:14:29.4309000Z - columnar shuffle on map [string] (4 seconds, 651 milliseconds)
2025-05-24T02:14:33.6948025Z - columnar shuffle on map [binary] (4 seconds, 264 milliseconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Bump arrow to 18.3.0 [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773#issuecomment-2900746498
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1773?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 57.03%. Comparing base [(`f09f8af`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`b865e0e`)](https://app.codecov.io/gh/apache/datafusion-comet/commit/b865e0ee268965edb1ff0bdee8cd0d9ec2b4d1f7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 205 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/1773?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...rc/main/java/org/apache/arrow/c/ArrowImporter.java](https://app.codecov.io/gh/apache/datafusion-comet/pull/1773?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Farrow%2Fc%2FArrowImporter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9hcnJvdy9jL0Fycm93SW1wb3J0ZXIuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/1773?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #1773 +/- ##
============================================
+ Coverage 56.12% 57.03% +0.91%
- Complexity 976 1052 +76
============================================
Files 119 128 +9
Lines 11743 12467 +724
Branches 2251 2330 +79
============================================
+ Hits 6591 7111 +520
- Misses 4012 4168 +156
- Partials 1140 1188 +48
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/1773?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Bump arrow to 18.3.0 [datafusion-comet]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773#issuecomment-2906791709
I have rebased this branch onto the latest main. The [ubuntu-latest/java 17-spark-4.0/java](https://github.com/apache/datafusion-comet/actions/runs/15222051464/job/42819140331?pr=1773#logs) test slowed down then failed. I'm not sure if it is transient or not.
```
Sat, 24 May 2025 02:45:30 GMT - columnar shuffle on map [float] (3 seconds, 751 milliseconds)
Sat, 24 May 2025 02:45:34 GMT - columnar shuffle on map [double] (3 seconds, 776 milliseconds)
Sat, 24 May 2025 02:45:42 GMT - columnar shuffle on map [date] (7 seconds, 957 milliseconds)
Sat, 24 May 2025 02:54:54 GMT - columnar shuffle on map [timestamp] (9 minutes, 11 seconds)
Sat, 24 May 2025 03:05:03 GMT [INFO] ------------------------------------------------------------------------
Sat, 24 May 2025 03:05:03 GMT [INFO] Reactor Summary for Comet Project Parent POM 0.9.0-SNAPSHOT:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Bump arrow to 18.3.0 [datafusion-comet]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773#issuecomment-2900575118
Unfortunately Arrow Java requires Java 11 since version 18, so we cannot bump the version number as we have to support Java 8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Bump arrow to 18.3.0 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773#issuecomment-2901657681
Thanks @Kontinuation. Let's talk about dropping support for Java 8? https://github.com/apache/datafusion-comet/issues/1775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Bump arrow to 18.3.0 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: Bump arrow to 18.3.0 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1773:
URL: https://github.com/apache/datafusion-comet/pull/1773#issuecomment-2905570804
@Kontinuation We merged the PR to stop building with JDK 8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org