You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jarohen (via GitHub)" <gi...@apache.org> on 2024/03/26 10:38:50 UTC
[I] [Java] ListVector.setNull doesn't update lastSet - performance issue [arrow]
jarohen opened a new issue, #40796:
URL: https://github.com/apache/arrow/issues/40796
### Describe the enhancement requested
In `ListVector`, `setNull` doesn't update lastSet. This means that, if you set many null values in a row, the offset buffer is unnecessarily re-set for the intervening values - e.g.:
* If I `.setNull(3)` with `lastSet == 2`, I set offset 3
* If I then `.setNull(4)`, `.setNull(5)`, `.setNull(6)`, these set 3->4, 3->5, and 3->6 respectively.
* With large and sparse enough vectors, this adds significant time to our internal benchmarks.
Naively, I'd put `lastSet = index` in `setNull` in the same way as `startNewValue`, but aware there might be an implicit/nuanced reason for not doing so?
Cheers,
James
### Component(s)
Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] ListVector.setNull doesn't update lastSet - performance issue [arrow]
Posted by "jarohen (via GitHub)" <gi...@apache.org>.
jarohen commented on issue #40796:
URL: https://github.com/apache/arrow/issues/40796#issuecomment-2021706351
@vibhatha PR opened - #40810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] ListVector.setNull doesn't update lastSet - performance issue [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #40796:
URL: https://github.com/apache/arrow/issues/40796#issuecomment-2022783757
Issue resolved by pull request 40810
https://github.com/apache/arrow/pull/40810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] ListVector.setNull doesn't update lastSet - performance issue [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #40796:
URL: https://github.com/apache/arrow/issues/40796#issuecomment-2021670323
@jarohen Thanks for raising this issue. I haven't evaluated this on my own yet, but just curious, planning on making a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] ListVector.setNull doesn't update lastSet - performance issue [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #40796:
URL: https://github.com/apache/arrow/issues/40796#issuecomment-2021709481
> Hey @vibhatha 👋 I certainly could, although I'm a little concerned that our fix misses a nuance of how `lastSet` is intended to be used. If someone with knowledge of the context could confirm we haven't missed anything, yep - more than happy to submit a PR.
I totally understand. Let's see if we can review the PR and make improvements. Thanks for working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] ListVector.setNull doesn't update lastSet - performance issue [arrow]
Posted by "jarohen (via GitHub)" <gi...@apache.org>.
jarohen commented on issue #40796:
URL: https://github.com/apache/arrow/issues/40796#issuecomment-2021691615
Hey @vibhatha :wave: I certainly could, although I'm a little concerned that our fix misses a nuance of how `lastSet` is intended to be used. If someone with knowledge of the context could confirm we haven't missed anything, yep - more than happy to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] ListVector.setNull doesn't update lastSet - performance issue [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #40796: [Java] ListVector.setNull doesn't update lastSet - performance issue
URL: https://github.com/apache/arrow/issues/40796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org