You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wankunde (via GitHub)" <gi...@apache.org> on 2026/04/15 10:06:49 UTC
[PR] [CPP] Ignore null values when copy velox buffer to arrow [gluten]
wankunde opened a new pull request, #11944:
URL: https://github.com/apache/gluten/pull/11944
## What changes are proposed in this pull request?
Bug fix for VeloxHashShuffleWriter.
Description:
* VeloxHashShuffleWriter::write() method will try to convert the input velox flat vectors into arrow buffer.
* For StringView vector, VeloxHashShuffleWriter will calculate the total size of all input StringView, and initialize the arrow buffer.
* But for null values, we don't really need to copy the dirty values to arrow buffers, and can save the execution memory.
Test logs in our productions:
```log
I20260415 17:50:28.520978 5474 VeloxHashShuffleWriter.cc:156] collectFlatVectorBufferStringView for VARCHAR
I20260415 17:50:28.521070 5474 VeloxHashShuffleWriter.cc:110] DEBUG, lengthBufferSize = 40, flatVector->size() = 10
I20260415 17:50:28.521078 5474 VeloxHashShuffleWriter.cc:114] DEBUG flatVector = [FLAT VARCHAR: 10 elements, 9 nulls] data: 0: null
1: null
2: null
3: 114535255
4: null
5: null
6: null
7: null
8: null
I20260415 17:50:28.521092 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 0, isNull = true, rawValues[i].size() = 1957213616, length = 0, offset = 0
I20260415 17:50:28.521100 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 1, isNull = true, rawValues[i].size() = 2588080704, length = 0, offset = 0
I20260415 17:50:28.521106 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 2, isNull = true, rawValues[i].size() = 2706762760, length = 0, offset = 0
I20260415 17:50:28.521111 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 3, isNull = false, rawValues[i].size() = 9, length = 9, offset = 9
I20260415 17:50:28.521117 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 4, isNull = true, rawValues[i].size() = 0, length = 0, offset = 9
I20260415 17:50:28.521122 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 5, isNull = true, rawValues[i].size() = 1, length = 0, offset = 9
I20260415 17:50:28.521127 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 6, isNull = true, rawValues[i].size() = 2832791872, length = 0, offset = 9
I20260415 17:50:28.521132 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 7, isNull = true, rawValues[i].size() = 2588222376, length = 0, offset = 9
I20260415 17:50:28.521138 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 8, isNull = true, rawValues[i].size() = 0, length = 0, offset = 9
I20260415 17:50:28.521143 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 9, isNull = true, rawValues[i].size() = 0, length = 0, offset = 9
```
## How was this patch tested?
Tested in out production environment.
## Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Ignore null values when copy velox string buffer to arrow [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #11944:
URL: https://github.com/apache/gluten/pull/11944#issuecomment-4261775847
@wankunde thanks for the fix, would you please also add a unit test to guard on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CPP] Ignore null values when copy velox buffer to arrow [gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #11944:
URL: https://github.com/apache/gluten/pull/11944#issuecomment-4259565361
Retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Ignore null values when copy velox string buffer to arrow [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW merged PR #11944:
URL: https://github.com/apache/gluten/pull/11944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Ignore null values when copy velox string buffer to arrow [gluten]
Posted by "wankunde (via GitHub)" <gi...@apache.org>.
wankunde commented on PR #11944:
URL: https://github.com/apache/gluten/pull/11944#issuecomment-4277589594
> @wankunde thanks for the fix, would you please also add a unit test to guard on this?
Hi， @zhouyuan I did not reproduce this issue in my local UT, however, according to the velox document (https://facebookincubator.github.io/velox/develop/vectors.html), the size of an null StringViews does indeed exhibit unexpected behavior.
<img width="815" height="273" alt="image" src="https://github.com/user-attachments/assets/e1d24a07-d879-4201-b9b2-2394a14ee87f" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org