You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "rowillia (via GitHub)" <gi...@apache.org> on 2023/07/26 03:37:44 UTC
[GitHub] [arrow] rowillia opened a new issue, #36879: Crash when performing `join` on a table larger than INT32_MAX bytes
rowillia opened a new issue, #36879:
URL: https://github.com/apache/arrow/issues/36879
### Describe the bug, including details regarding any error messages, version, and platform.
pyarrow version: 12.0.1
Minimal Repro:
```python
import pyarrow as pa
ROW_COUNT = 15_000
query = pa.Table.from_pydict({
"key": [bytes.fromhex('ABC123'), bytes.fromhex('DEF431')]
})
t = pa.Table.from_pydict(
{
"blob_2": [x.to_bytes(4, 'little') * 25_000 for x in range(ROW_COUNT)],
"blob_1": [x.to_bytes(4, 'little') * 25_000 for x in range(ROW_COUNT)],
"key": [x.to_bytes(4, 'little') for x in range(ROW_COUNT)]
},
schema=pa.schema(
[
("blob_2", pa.large_binary()),
("blob_1", pa.large_binary()),
("key", pa.binary()),
]
)
)
print(f"Table size: {t.nbytes} bytes (bigger than INT_MAX: {t.nbytes > (1 << 31)})")
for x in range(10):
print(f"Attempt {x}")
_ = query.join(t, 'key', join_type="inner")
```
Executing this results in a segfault:
```
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append
```
Note that flipping the join (e.g. `_ = t.join(q, 'key', join_type="inner")`) no longer segfaults.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Crash when performing `join` on a table larger than INT32_MAX bytes [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 closed issue #36879: [Python][C++] Crash when performing `join` on a table larger than INT32_MAX bytes
URL: https://github.com/apache/arrow/issues/36879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Crash when performing `join` on a table larger than INT32_MAX bytes [arrow]
Posted by "idailylife (via GitHub)" <gi...@apache.org>.
idailylife commented on issue #36879:
URL: https://github.com/apache/arrow/issues/36879#issuecomment-1905623416
Is this (https://github.com/apache/arrow/issues/33151) related?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Crash when performing `join` on a table larger than INT32_MAX bytes [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on issue #36879:
URL: https://github.com/apache/arrow/issues/36879#issuecomment-6349552762
Yes, this shares the root cause of the `large_string` case in #33151 and the more detailed report in #54885.
Although the join key here is ordinary `binary`, the **`large_binary` non-key columns** select Basic Hash Join. It encodes all build-side payload columns together in a separate `RowEncoder`, whose cumulative byte offsets and total-length calculation are signed 32-bit.
In this example, each blob is `4 * 25000 = 100000` bytes. Each of the two payload columns is individually below 2 GiB, but together their encoded payload is approximately `15000 * 2 * (100000 + 9) = 3000270000` bytes. The 9-byte overhead is the null marker plus 64-bit encoded length. This exceeds the encoder's signed 32-bit range and explains the invalid vector resize / `std::length_error`. The relevant [cumulative append calculation](https://github.com/apache/arrow/blob/14132604b3949ba1c25cc79fff2e1782b9efc413/cpp/src/arrow/compute/row/row_encoder_internal.cc#L358-L379) is still present on current upstream main.
The right input is the build side, so reversing the inputs makes the small query table the build side and avoids this accumulation. No matching keys are needed to trigger the failure: the right-side payload is encoded while building the hash table, before probing. An equivalent C++ Acero build-payload regression reproduces this overflow on unpatched upstream code; the original Python script was not rerun.
The 2024 Swiss Join >4 GiB row-offset change (#43389) concerns a different implementation and does not cover this Basic Join path.
Closing as a duplicate of #54885, where the same unresolved RowEncoder limitation will be tracked for both keys and payloads. This is consolidation, not a claim that the bug is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org