You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "PPL143 (via GitHub)" <gi...@apache.org> on 2025/06/04 17:09:51 UTC
[I] join statement with regex causes panic [datafusion]
PPL143 opened a new issue, #16252:
URL: https://github.com/apache/datafusion/issues/16252
### Describe the bug
Using regex functions or regex logical operators will result to overflow.
### To Reproduce
> CREATE EXTERNAL TABLE bigdata STORED AS CSV LOCATION 'bigdata.csv' OPTIONS ('has_header' 'true');
0 row(s) fetched.
Elapsed 0.007 seconds.
>
CREATE EXTERNAL TABLE find STORED AS CSV LOCATION 'find.csv' OPTIONS ('has_header' 'true');
0 row(s) fetched.
Elapsed 0.001 seconds.
>
select count(1) from bigdata;
+-----------------+
| count(Int64(1)) |
+-----------------+
| 2278791 |
+-----------------+
1 row(s) fetched.
Elapsed 0.023 seconds.
> select count(1) from find;
+-----------------+
| count(Int64(1)) |
+-----------------+
| 27 |
+-----------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
>
select * from bigdata join find ON bigdata.a ~* find.regex;
thread 'tokio-runtime-worker' panicked at ..\index.crates.io-1949cf8c6b5b557f\arrow-select-55.1.0\src\take.rs:480:45:
overflow
### Expected behavior
_No response_
### Additional context
[data.zip](https://github.com/user-attachments/files/20595521/data.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] join statement with regex causes panic [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #16252:
URL: https://github.com/apache/datafusion/issues/16252#issuecomment-3702912857
This now seems to work ... extremely slowly though
```sql
$ datafusion-cli
DataFusion CLI v51.0.0
> CREATE EXTERNAL TABLE bigdata STORED AS CSV LOCATION 'benchmarks/data/bigdata.csv' OPTIONS ('has_header' 'true');
0 row(s) fetched.
Elapsed 0.005 seconds.
> CREATE EXTERNAL TABLE find STORED AS CSV LOCATION 'benchmarks/data/find.csv' OPTIONS ('has_header' 'true');
0 row(s) fetched.
Elapsed 0.001 seconds.
> select count(1) from bigdata;
+-----------------+
| count(Int64(1)) |
+-----------------+
| 2278791 |
+-----------------+
1 row(s) fetched.
Elapsed 0.019 seconds.
> select count(1) from find;
+-----------------+
| count(Int64(1)) |
+-----------------+
| 27 |
+-----------------+
1 row(s) fetched.
Elapsed 0.003 seconds.
> select * from bigdata join find ON bigdata.a ~* find.regex;
+---------+------------+----+--------------------------------------------------------------+
| id | a | id | regex |
+---------+------------+----+--------------------------------------------------------------+
| 2186283 | 3<_C:_g7lm | 4 | 3(.+)c(.+) |
| 2186288 | bB[60FsIB2 | 7 | 6(.+)f(.+) |
| 2186293 | o2^98R^BmX | 3 | 2(.+)b(.+) |
| 2186296 | `4ruUpFgdc | 5 | 4(.+)d(.+) |
| 2186305 | 5RtP\eofPf | 6 | 5(.+)e(.+) |
| 2186306 | 7^8jhQy]fO | 8 | 7(.+)f(.+) |
| 2186306 | 7^8jhQy]fO | 9 | 8(.+)f(.+) |
| 2186341 | W3TX5dEd4B | 6 | 5(.+)e(.+) |
| 2186350 | bt1WHsAxp< | 2 | 1(.+)a(.+) |
| 2186364 | 4DrDXt5]0w | 5 | 4(.+)d(.+) |
| 2186391 | EH\432=_C6 | 4 | 3(.+)c(.+) |
| 2186399 | e7^c?f;Xnu | 8 | 7(.+)f(.+) |
| 2186408 | 2tKSBj5g7> | 3 | 2(.+)b(.+) |
| 2186410 | SM6iGFAj;_ | 7 | 6(.+)f(.+) |
| 2186424 | Q54tDfJ[Wv | 5 | 4(.+)d(.+) |
| 2186459 | @:1TOntmAG | 2 | 1(.+)a(.+) |
| 2186485 | UEVn?S2wB? | 3 | 2(.+)b(.+) |
| 2186498 | Q5X?jteTsZ | 6 | 5(.+)e(.+) |
| 2186506 | I5RhE7^2Dd | 6 | 5(.+)e(.+) |
| 2186518 | 7^05F``Gd3 | 8 | 7(.+)f(.+) |
| 2186551 | :3<cOqgFIe | 4 | 3(.+)c(.+) |
| 2186566 | 2bv6cm9jFx | 7 | 6(.+)f(.+) |
| 2186576 | fu3R\eJSCl | 4 | 3(.+)c(.+) |
| 2186580 | 4p7Ij@:aFk | 8 | 7(.+)f(.+) |
| 2186602 | S73RC<;?WV | 4 | 3(.+)c(.+) |
| 2186604 | <:7yfXM:]7 | 8 | 7(.+)f(.+) |
| 2186612 | <5S4y2ylDP | 5 | 4(.+)d(.+) |
| 2186616 | o5CnDH8eGy | 6 | 5(.+)e(.+) |
| 2186629 | 1UVAY5EN\> | 2 | 1(.+)a(.+) |
| 2186651 | I`\16KSKFF | 7 | 6(.+)f(.+) |
| 2186663 | B[2e3GHc0D | 4 | 3(.+)c(.+) |
| 2186665 | d5me<KJdG] | 6 | 5(.+)e(.+) |
| 2186668 | 8Dul4xfY4c | 9 | 8(.+)f(.+) |
| 2186672 | R5LtE4hyKQ | 6 | 5(.+)e(.+) |
| 2186681 | cp61[jpaix | 2 | 1(.+)a(.+) |
| 2186700 | 1FaFrmqrt< | 2 | 1(.+)a(.+) |
| 2186704 | B7687iqfpg | 7 | 6(.+)f(.+) |
| 2186704 | B7687iqfpg | 8 | 7(.+)f(.+) |
| 2186704 | B7687iqfpg | 9 | 8(.+)f(.+) |
| 2186704 | B7687iqfpg | 25 | ([A-Z]{1})(0[0-9]{4}|[1-6][0-9]{4}|7[0-8][0-9]{3}|7[0-9]{3}) |
| . |
| . |
| . |
+---------+------------+----+--------------------------------------------------------------+
189560 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
Elapsed 3180.419 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] join statement with regex causes panic [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on issue #16252:
URL: https://github.com/apache/datafusion/issues/16252#issuecomment-2941848939
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] join statement with regex causes panic [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on issue #16252:
URL: https://github.com/apache/datafusion/issues/16252#issuecomment-2944193292
https://github.com/apache/arrow-rs/blame/6deefb7d6163dceb360236078d5d9f10d1021422/arrow-select/src/take.rs#L480
When the length overflows, it will panic, which may not be a good idea. Because overflow does not necessarily mean there is a problem with the program. In the current case, it is just because the indices array is too large. One possible solution is to automatically convert to LargeUtf8 when we find overflow, but this may break some schema consistency checks. Another solution is to allow returning Vec<ArrayRef>. But these will have some API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org