You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "niebayes (via GitHub)" <gi...@apache.org> on 2025/03/27 13:49:02 UTC
[I] [Bug] datafusion-cli may failed to read csv files [datafusion]
niebayes opened a new issue, #15456:
URL: https://github.com/apache/datafusion/issues/15456
### Describe the bug
Originally posted at https://github.com/clflushopt/tpchgen-rs/issues/73
**Describe the bug**
Using datafusion-cli to read csv files generated by tpchgen-cli sometimes fails.
This issue was originally posted at https://github.com/clflushopt/tpchgen-rs/pull/66#issuecomment-2754881275
**To Reproduce**
Steps to reproduce the behavior:
1. Git clone the latest of the main branch (rev: https://github.com/clflushopt/tpchgen-rs/commit/cb325ad8a1754c21aa00708bf4302b1899d8bef4)
2. Build with cargo: `cargo build --release`
3. Generate csv files in a dedicated directory: `./target/release/tpchgen-cli -f csv -o gen`
4. Install the latest release of `datafusion-cli`: `cargo install datafusion-cli`
5. Ensure `datafusion-cli` is of version `46.0.1`
- Start datafusion-cli by running `datafusion-cli` in your terminal.
- Run sql `select version()` which should print a message containing `46.0.1`.
6. Use `datafusion-cli` to read the `part.csv` file by running sql `select * from './gen/part.csv'`. Note be sure to not add extra limit clause.
7. The datafusion-cli should report an error like: `Arrow error: Parser error: Error while parsing value p_partkey for column 0 at line 24597`
**Expected behavior**
All csv files generated by tpchgen-cli should be readable by datafusion-cli.
**Screenshots**
<img width="874" alt="Image" src="https://github.com/user-attachments/assets/53dce417-9709-4786-a777-b587ec35673b" />
**Environment (please complete the following information):**
- OS: Darwin Mac 24.0.0 Darwin Kernel Version 24.0.0: Mon Aug 12 20:52:18 PDT 2024; root:xnu-11215.1.10~2/RELEASE_ARM64_T8122 arm64
- Compiler Version: 1.85.1
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2760334856
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2761675489
Nice find @chenkovsky -- so looks like there is some bug in the data generator after all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "niebayes (via GitHub)" <gi...@apache.org>.
niebayes commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2758144809
This https://github.com/apache/arrow-rs/issues/7344 may help debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #15456: [Bug] datafusion-cli may fail to read csv files
URL: https://github.com/apache/datafusion/issues/15456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2759121460
I also tried disabling file scan repartition and it still happens
```sql
> set datafusion.optimizer.repartition_file_scans = false;
0 row(s) fetched.
Elapsed 0.001 seconds.
> explain select * from 'part.csv';
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan | TableScan: part.csv projection=[p_partkey, p_name, p_mfgr, p_brand, p_type, p_size, p_container, p_retailprice, p_comment] |
| physical_plan | DataSourceExec: file_groups={1 group: [[Users/andrewlamb/Downloads/part.csv]]}, projection=[p_partkey, p_name, p_mfgr, p_brand, p_type, p_size, p_container, p_retailprice, p_comment], file_type=csv, has_header=true |
| | |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.017 seconds.
> select * from 'part.csv';
Arrow error: Parser error: Error while parsing value p_partkey for column 0 at line 100001
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "niebayes (via GitHub)" <gi...@apache.org>.
niebayes commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2760766577
> why there are two head rows
I didn't find this. You might find the cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2759119415
The problem can be reproduced without the tpchdbgen CLI:
Step 1: download [part.zip](https://github.com/user-attachments/files/19492616/part.zip)
Step 2: unzip
```shell
unzip part.zip
```
Step 3: run query:
```shell
$ datafusion-cli -c "select * from 'part.csv'"
```
This results in this error:
```
andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ datafusion-cli -c "select * from 'part.csv'"
DataFusion CLI v46.0.1
Error: Arrow error: Parser error: Error while parsing value p_partkey for column 0 at line 12057
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may failed to read csv files [datafusion]
Posted by "niebayes (via GitHub)" <gi...@apache.org>.
niebayes commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2758128963
I have tested some scenarios:
1. ok: arrow::csv::Reader read csv files generated by tpchgen-cli
2. ok: arrow::csv::Reader read other csv files.
3. ok: datafusion-cli read other csv files.
4. err: datafusion-cli read csv files generated by tpchgen-cli
Some might-not-be-correct discovery:
- The value that makes parsing fail must be the first value of a certain row
- The data type of the value must be integer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "niebayes (via GitHub)" <gi...@apache.org>.
niebayes commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2760761409
The line number in the error message is the row index of a certain record batch, not the line number in the csv file. I have filed an issue to arrow-rs for making this error message more user-friendly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2761888196
Turns out this is a bug in the generator -- https://github.com/clflushopt/tpchgen-rs/issues/73#issuecomment-2761885245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] datafusion-cli may fail to read csv files [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky commented on issue #15456:
URL: https://github.com/apache/datafusion/issues/15456#issuecomment-2760736404
```
grep -n "p_partkey" part.csv
```
why there are two head rows
```
1:p_partkey,p_name,p_mfgr,p_brand,p_type,p_size,p_container,p_retailprice,p_comment
100002:p_partkey,p_name,p_mfgr,p_brand,p_type,p_size,p_container,p_retailprice,p_comment
```
BTW it seems that line number in error message is not correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org