You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/08/11 12:08:24 UTC
[I] Change default value of `datafusion.catalog.has_header` to `true` [datafusion]
alamb opened a new issue, #11936:
URL: https://github.com/apache/datafusion/issues/11936
### Is your feature request related to a problem or challenge?
As @jgranduel notes in https://github.com/apache/datafusion/issues/11848
I think most people would expect when reading this data as a csv file, we would get two columns `a` and `b`
```csv
a,b
1,2
```
However, datafusion currently treats the CSV file as "without header" by default resulting in
```
> datafusion-cli.exe
DataFusion CLI v40.0.0
> select * from './data_dir/';
+----------+----------+
| column_1 | column_2 |
+----------+----------+
| a | b |
| 1 | 2 |
| a | b |
| 3 | 4 |
+----------+----------+
4 row(s) fetched.
```
### Describe the solution you'd like
I propose removing this "rough edge" and changing the datafusion default behavior to be what I think most people would expect
```sql
+---+---+
| a | b |
+---+---+
| 1 | 2 |
+---+---+
```
### Describe alternatives you've considered
I think we should change the default vale of `datafusion.catalog.has_header` from `false` to `true`
https://github.com/apache/datafusion/blob/79fa6f9098be9a6e5b269cd3642694765b230ff1/datafusion/common/src/config.rs#L184-L186
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Change default value of `datafusion.catalog.has_header` to `true` [datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on issue #11936:
URL: https://github.com/apache/datafusion/issues/11936#issuecomment-2282803786
I think it'll be fine because there is already CSVReadOptions with default `has_header: true` ([link](https://github.com/apache/datafusion/blob/fd237f8705b18fa089fdfb8dd5b04655ccb4d691/datafusion/core/src/datasource/file_format/options.rs#L100)), which produces some inconsistency in default handling csv.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Change default value of `datafusion.catalog.has_header` to `true` [datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa closed issue #11936: Change default value of `datafusion.catalog.has_header` to `true`
URL: https://github.com/apache/datafusion/issues/11936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Change default value of `datafusion.catalog.has_header` to `true` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11936:
URL: https://github.com/apache/datafusion/issues/11936#issuecomment-2284459200
I think default to "no headers" is unlikely to have been a conscious choice (I suspect it was a historical accident)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org