You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alex (via GitHub)" <gi...@apache.org> on 2024/10/10 14:28:59 UTC
[I] CSVReader behavior with dataset that has duplicate column headers is confusing [datafusion]
alex opened a new issue, #12852:
URL: https://github.com/apache/datafusion/issues/12852
### Describe the bug
When working with a CSV file that has duplicate column headers (e.g., https://github.com/openssl/openssl/blob/master/test/recipes/80-test_cmp_http_data/test_connection.csv) the behavior is confusing.
Specifically, any query against a table backed by that file produces the error:
> Arrow error: Csv error: incorrect number of fields for line 1, expected 14 got more than 14
However, all rows in that CSV have 20 fields.
Based on looking at the results of a `limit 0` query, I can see that the schema is effectively dropping all duplicate columns from the expected schema, and therefore the following rows do not have the expected number of cells.
### To Reproduce
Run queries against the linked CSV file.
### Expected behavior
I believe expected behavior would be to either a) automatically rename those columns (adding `_{n}` perhaps), or b) provide a clear error that schemas with duplicate column names are not supported.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CSVReader behavior with dataset that has duplicate column headers is confusing [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #12852:
URL: https://github.com/apache/datafusion/issues/12852#issuecomment-2469474930
Having thought about it some more, I think the use of `Schema::try_merge` is actually incorrect for CSV files, because the CSV reading process assumes that the fields in the `Schema` are in the same order as they appear in the file.
So, if two CSVs are read in with the same columns but out of order, this will cause data to appear in the wrong columns. This might error out if there is a mismatch in types, but could also just silently return bogus data.
My intuition is that the code needs to be changed to merge CSV schemas based on field index not field name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CSVReader behavior with dataset that has duplicate column headers is confusing [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #12852:
URL: https://github.com/apache/datafusion/issues/12852#issuecomment-2469435302
This looks to be caused by the call to `Schema::try_merge` at `datasource/file_format/csv.rs:328`. That function merges schemas together by joining on field name, so a side effect is the merging of fields within a single schema that happen to have the same name.
I think an important question is whether it makes sense for a schema to have multiple fields with the same name in the first place, or whether it should be disallowed. While maybe it's defensible given fields can be accessed by index, it does make it pretty hard to sensibly merge schemas based on field name as `Schema::try_merge` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CSVReader behavior with dataset that has duplicate column headers is confusing [datafusion]
Posted by "alex (via GitHub)" <gi...@apache.org>.
alex commented on issue #12852:
URL: https://github.com/apache/datafusion/issues/12852#issuecomment-2469438076
From the perspective of a user, I think the ideal default behavior would be register a clean error in that case.
But I'd also like the ability to somehow map the duplicate field names to something (e.g., `{field}_{i}`). If it just did that by default, that'd also be fine.
But sadly, CSVs with duplicate columns names are kind of common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CSVReader behavior with dataset that has duplicate column headers is confusing [datafusion]
Posted by "jpsrobertson (via GitHub)" <gi...@apache.org>.
jpsrobertson commented on issue #12852:
URL: https://github.com/apache/datafusion/issues/12852#issuecomment-2469427313
bumping as i'm also experiencing this problem!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] CSVReader behavior with dataset that has duplicate column headers is confusing [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #12852: CSVReader behavior with dataset that has duplicate column headers is confusing
URL: https://github.com/apache/datafusion/issues/12852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org