You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "cj-zhukov (via GitHub)" <gi...@apache.org> on 2025/01/16 05:53:30 UTC
[I] Add a hint about expected extension in error message in register_csv, register_parquet, register_json [datafusion]
cj-zhukov opened a new issue, #14144:
URL: https://github.com/apache/datafusion/issues/14144
### Describe the bug
When attempting to register an existing file with different format using the register_csv, register_json, register_parquet, instead of receiving an error, the operation silently returns an empty dataframe.
### To Reproduce
```rust
let file = "foo.parquet"; // foo.parquet does exist
let ctx = SessionContext::new();
ctx.register_csv("foo", file, CsvReadOptions::default()).await?; // trying to register parquet file with register_csv
// ctx.register_json("foo", file, NdJsonReadOptions::default()).await?; // or same using register_json
let res = ctx.sql("select * from foo").await?;
res.show().await?;
// Observe that no error is raised, but an empty DataFrame is returned.
// ++
// ++
```
### Expected behavior
Introduce a validation step in register_csv, register_json, register_parquet to check the file extension or validate the file's format. If the file does not match the expected format, return a clear error message:
```rust
Error: Execution("File path 'path/to/file' does not match the expected extension '.extension'")
```
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add a hint about expected extension in error message in register_csv, register_parquet, register_json [datafusion]
Posted by "cj-zhukov (via GitHub)" <gi...@apache.org>.
cj-zhukov commented on issue #14144:
URL: https://github.com/apache/datafusion/issues/14144#issuecomment-2671430609
@devhprl hi, thanks for the comment! When I was working on this, I took in consideration how read_xyz methods work. And they check if the file extension matches the expected extension. So it was one of the reason why I chose this approach for register_xyz methods as well. Actually, I don't see any good reason not to relay on file extension in this regard. For example, if I have some foo.parquet file, I expect to read (or register) it into dataframe with read_parquet method (but not read_csv).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add a hint about expected extension in error message in register_csv, register_parquet, register_json [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14144: Add a hint about expected extension in error message in register_csv, register_parquet, register_json
URL: https://github.com/apache/datafusion/issues/14144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add a hint about expected extension in error message in register_csv, register_parquet, register_json [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14144:
URL: https://github.com/apache/datafusion/issues/14144#issuecomment-2676880600
> [@cj-zhukov](https://github.com/cj-zhukov) [@alamb](https://github.com/alamb) hi, I've just hit this one. judging file content by an extension doesn't feel fully right to me (as you can put there basically anything). yes if there is an content mismatch (expected vs actual) then there will be an undesired result (although I would expect some parsing error or something). nevertheless enforcing untrustworthy extensions doesn't seem to bring much value in that regard. in our case we intentionally omit the extensions completely in the storage and have other ways how the content info gets propagated (to the point where register_xyz gets picked). thanks for the great lib
I agree that ensuring there is some way to override the format used to interpret files (e.g. reading a `.dat` file as parquet) is a good feature. @devhprl if there is something preventing you from doing that, please do file an issue with a reproducer and we can help make sure the APIs support your usecase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add a hint about expected extension in error message in register_csv, register_parquet, register_json [datafusion]
Posted by "devhprl (via GitHub)" <gi...@apache.org>.
devhprl commented on issue #14144:
URL: https://github.com/apache/datafusion/issues/14144#issuecomment-2681308324
thanks guys. will dig into that a bit. in the meantime I am slightly adjusting our case to either fit in to the current validations or shape up the need properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add a hint about expected extension in error message in register_csv, register_parquet, register_json [datafusion]
Posted by "devhprl (via GitHub)" <gi...@apache.org>.
devhprl commented on issue #14144:
URL: https://github.com/apache/datafusion/issues/14144#issuecomment-2669590997
@cj-zhukov @alamb hi, I've just hit this one. judging file content by an extension doesn't feel fully right to me (as you can put there basically anything). yes if there is an content mismatch (expected vs actual) then there will be an undesired result (although I would expect some parsing error or something). nevertheless enforcing untrustworthy extensions doesn't seem to bring much value in that regard. in our case we intentionally omit the extensions completely in the storage and have other ways how the content info gets propagated (to the point where register_xyz gets picked). thanks for the great lib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add a hint about expected extension in error message in register_csv, register_parquet, register_json [datafusion]
Posted by "cj-zhukov (via GitHub)" <gi...@apache.org>.
cj-zhukov commented on issue #14144:
URL: https://github.com/apache/datafusion/issues/14144#issuecomment-2594596672
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org