You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/11/15 18:14:59 UTC
[I] Port tests in `parquet_schema.rs` to sqllogictest [arrow-datafusion]
alamb opened a new issue, #8207:
URL: https://github.com/apache/arrow-datafusion/issues/8207
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port tests in `parquet_schema.rs` to sqllogictest [arrow-datafusion]
Posted by "matthewmturner (via GitHub)" <gi...@apache.org>.
matthewmturner commented on issue #8207:
URL: https://github.com/apache/arrow-datafusion/issues/8207#issuecomment-1868314796
@alamb I could be wrong (I'm still getting used to the code base again) - but it looks like porting these tests may require adding the ability to set whether to read metadata or not from SQL (i.e. add `WITH METADATA` similar to `WITH HEADER ROW`). From what ive seen choosing whether to read metadata is only possible by explicitly setting `ParquetReadOptions` with `ctx.read_parquet`. If you agree I can create an issue for that and work on it before coming back to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port tests in `parquet_schema.rs` to sqllogictest [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8207:
URL: https://github.com/apache/arrow-datafusion/issues/8207#issuecomment-1868512835
I think you are correct that this test is not really a SQL level test (as we can't control the metadata reading) @matthewmturner
Porting this to SQL doesn't make sense. I think a better thing would be to move the test into the "parquet" tests as it is really a test of the parquet reader. I proposed doing so in https://github.com/apache/arrow-datafusion/pull/8644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port tests in `parquet_schema.rs` to sqllogictest [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8207:
URL: https://github.com/apache/arrow-datafusion/issues/8207#issuecomment-1869492106
> Got it - would you just be able to expand on why it doesnt make sense to add the WITH METADATA as a an option from SQL?
In my mind it is because the usecase for preserving the metadata is when DataFusion is used as a library to build other systems (which may interpret the schema level metadata) and in which case the user is likely to be using lower level APIs than SQL.
That being said, if anyone has a use case for controlling schema metadata output at the SQL level it would make sense to add it to the SQL interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port tests in `parquet_schema.rs` to sqllogictest [arrow-datafusion]
Posted by "matthewmturner (via GitHub)" <gi...@apache.org>.
matthewmturner commented on issue #8207:
URL: https://github.com/apache/arrow-datafusion/issues/8207#issuecomment-1867763652
ill take this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port tests in `parquet_schema.rs` to sqllogictest [arrow-datafusion]
Posted by "matthewmturner (via GitHub)" <gi...@apache.org>.
matthewmturner commented on issue #8207:
URL: https://github.com/apache/arrow-datafusion/issues/8207#issuecomment-1868523795
Got it - would you just be able to expand on why it doesnt make sense to add the `WITH METADATA` as a an option from SQL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port tests in `parquet_schema.rs` to sqllogictest [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8207: Port tests in `parquet_schema.rs` to sqllogictest
URL: https://github.com/apache/arrow-datafusion/issues/8207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org