You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2023/05/23 10:37:11 UTC
[GitHub] [arrow-datafusion] tustvold opened a new issue, #6427: Support Type Coercion in Parquet Reader
tustvold opened a new issue, #6427:
URL: https://github.com/apache/arrow-datafusion/issues/6427
### Is your feature request related to a problem or challenge?
https://github.com/apache/arrow-datafusion/pull/6404 by @e1ijah1 added support for type coercion within SchemaAdapter, we should hook up ParquetExec to make use of this new logic
### Describe the solution you'd like
I would like to be able to read a set of parquet files with different schema and have them coerced to the same schema
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] e1ijah1 commented on issue #6427: Support Type Coercion in Parquet Reader
Posted by "e1ijah1 (via GitHub)" <gi...@apache.org>.
e1ijah1 commented on issue #6427:
URL: https://github.com/apache/arrow-datafusion/issues/6427#issuecomment-1559227643
I'm interested and can complete this new issue too, but I'm uncertain on how to integrate SchemaAdapter into ParquetExec for mixed schemas. Could you please provide me with a little guidance or a high-level overview of how this might be done? Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on issue #6427: Support Type Coercion in Parquet Reader
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6427:
URL: https://github.com/apache/arrow-datafusion/issues/6427#issuecomment-1559256355
I think a first step would be switching ParquetOpener::open from using map_projections to the new API. I'm not sure what work might be required to make predicate pushdown work correctly, but I think this would be a good first step
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] e1ijah1 commented on issue #6427: Support Type Coercion in Parquet Reader
Posted by "e1ijah1 (via GitHub)" <gi...@apache.org>.
e1ijah1 commented on issue #6427:
URL: https://github.com/apache/arrow-datafusion/issues/6427#issuecomment-1566442489
@tustvold
I have implemented this feature in my latest commit. Could you please review it when you are available? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold closed issue #6427: Support Type Coercion in Parquet Reader
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #6427: Support Type Coercion in Parquet Reader
URL: https://github.com/apache/arrow-datafusion/issues/6427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org