You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2024/04/17 10:49:58 UTC
[I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
tustvold opened a new issue, #5657:
URL: https://github.com/apache/arrow-rs/issues/5657
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
The parquet reader automatically uses an embedded arrow schema to hint type inference for decode. In particular if the hinted type is compatible with the underlying parquet type, it performs a cast.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
In situations where the writer was not an arrow writer this schema is not available, and therefore the arrow types are inferred from the parquet schema. This is not always desirable:
* https://github.com/apache/arrow-rs/issues/5655
* https://github.com/apache/arrow-rs/pull/5605#issuecomment-2060960080
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2063038264
> > I think my expectation would be for you to provide the `SchemaRef` for the entire file
>
Basically agree with your idea
In the datafusion, the `ParquetExec` of `FileScanConfig` contains the schema for the parquet file, but I think the provided `SchemaRef` should be the optional for the parquet reader when infer the data type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2063031558
> I think my expectation would be for you to provide the `SchemaRef` for the entire file
In the datafusion, the `ParquetExec` of `FileScanConfig` contains the schema for the parquet file, but I think the provided `SchemaRef` should be the optional for the parquet reader when infer the data type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2065387747
Something like
```rust
let file = File::open("data.parquet").unwrap();
let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
// specify the arrow schema to read from this parquet file
// will error if the types in the parquet file can not be converted
// into the specific types.
// Will ignore any embedded metadata about types when written
.schema(schema)
println!("Converted arrow schema is: {}", builder.schema());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2062331038
Let me try the remaining part if it is ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2063030971
> ```rust
> ParquetRecordBatchReaderBuilder
> ```
In the datafusion, the `ParquetExec` of `FileScanConfig` contains the schema for the parquet file, but I think the provided `SchemaRef` should be the optional for the parquet reader when infer the data type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2065682971
> Something like
>
> ```rust
> let file = File::open("data.parquet").unwrap();
>
> let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
> // specify the arrow schema to read from this parquet file
> // will error if the types in the parquet file can not be converted
> // into the specific types.
> // Will ignore any embedded metadata about types when written
> .schema(schema)
>
> println!("Converted arrow schema is: {}", builder.schema());
> ```
Do we need to add some checker in the function of the `schema` which is used to compare the schema with schema inferred from the parquet file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2204013362
`label_issue.py` automatically added labels {'arrow'} from #5605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2068514681
> The inference logic is already setup to use the arrow schema as a hint as opposed to authoritative , if you give it something invalid it will just ignore it
thanks, got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2061863599
Here is one potential API
```rust
let file = File::open("data.parquet").unwrap();
let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
// specify column "time" should be UTC
// will error if this type can not be read from parquet
.with_column_type("time", DateTime::Timestamp(Nanoseconds, Some("UTC"))
println!("Converted arrow schema is: {}", builder.schema());
```
I am not quite sure how to handle identifying nested types with a single column name
Like if the parquet file has
```json
{
"my_object": {
"time": "12-01-02"
}
}
```
maybe we would refer to the `time` field like `"my_object.time"`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2065839366
The inference logic is already setup to use the arrow schema as a hint as opposed to authoritative , if you give it something invalid it will just ignore it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2062024150
I think my expectation would be for you to provide the `SchemaRef` for the entire file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide Arrow Schema Hint to Parquet Reader [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5657:
URL: https://github.com/apache/arrow-rs/issues/5657#issuecomment-2204013443
`label_issue.py` automatically added labels {'parquet'} from #5939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org