You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/04/09 18:04:19 UTC
[PR] feat: support reading and writing`StringView` and `BinaryView` in parquet (part 1) [arrow-rs]
alamb opened a new pull request, #5618:
URL: https://github.com/apache/arrow-rs/pull/5618
# Which issue does this PR close?
First part of https://github.com/apache/arrow-rs/issues/5530
# Rationale for this change
This is the first 3 commits from https://github.com/apache/arrow-rs/pull/5557 by @ariesdevil for initial support of reading `StringViewArray` and `BinaryViewArray` from parquet
The performance is non ideal (it copies string data several times) but it does include benchmarks and tests.
Thus I would like to merge this in and then continue iterating on the design in https://github.com/apache/arrow-rs/pull/5557 to keep the PRs smaller and more manageable
# What changes are included in this PR?
1. Basic support for reading/writing`StringViewArray` and `BinaryViewArray` from parquet
2. Tests for same
3. Benchmarks
# Are there any user-facing changes?
The arrow reader/writer can now read /write `StringViewArray` and `BinaryViewArray`s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support reading and writing`StringView` and `BinaryView` in parquet (part 1) [arrow-rs]
Posted by "ariesdevil (via GitHub)" <gi...@apache.org>.
ariesdevil commented on PR #5618:
URL: https://github.com/apache/arrow-rs/pull/5618#issuecomment-2045839896
> Just a minor question: during writing parquet, arrow will serialize self-schema. Would BinaryView/StringView keep it's schema, or using Binary/String here?
View type will keep its schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support reading and writing`StringView` and `BinaryView` in parquet (part 1) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5618:
URL: https://github.com/apache/arrow-rs/pull/5618#issuecomment-2045923645
Thanks again @ariesdevil and @mapleFU
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support reading and writing`StringView` and `BinaryView` in parquet (part 1) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5618:
URL: https://github.com/apache/arrow-rs/pull/5618#issuecomment-2045921967
Merging this one in as it doesn't expose any public PRs and has been reviewed as a prior PR https://github.com/apache/arrow-rs/pull/5557#pullrequestreview-1974957324
Let's continue the conversation on https://github.com/apache/arrow-rs/pull/5557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support reading and writing`StringView` and `BinaryView` in parquet (part 1) [arrow-rs]
Posted by "ariesdevil (via GitHub)" <gi...@apache.org>.
ariesdevil commented on PR #5618:
URL: https://github.com/apache/arrow-rs/pull/5618#issuecomment-2045838145
Thanks for splitting this PR into small parts, looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support reading and writing`StringView` and `BinaryView` in parquet (part 1) [arrow-rs]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #5618:
URL: https://github.com/apache/arrow-rs/pull/5618#issuecomment-2045808667
Just a minor question: during writing parquet, arrow will serialize self-schema. Would BinaryView/StringView keep it's schema, or using Binary/String here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support reading and writing`StringView` and `BinaryView` in parquet (part 1) [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #5618:
URL: https://github.com/apache/arrow-rs/pull/5618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org