You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "neilechao (via GitHub)" <gi...@apache.org> on 2025/03/25 23:50:17 UTC

[I] [C++][Parquet] Add variant support [arrow]

neilechao opened a new issue, #45937:
URL: https://github.com/apache/arrow/issues/45937

   ### Describe the enhancement requested
   
   Variant encoding spec: https://github.com/apache/parquet-format/blob/d784f11f4485e64fdeaa614e0bde125f5132093d/VariantEncoding.md
   
   Variant shredding spec: https://github.com/apache/parquet-format/blob/d784f11f4485e64fdeaa614e0bde125f5132093d/VariantShredding.md
   
   Loose order of steps:
   1. Logical type definition
   2. Decoding once parquet_testing includes variant examples
   3. Encoding
   4. Shredding variant representation
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4795158809

   @qzyu999 No worries. Thanks for taking time and effort to work on this! I would really appreciate it if you could help review related PRs shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-2785312680

   Issue resolved by pull request 45375
   https://github.com/apache/arrow/pull/45375


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4692573242

   > A quick heads up that [@HuaHuaY](https://github.com/HuaHuaY) is working on this for quite a while. We are reviewing the PR internally and will put it out once ready.
   
   Thanks for update 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-2791314362

   @neilechao Agreed. The merge script automatically closed this issue because that PR links to it.
   
   @zeroshade There was a discussion about this: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352. We might need more implementation detail to decide what detail to propose. (cc @alamb)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "neilechao (via GitHub)" <gi...@apache.org>.
neilechao commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-2790336303

   @wgtmac - we should probably keep this open until all of the sub-issues are done. Should I can create a sub issue for the logical type itself and attach the PR-45375 to that new sub issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4795750750

   > [@qzyu999](https://github.com/qzyu999) No worries. Thanks for taking time and effort to work on this! I would really appreciate it if you could help review related PRs shortly.
   
   You're welcome and I'll make sure to do so.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-2791308992

   @wgtmac we should probably consider adding variant as a canonical extension type?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-5112253045

   You're right. Feel free to take a review :) @yarrow-sparrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "Tishj (via GitHub)" <gi...@apache.org>.
Tishj commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-3067256021

   > It would be great for folks working on this here and in other languages if some example files could be added to https://github.com/apache/parquet-testing 🙏🏻
   
   There are parquet files with Variants in them here: https://github.com/duckdb/duckdb/pull/18224
   As the description says, these are generated by apache/iceberg
   
   Only notable difference is that they are rewritten to add the Variant logical type to the top level struct of the Variant column


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4676483230

   A quick heads up that @HuaHuaY is working on this for quite a while. We are reviewing the PR internally and will put it out once ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac closed issue #45937: [C++][Parquet] Add variant support
URL: https://github.com/apache/arrow/issues/45937


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "yarrow-sparrow (via GitHub)" <gi...@apache.org>.
yarrow-sparrow commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-5109463064

   Hi @wgtmac, can you confirm if https://github.com/apache/arrow/pull/50252 is the main PR for Variant support and https://github.com/apache/arrow/pull/50121 is abandoned for now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-2781404046

   It would be great for folks working on this here and in other languages if some example files could be added to https://github.com/apache/parquet-testing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-2797915137

   >  @wgtmac we should probably consider adding variant as a canonical extension type?
   
   
   Yeah, I agree variant would be a good candidate for canonical extension type. We are now in the active process of adding Variant support in Rust / arrow-rs (and will definitely be using this PR as a guide as we implement it). Unless we have good reason to deviate, I intend to encourage following the same extension type convention as in this PR
   
   That would make it even more compelling to discuss adding as a canonical extension type I would think.
   
   Maybe it is something to bring up on the mailing list, to seed the dicussion
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4411814080

   I will have a try since this has been stale for a long time. @soumilshah1995 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4366286924

   any ETA on this ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "sungwy (via GitHub)" <gi...@apache.org>.
sungwy commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4790637156

   Hi @wgtmac - just a heads up, it looks like @qzyu999 is also working on this issue and has raised a few PRs. Could we review these and/or consolidate @HuaHuaY 's ongoing efforts so we don't duplicate the work?
   
   - https://github.com/apache/arrow/pull/50232
   - https://github.com/apache/arrow/pull/50121
   - https://github.com/apache/arrow/pull/50122


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [C++][Parquet] Add variant support [arrow]

Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on issue #45937:
URL: https://github.com/apache/arrow/issues/45937#issuecomment-4791269543

   > Hi [@wgtmac](https://github.com/wgtmac) - just a heads up, it looks like [@qzyu999](https://github.com/qzyu999) is also working on this issue and has raised a few PRs. Could we review these and/or consolidate [@HuaHuaY](https://github.com/HuaHuaY) 's ongoing efforts so we don't duplicate the work?
   > 
   > * [GH-45948: [C++][Parquet] Variant shredding #50232](https://github.com/apache/arrow/pull/50232)
   > * [GH-45946: [C++][Parquet] Variant decoding #50121](https://github.com/apache/arrow/pull/50121)
   > * [GH-45947 : [C++][Parquet] Variant encoding #50122](https://github.com/apache/arrow/pull/50122)
   
   Hi @sungwy, I spoke to @wgtmac about this in Slack, he is aware. Apologies I should've pinged @wgtmac before starting, I had seen he was assigned, but after a month I thought it was abandoned. I will make sure to avoid this mistake in the future so that we don't waste developer time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org