You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "lifulong (via GitHub)" <gi...@apache.org> on 2026/05/29 05:40:36 UTC
[I] All data is null for one column in one row group for parquet, arrow will encode with dictionary, while parquet-java use encode plain [arrow]
lifulong opened a new issue, #50062:
URL: https://github.com/apache/arrow/issues/50062
### Describe the bug, including details regarding any error messages, version, and platform.
https://github.com/apache/parquet-java/blob/b8f33308534e990003e48a2e66036ccc83fd5db4/parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java#L81
https://github.com/apache/parquet-java/blob/b8f33308534e990003e48a2e66036ccc83fd5db4/parquet-column/src/main/java/org/apache/parquet/column/values/dictionary/DictionaryValuesWriter.java#L129
parquet-java will check raw data size, if raw data size is zero, it will fall back use plain encode. ((encodedSize + dictionaryByteSize) < rawSize)
while arrow still use dictionary encode.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] All data is null for one column in one row group for parquet, arrow will encode with dictionary, while parquet-java use encode plain [arrow]
Posted by "lifulong (via GitHub)" <gi...@apache.org>.
lifulong commented on issue #50062:
URL: https://github.com/apache/arrow/issues/50062#issuecomment-4623334470
> I agree that currently it looks weird but still spec-conforming. Do you want to propose a fix?
@wgtmac Thanks for your reply. If you think aligning with the implementation of parquet-java is preferable, I can attempt to submit a fix when I have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] All data is null for one column in one row group for parquet, arrow will encode with dictionary, while parquet-java use encode plain [arrow]
Posted by "lifulong (via GitHub)" <gi...@apache.org>.
lifulong commented on issue #50062:
URL: https://github.com/apache/arrow/issues/50062#issuecomment-4592068615
@wgtmac Can you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] All data is null for one column in one row group for parquet, arrow will encode with dictionary, while parquet-java use encode plain [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #50062:
URL: https://github.com/apache/arrow/issues/50062#issuecomment-4623508850
I just think that an empty dict page is meaningless :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] All data is null for one column in one row group for parquet, arrow will encode with dictionary, while parquet-java use encode plain [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #50062:
URL: https://github.com/apache/arrow/issues/50062#issuecomment-4682566993
This is more of an optimization than a conformance fix, but I agree it's probably worth doing.
We may even use a threshold of non-null values, but I don't know what the value should be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] All data is null for one column in one row group for parquet, arrow will encode with dictionary, while parquet-java use encode plain [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #50062:
URL: https://github.com/apache/arrow/issues/50062#issuecomment-4598793710
I agree that currently it looks weird but still spec-conforming. Do you want to propose a fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org