You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/29 16:16:26 UTC
[GitHub] [arrow-rs] XinyuZeng opened a new issue, #3408: parquet-fromcsv with writer version v2 does not stop
XinyuZeng opened a new issue, #3408:
URL: https://github.com/apache/arrow-rs/issues/3408
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
When using the executable parquet-fromcsv to convert csv into parquet and set writer version to 2, the program won't stop. The result Parquet file's size grows indefinitely until filling up the disk.
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
I've run with two different schema and csv files. One is from TPCH lineitem table SF10. The command is:
`parquet-fromcsv -s test_schema.txt -i core_test.csv -o core_test.parquet -w 2`
Without `-w 2` works fine. I can provide the data if an arbitrary schema and csv file is not reproducible.
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #3408: parquet-fromcsv with writer version v2 does not stop
Posted by GitBox <gi...@apache.org>.
tustvold closed issue #3408: parquet-fromcsv with writer version v2 does not stop
URL: https://github.com/apache/arrow-rs/issues/3408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] askoa commented on issue #3408: parquet-fromcsv with writer version v2 does not stop
Posted by GitBox <gi...@apache.org>.
askoa commented on issue #3408:
URL: https://github.com/apache/arrow-rs/issues/3408#issuecomment-1369967229
I'll pick this up. The issue is with the column `l_comment`. This column exhausts `dict_encoder` quickly and starts using `FallbackEncoder`.
In `V1` the `FallbackEncoderImpl::PLAIN` encoder is used where the buffer is cleared in `flush_data_page`
https://github.com/apache/arrow-rs/blob/08a976f217610cc8b1dea3375fe577027460ecc3/parquet/src/arrow/arrow_writer/byte_array.rs#L282-L284
In `V2` the `FallbackEncoderImpl::Delta` encoder is used where the buffer is not cleared in `flush_data_page`
https://github.com/apache/arrow-rs/blob/08a976f217610cc8b1dea3375fe577027460ecc3/parquet/src/arrow/arrow_writer/byte_array.rs#L307
The effect of this is ever growing buffer size. This ever growing buffer is written to output every mini batch (1000 rows). Hence the program's output consumes all the disk space. The fix would be to use `std::mem::take(buffer)` in `FallbackEncoderImpl::Delta`.
I see the same issue in `FallbackEncoderImpl::DeltaLength` too. I am assuming this should be fixed too. However I am not sure where this is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #3408: parquet-fromcsv with writer version v2 does not stop
Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #3408:
URL: https://github.com/apache/arrow-rs/issues/3408#issuecomment-1369978628
> FallbackEncoderImpl::DeltaLength
This will be used if the user has manually specified an encoding to use for the column, in addition to the default dictionary encoding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #3408: parquet-fromcsv with writer version v2 does not stop
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3408:
URL: https://github.com/apache/arrow-rs/issues/3408#issuecomment-1369975881
cc @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #3408: parquet-fromcsv with writer version v2 does not stop
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3408:
URL: https://github.com/apache/arrow-rs/issues/3408#issuecomment-1368211945
Thank you for the report @XinyuZeng !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org