[jira] [Commented] (PARQUET-743) DictionaryFilters can re-use StreamBytesInput when compressed

["Patrick Woody (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15555870#comment-15555870 ] 

Patrick Woody commented on PARQUET-743:
---------------------------------------

Copied from mailing list
{code}
Hey all,

Running a parquet-mr build off of master and I'm seeing some interesting behavior when using a DictionaryFilter to prune row groups. Basically, if I have an And or Or filter the DictionaryPage object gets re-used. This seems to be a problem for StreamBytesInput because the stream gets exhausted after the first toByteArray call. My current workaround is to synchronize and just re-use the byte array after the first read, but I'd be curious as to what people think the best approach to solving this is and if we should be reusing the BytesInput at all.

Best,
Patrick 


---

Looking a bit more - it looks like this is because decompression converts to a StreamBytesInput automatically. The current tests run with the uncompressed codec, so it doesn't hit this issue. I've put up a commit here that demonstrates the issue and my current workaround: https://github.com/palantir/parquet-mr/pull/10/commits/70cc00cba5c294d4c860bd4cd2c48c2d083a5809.

Thanks,
{code}

> DictionaryFilters can re-use StreamBytesInput when compressed
> -------------------------------------------------------------
>
>                 Key: PARQUET-743
>                 URL: https://issues.apache.org/jira/browse/PARQUET-743
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-mr
>    Affects Versions: 1.9.0
>            Reporter: Patrick Woody
>
> When using an And or Or DictionaryFilter, we re-use the BytesInput across reads. This is problematic when compressed because compressed BytesInputs get converted over to StreamBytesInputs which can only be used once.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)