[jira] [Comment Edited] (PARQUET-36) FilteringPrimitiveConverter should support dictionaries

["MIchael Davies (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300176#comment-14300176 ] 

MIchael Davies edited comment on PARQUET-36 at 2/2/15 9:57 AM:
---------------------------------------------------------------

I'm happy to start looking at this. I can let you know if I get anywhere.  Is there a dev forum that can be used for discussion - or is JIRA best?

I think the simplest way to support this would be to add methods to ValueInspector and autogenerated subclasses - something like:
{code}
class ValueInspector {
        private boolean[] dictionaryResults;
        
        /** Populates array of boolean based on whether values pass filter */
        void setDictionary(Dictionary dictionary) {
            dictionaryResults = fillInDictionaryResults(dictionary);
        }
        
        /** Update based on element from dictionary */
        void updateFromDictionary(int dictionaryId) {
            setResult(dictionaryResults[dictionaryId]);
        }
        
        /** Return an array of boolean that match the predicate for each value in the dictionary */
        abstract protected boolean[] fillInDictionaryResults(Dictionary dictionary);
}
{code}

It would be nice if dictionary based predicate evaluation could be used to skip entire rows where appropriate, but this is more work



was (Author: michael davies):
I'm happy to start looking at this. I can let you know if I get anywhere.  Is there a dev forum that can be used for discussion - or is JIRA best?

> FilteringPrimitiveConverter should support dictionaries
> -------------------------------------------------------
>
>                 Key: PARQUET-36
>                 URL: https://issues.apache.org/jira/browse/PARQUET-36
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>            Reporter: Alex Levenson
>            Priority: Minor
>              Labels: filter2
>
> If the delegated PrimitiveConverter supports dictionaries, then FilteringPrimitiveConverter should too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)