[jira] [Updated] (ARROW-1407) Dictionaries can only hold a maximum of 4096 indices

["Rok Mihevc (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rok Mihevc updated ARROW-1407:
------------------------------
    External issue URL: https://github.com/apache/arrow/issues/17433

> Dictionaries can only hold a maximum of 4096 indices
> ----------------------------------------------------
>
>                 Key: ARROW-1407
>                 URL: https://issues.apache.org/jira/browse/ARROW-1407
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Java
>    Affects Versions: 0.6.0
>            Reporter: Shayan Monshizadeh
>            Assignee: Li Jin
>            Priority: Minor
>             Fix For: 0.7.0
>
>         Attachments: Screen Shot 2017-08-22 at 7.14.07 PM.png
>
>
> Dictionaries seem to only be able to hold 4096 indices, meaning only vectors with 4096 values or less can be turned into dictionaries. The image attached is a stack trace of what happens when try to encode a dictionary with a vector containing 4097 strings, and a dictionary containing two distinct values. 
> Basically the error can be traced to line 95 of DictionaryEncoder.java (`setter.invoke(mutator, i, encoded);`). It seems that the indices array which hold the encoded values is allocated on line 84 as `indices.allocateNew()` and it seems that `allocateNew()` only allocates 4096 bytes of data initially. The code runs if there are 4096 rows of data or less. Anymore and the same error is given.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)