You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "Jacky Li (Jira)" <ji...@apache.org> on 2019/12/28 03:02:00 UTC
[jira] [Resolved] (CARBONDATA-3519) Optimizations in write step to
avoid unnecessary memory blk allocation/free
[ https://issues.apache.org/jira/browse/CARBONDATA-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacky Li resolved CARBONDATA-3519.
----------------------------------
Fix Version/s: 2.0.0
Resolution: Fixed
> Optimizations in write step to avoid unnecessary memory blk allocation/free
> ---------------------------------------------------------------------------
>
> Key: CARBONDATA-3519
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3519
> Project: CarbonData
> Issue Type: Improvement
> Components: core
> Reporter: Venugopal Reddy K
> Priority: Minor
> Fix For: 2.0.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> +*Issue-1:*+
> {color:#0747a6}*Context:*{color}
> For a string column with local dictionary enabled, a column page of
> `{color:#de350b}UnsafeFixLengthColumnPage{color}` with datatype `{color:#de350b}{{DataTypes.BYTE_ARRAY}}{color}` is created for `{color:#de350b}{{encodedPage}}{color}` along with regular `{color:#de350b}{{actualPage}}{color}` of `{color:#de350b}{{UnsafeVarLengthColumnPage}}{color}`.
> We have `{color:#de350b}*{{capacity}}*{color}` field in the `{color:#de350b}{{UnsafeFixLengthColumnPage}}{color}`. And this field indicates the capacity of allocated
> `{color:#de350b}{{memoryBlock}}{color}` for the page. `{{{color:#de350b}ensureMemory{color}()}}` method gets called while adding rows to check if `{color:#de350b}{{totalLength + requestSize > capacity}}{color}` to allocate a new memoryBlock. If there is no room to add the next row, allocates a new block, copy the old context(prev rows) and free the old memoryBlock.
> {color:#0747a6} *Problem:*{color}
> While, `{color:#de350b}{{UnsafeFixLengthColumnPage}}{color}` with with datatype `{color:#de350b}{{DataTypes.BYTE_ARRAY}}{color}` is created for `{color:#de350b}{{encodedPage}}{color}`, we have not assigned the *`{color:#de350b}{{capacity}}{color}`* field with allocated memory block size. Hence, for each add row to tablePage, *ensureMemory() check always fails*, allocates a new column page memoryBlock, copy the old context(prev rows) and free the old memoryBlock. This *allocation of new memoryBlock and free of old memoryBlock happens for each row row addition* for the string columns with local dictionary.
>
> +*Issue-2:*+
> {color:#0747a6}*Context:*{color}
> In`{color:#de350b}VarLengthColumnPageBase{color}`, we have a `{color:#de350b}rowOffset{color}` column page of `{color:#de350b}UnsafeFixLengthColumnPage{color}` of datatype `{color:#de350b}INT{color}`
> to maintain the data offset to {color:#172b4d}each{color} row of variable length columns. This `{color:#de350b}rowOffset{color}` page allocates to be size of page.
> {color:#0747a6} *Problem:*{color}
> {color:#172b4d}If we have 10 rows in the page, we need 11 rows for its rowOffset page. Because we always keep 0 as offset to 1st row. So an additional row is required for rowOffset page[pasted code below to show the reference]. Otherwise, *ensureMemory() check always fails for the last row*(10th row in this case) of data and *allocates a new rowOffset page memoryBlock, copy the old context(prev rows) and free the old memoryBlock*. This *can happen for the string columns with local dictionary, direct dictionary columns, global disctionary columns*.{color}
>
> {code:java}
> public abstract class VarLengthColumnPageBase extends ColumnPage {
> ...
> @Override
> public void putBytes(int rowId, byte[] bytes) {
> ...
> if (rowId == 0) {
> rowOffset.putInt(0, 0); ==> offset to 1st row is 0.
> }
> rowOffset.putInt(rowId + 1, rowOffset.getInt(rowId) + bytes.length);
> putBytesAtRow(rowId, bytes);
> totalLength += bytes.length;
> }
> ...
> }
>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)