You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by kumarvishal09 <gi...@git.apache.org> on 2019/01/11 09:50:05 UTC
[GitHub] carbondata pull request #3068: [HOTFIX] Fixed NPE during query with Local Di...
GitHub user kumarvishal09 opened a pull request:
https://github.com/apache/carbondata/pull/3068
[HOTFIX] Fixed NPE during query with Local Dictionary
**Problem:**
Query is failing with NPE when some blocklet encoded with local dictionary and some without local dictionary.
**Root Cause:**
This is coming because in carbonvectorProxy setDictionary with null it is not setting the dictionary to null because of this it is treated like a local dictionary column but column is not encoded with dictionary.
**Solution:**
Set dictionary to null
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kumarvishal09/incubator-carbondata master_102019
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/3068.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3068
----
commit 7fdc042bcf2c7bc05135a809cab8ccd45dfbc01c
Author: kumarvishal09 <ku...@...>
Date: 2019-01-11T09:44:53Z
fixed NPE in LocalDictionary Query
----
---
[GitHub] carbondata issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionar...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3068
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2269/
---
[GitHub] carbondata issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionar...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3068
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2488/
---
[GitHub] carbondata issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionar...
Posted by qiuchenjian <gi...@git.apache.org>.
Github user qiuchenjian commented on the issue:
https://github.com/apache/carbondata/pull/3068
why does one segment have some blocklet encoded with local dictionary and some without local dictionary ?
---
[GitHub] carbondata issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionar...
Posted by qiuchenjian <gi...@git.apache.org>.
Github user qiuchenjian commented on the issue:
https://github.com/apache/carbondata/pull/3068
@ravipesala I remember if the local dictionary value count reach threshold, it will go back to original value， right？
Does some nodes's local dictionary reach the threshold, bug other's not, so this scene appeared ？（some shards have local dictionary, some shards don't have local dictionary）
---
[GitHub] carbondata issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionar...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3068
> why does one segment have some blocklet encoded with local dictionary and some without local dictionary ?
It is because carbon generates a dictionary based on the column value count threshold, so once it reaches that threshold it stops generating the dictionary. There are scenarios where some blocks/blocklets are with in threshold and some are not, thats why some blocks has local dictionary and some don't have
---
[GitHub] carbondata issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionar...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/3068
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10527/
---
[GitHub] carbondata issue #3068: [HOTFIX] Fixed NPE during query with Local Dictionar...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/3068
LGTM
---