You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2018/01/10 15:49:51 UTC
[GitHub] carbondata pull request #1789: [WIP] Fix avoid reading of all block informat...
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1789
[WIP] Fix avoid reading of all block information in driver for old stores.
Problem:
For old stores prior to 1.2 version there is no blocklet information stored in carbonindex file. So the new code needs to read all carbondata files footers inside the driver to get the blocklet information. That makes the first time queries become slower. As observed count(*) query was taking 2 swconds on old version and after upgrade it takes very long time.
Solution:
If there is no information blocklet available in carbonindex file then don't read carbondata files footer in driver side. Instead read carbondata files in executor to get the blocklet information.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed?
- [X] Any backward compatibility impacted?
- [X] Document update required?
- [X] Testing done
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata datamap-pld-store
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1789.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1789
----
commit f0aaf4d8d0e4761227bc1adff29d33798c88bd12
Author: ravipesala <ra...@...>
Date: 2018-01-10T15:35:48Z
Fix avoid reading of all block information in driver for old stores.
----
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1789
retest sdv please
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1465/
---
[GitHub] carbondata issue #1789: [WIP] Fix avoid reading of all block information in ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1451/
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1503/
---
[GitHub] carbondata issue #1789: [WIP] Fix avoid reading of all block information in ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1789
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2821/
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2735/
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1789
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2834/
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1789
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2831/
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1789
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2862/
---
[GitHub] carbondata issue #1789: [WIP] Fix avoid reading of all block information in ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1463/
---
[GitHub] carbondata pull request #1789: [CARBONDATA-2020] Fix avoid reading of all bl...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1789
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1789
LGTM
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2698/
---
[GitHub] carbondata issue #1789: [WIP] Fix avoid reading of all block information in ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2696/
---
[GitHub] carbondata issue #1789: [WIP] Fix avoid reading of all block information in ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1789
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2684/
---
[GitHub] carbondata issue #1789: [CARBONDATA-2020] Fix avoid reading of all block inf...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1789
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2842/
---