You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by kushalsaha <gi...@git.apache.org> on 2017/09/05 11:43:54 UTC
[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...
GitHub user kushalsaha opened a pull request:
https://github.com/apache/carbondata/pull/1324
[CARBONDATA-1291]:carbonData query performance improvement when numbe…
…r of carbon blocks are high
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[CARBONDATA-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- What manual testing you have done?
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
---
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kushalsaha/carbondata CARBONDATA-1291
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1324.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1324
----
commit 9b680cd01151c06c96e68be523c8c254e798b9fe
Author: kushalsaha <ku...@gmail.com>
Date: 2017-09-05T11:40:30Z
[CARBONDATA-1291]:carbonData query performance improvement when number of carbon blocks are high
----
---
[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1324#discussion_r137213780
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
@@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
for (DataRefNode dataRefNode : dataRefNodes) {
BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
+ String[] deleteDeltaFilePath = null;
if (isIUDTable) {
// In case IUD is not performed in this table avoid searching for
// invalidated blocks.
if (CarbonUtil
- .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
- invalidBlockVOForSegmentId, updateStatusManager)) {
+ .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
--- End diff --
`CarbonInputFormat.java` is not used now. please use `CarbonTableInputFormat`
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
Can one of the admins verify this patch?
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/787/
---
[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1324
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
retest this please
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
Can one of the admins verify this patch?
---
[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...
Posted by kushalsaha <gi...@git.apache.org>.
Github user kushalsaha commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1324#discussion_r138850252
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
@@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
for (DataRefNode dataRefNode : dataRefNodes) {
BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
+ String[] deleteDeltaFilePath = null;
if (isIUDTable) {
// In case IUD is not performed in this table avoid searching for
// invalidated blocks.
if (CarbonUtil
- .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
- invalidBlockVOForSegmentId, updateStatusManager)) {
+ .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
--- End diff --
Do i need to revert the changes done in CarbonInputFormat.java
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
Can one of the admins verify this patch?
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1324
Can one of the admins verify this patch?
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1324
Can one of the admins verify this patch?
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1324
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/144/
---
[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...
Posted by kushalsaha <gi...@git.apache.org>.
Github user kushalsaha commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1324#discussion_r138870106
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
@@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
for (DataRefNode dataRefNode : dataRefNodes) {
BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
+ String[] deleteDeltaFilePath = null;
if (isIUDTable) {
// In case IUD is not performed in this table avoid searching for
// invalidated blocks.
if (CarbonUtil
- .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
- invalidBlockVOForSegmentId, updateStatusManager)) {
+ .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
--- End diff --
Changes done in CarbonTableInputFormat and changes is pushed
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
retest this please
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1324
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/18/
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
@kushalsaha Please provide some details on the PR why this change required
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/764/
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1324
Can one of the admins verify this patch?
---
[GitHub] carbondata issue #1324: [CARBONDATA-1291]:carbonData query performance impro...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1324
LGTM
---
[GitHub] carbondata pull request #1324: [CARBONDATA-1291]:carbonData query performanc...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1324#discussion_r138863711
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
@@ -460,21 +460,22 @@ private BitSet setMatchedPartitions(String partitionIds, CarbonTable carbonTable
for (DataRefNode dataRefNode : dataRefNodes) {
BlockBTreeLeafNode leafNode = (BlockBTreeLeafNode) dataRefNode;
TableBlockInfo tableBlockInfo = leafNode.getTableBlockInfo();
+ String[] deleteDeltaFilePath = null;
if (isIUDTable) {
// In case IUD is not performed in this table avoid searching for
// invalidated blocks.
if (CarbonUtil
- .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
- invalidBlockVOForSegmentId, updateStatusManager)) {
+ .isInvalidTableBlock(tableBlockInfo.getSegmentId(), tableBlockInfo.getFilePath(),
--- End diff --
you can keep here , not an issue since this class going to be removed soon. Please do changes in `CarbonTableInputFormat`
---