You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2017/09/21 05:47:38 UTC
[GitHub] carbondata pull request #1376: [WIP] Get the detailed blocklet information u...
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1376
[WIP] Get the detailed blocklet information using default BlockletDataMap from other datamaps
All the detail information of blocklet which is need for exceuting query is present only BlockletDataMap. It is actually default datamap.
So if new datamap is added then it gives only information of blocklet and blockid, it is insuffucient information to exceute query.
Now this PR adds the functionality of retrieving detailed blocklet information from the BlockletDataMap based on block and blocklet id. So now new datamaps can only concentrate on business logic and return only block and blockletid.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata datamap-refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1376.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1376
----
commit 5dce79bf6ac1bb279c24b0f56734778abc4664d7
Author: Ravindra Pesala <ra...@gmail.com>
Date: 2017-09-21T05:40:57Z
Refactor datamap to get the detailed blocklet information from default BlockletDataMap
----
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1376
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/191/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/136/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141535661
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -78,8 +89,55 @@ public DataMapWriter createWriter(String segmentId) {
new TableBlockIndexUniqueIdentifier(identifier, segmentId, listFiles[i].getName()));
}
}
+ return tableBlockIndexUniqueIdentifiers;
+ }
- return cache.getAll(tableBlockIndexUniqueIdentifiers);
+ /**
+ * Get the blocklet detail information based on blockletid, blockid and segmentid. This method is
+ * exclusively for BlockletDataMapFactory as detail information is only available in this default
+ * datamap.
+ */
+ @Override
+ public List<ExtendedBlocklet> getDetailedBlocklets(List<Blocklet> blocklets, String segmentId)
+ throws IOException {
+ List<ExtendedBlocklet> detailedBlocklets = new ArrayList<>();
+ // If it is already detailed blocklet then type cast and return same
+ if (blocklets.size() > 0 && blocklets.get(0) instanceof ExtendedBlocklet) {
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add((ExtendedBlocklet) blocklet);
+ }
+ return detailedBlocklets;
+ }
+ List<TableBlockIndexUniqueIdentifier> identifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+ // Retrieve each blocklets detail information from blocklet datamap
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add(getDetailedBlocklet(identifiers, blocklet));
+ }
+ return detailedBlocklets;
+ }
+
+ @Override
+ public ExtendedBlocklet getDetailedBlocklet(Blocklet blocklet, String segmentId)
--- End diff --
ok
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/946/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141525864
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -78,8 +90,59 @@ public DataMapWriter createWriter(String segmentId) {
new TableBlockIndexUniqueIdentifier(identifier, segmentId, listFiles[i].getName()));
}
}
+ return tableBlockIndexUniqueIdentifiers;
+ }
- return cache.getAll(tableBlockIndexUniqueIdentifiers);
+ /**
+ * Get the blocklet detail information based on blockletid, blockid and segmentid. This method is
+ * exclusively for BlockletDataMapFactory as detail information is only available in this default
+ * datamap.
+ * @param blocklets
--- End diff --
ok
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/942/
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/895/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141349763
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java ---
@@ -561,7 +567,7 @@ protected Expression getFilterPredicates(Configuration configuration) {
return resultFilterredBlocks;
}
- private org.apache.carbondata.hadoop.CarbonInputSplit convertToCarbonInputSplit(Blocklet blocklet)
+ private CarbonInputSplit convertToCarbonInputSplit(DetailedBlocklet blocklet)
throws IOException {
blocklet.updateLocations();
org.apache.carbondata.hadoop.CarbonInputSplit split =
--- End diff --
Why not just use `CarbonInputSplit` instead of `org.apache.carbondata.hadoop.CarbonInputSplit`
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141534437
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -78,8 +89,55 @@ public DataMapWriter createWriter(String segmentId) {
new TableBlockIndexUniqueIdentifier(identifier, segmentId, listFiles[i].getName()));
}
}
+ return tableBlockIndexUniqueIdentifiers;
+ }
- return cache.getAll(tableBlockIndexUniqueIdentifiers);
+ /**
+ * Get the blocklet detail information based on blockletid, blockid and segmentid. This method is
+ * exclusively for BlockletDataMapFactory as detail information is only available in this default
+ * datamap.
+ */
+ @Override
+ public List<ExtendedBlocklet> getDetailedBlocklets(List<Blocklet> blocklets, String segmentId)
+ throws IOException {
+ List<ExtendedBlocklet> detailedBlocklets = new ArrayList<>();
+ // If it is already detailed blocklet then type cast and return same
+ if (blocklets.size() > 0 && blocklets.get(0) instanceof ExtendedBlocklet) {
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add((ExtendedBlocklet) blocklet);
+ }
+ return detailedBlocklets;
+ }
+ List<TableBlockIndexUniqueIdentifier> identifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+ // Retrieve each blocklets detail information from blocklet datamap
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add(getDetailedBlocklet(identifiers, blocklet));
+ }
+ return detailedBlocklets;
+ }
+
+ @Override
+ public ExtendedBlocklet getDetailedBlocklet(Blocklet blocklet, String segmentId)
--- End diff --
modify the function name
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141525805
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -69,6 +73,14 @@ public DataMapWriter createWriter(String segmentId) {
@Override
public List<DataMap> getDataMaps(String segmentId) throws IOException {
List<TableBlockIndexUniqueIdentifier> tableBlockIndexUniqueIdentifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+
--- End diff --
ok
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/264/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/129/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/261/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/252/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/892/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/884/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/883/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/137/
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/186/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141350038
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/DetailedBlocklet.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.indexstore;
+
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+
+/**
+ * Detailed blocklet information
+ */
+public class DetailedBlocklet extends Blocklet {
+
+ private String segmentId;
+
+ private BlockletDetailInfo detailInfo;
+
+ private long length;
+
+ private String[] location;
+
+ public DetailedBlocklet(String path, String blockletId) {
+ super(path, blockletId);
+ }
+
+ public BlockletDetailInfo getDetailInfo() {
+ return detailInfo;
+ }
+
+ public void setDetailInfo(BlockletDetailInfo detailInfo) {
+ this.detailInfo = detailInfo;
+ }
+
+ public void updateLocations() throws IOException {
--- End diff --
What is the purpose of this function?
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/891/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/262/
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/140/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141351076
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -78,8 +90,59 @@ public DataMapWriter createWriter(String segmentId) {
new TableBlockIndexUniqueIdentifier(identifier, segmentId, listFiles[i].getName()));
}
}
+ return tableBlockIndexUniqueIdentifiers;
+ }
- return cache.getAll(tableBlockIndexUniqueIdentifiers);
+ /**
+ * Get the blocklet detail information based on blockletid, blockid and segmentid. This method is
+ * exclusively for BlockletDataMapFactory as detail information is only available in this default
+ * datamap.
+ * @param blocklets
--- End diff --
remove javadoc that is empty
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141349094
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/DetailedBlocklet.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.indexstore;
+
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+
+/**
+ * Detailed blocklet information
+ */
+public class DetailedBlocklet extends Blocklet {
--- End diff --
It seems `BlockletDetailInfo` and `DetailedBlocklet` is a bit confusing, is there a better name?
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/260/
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/316/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141350886
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -69,6 +73,14 @@ public DataMapWriter createWriter(String segmentId) {
@Override
public List<DataMap> getDataMaps(String segmentId) throws IOException {
List<TableBlockIndexUniqueIdentifier> tableBlockIndexUniqueIdentifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+
--- End diff --
remove empty line
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/128/
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/311/
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1376
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/893/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141525775
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/DetailedBlocklet.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.indexstore;
+
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+
+/**
+ * Detailed blocklet information
+ */
+public class DetailedBlocklet extends Blocklet {
+
+ private String segmentId;
+
+ private BlockletDetailInfo detailInfo;
+
+ private long length;
+
+ private String[] location;
+
+ public DetailedBlocklet(String path, String blockletId) {
+ super(path, blockletId);
+ }
+
+ public BlockletDetailInfo getDetailInfo() {
+ return detailInfo;
+ }
+
+ public void setDetailInfo(BlockletDetailInfo detailInfo) {
+ this.detailInfo = detailInfo;
+ }
+
+ public void updateLocations() throws IOException {
--- End diff --
I have added the comment to this method. It is not added to constructor to avoid namenode calls during datamap pruning. So that it decouples this calls from external datamap.
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/138/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141534453
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -78,8 +89,55 @@ public DataMapWriter createWriter(String segmentId) {
new TableBlockIndexUniqueIdentifier(identifier, segmentId, listFiles[i].getName()));
}
}
+ return tableBlockIndexUniqueIdentifiers;
+ }
- return cache.getAll(tableBlockIndexUniqueIdentifiers);
+ /**
+ * Get the blocklet detail information based on blockletid, blockid and segmentid. This method is
+ * exclusively for BlockletDataMapFactory as detail information is only available in this default
+ * datamap.
+ */
+ @Override
+ public List<ExtendedBlocklet> getDetailedBlocklets(List<Blocklet> blocklets, String segmentId)
+ throws IOException {
+ List<ExtendedBlocklet> detailedBlocklets = new ArrayList<>();
+ // If it is already detailed blocklet then type cast and return same
+ if (blocklets.size() > 0 && blocklets.get(0) instanceof ExtendedBlocklet) {
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add((ExtendedBlocklet) blocklet);
+ }
+ return detailedBlocklets;
+ }
+ List<TableBlockIndexUniqueIdentifier> identifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+ // Retrieve each blocklets detail information from blocklet datamap
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add(getDetailedBlocklet(identifiers, blocklet));
+ }
+ return detailedBlocklets;
+ }
+
+ @Override
+ public ExtendedBlocklet getDetailedBlocklet(Blocklet blocklet, String segmentId)
+ throws IOException {
+ if (blocklet instanceof ExtendedBlocklet) {
+ return (ExtendedBlocklet) blocklet;
+ }
+ List<TableBlockIndexUniqueIdentifier> identifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+ return getDetailedBlocklet(identifiers, blocklet);
+ }
+
+ private ExtendedBlocklet getDetailedBlocklet(List<TableBlockIndexUniqueIdentifier> identifiers,
--- End diff --
modify the function name
---
[GitHub] carbondata issue #1376: [CARBONDATA-1505] Get the detailed blocklet informat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1376
LGTM
---
[GitHub] carbondata issue #1376: [WIP] Get the detailed blocklet information using de...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1376
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/253/
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141525258
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/DetailedBlocklet.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.indexstore;
+
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+
+/**
+ * Detailed blocklet information
+ */
+public class DetailedBlocklet extends Blocklet {
--- End diff --
Ok, I will name `DetailedBlocklet` as `ExtendedBlocklet`
---
[GitHub] carbondata pull request #1376: [CARBONDATA-1505] Get the detailed blocklet i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1376#discussion_r141535670
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java ---
@@ -78,8 +89,55 @@ public DataMapWriter createWriter(String segmentId) {
new TableBlockIndexUniqueIdentifier(identifier, segmentId, listFiles[i].getName()));
}
}
+ return tableBlockIndexUniqueIdentifiers;
+ }
- return cache.getAll(tableBlockIndexUniqueIdentifiers);
+ /**
+ * Get the blocklet detail information based on blockletid, blockid and segmentid. This method is
+ * exclusively for BlockletDataMapFactory as detail information is only available in this default
+ * datamap.
+ */
+ @Override
+ public List<ExtendedBlocklet> getDetailedBlocklets(List<Blocklet> blocklets, String segmentId)
+ throws IOException {
+ List<ExtendedBlocklet> detailedBlocklets = new ArrayList<>();
+ // If it is already detailed blocklet then type cast and return same
+ if (blocklets.size() > 0 && blocklets.get(0) instanceof ExtendedBlocklet) {
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add((ExtendedBlocklet) blocklet);
+ }
+ return detailedBlocklets;
+ }
+ List<TableBlockIndexUniqueIdentifier> identifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+ // Retrieve each blocklets detail information from blocklet datamap
+ for (Blocklet blocklet : blocklets) {
+ detailedBlocklets.add(getDetailedBlocklet(identifiers, blocklet));
+ }
+ return detailedBlocklets;
+ }
+
+ @Override
+ public ExtendedBlocklet getDetailedBlocklet(Blocklet blocklet, String segmentId)
+ throws IOException {
+ if (blocklet instanceof ExtendedBlocklet) {
+ return (ExtendedBlocklet) blocklet;
+ }
+ List<TableBlockIndexUniqueIdentifier> identifiers =
+ getTableBlockIndexUniqueIdentifiers(segmentId);
+ return getDetailedBlocklet(identifiers, blocklet);
+ }
+
+ private ExtendedBlocklet getDetailedBlocklet(List<TableBlockIndexUniqueIdentifier> identifiers,
--- End diff --
ok
---