You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ku...@apache.org on 2019/08/02 07:30:14 UTC
[carbondata] 02/03: [CARBONDATA-3481] Multi-thread pruning fails
when datamaps count is just near numOfThreadsForPruning
This is an automated email from the ASF dual-hosted git repository.
kumarvishal09 pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 575b7116e5cc0a7c25e17794a462a6ecdf4afb24
Author: ajantha-bhat <aj...@gmail.com>
AuthorDate: Thu Jul 25 18:50:19 2019 +0530
[CARBONDATA-3481] Multi-thread pruning fails when datamaps count is just near numOfThreadsForPruning
Cause : When the datamaps count is just near numOfThreadsForPruning,
As code is checking '>= ', last thread may not get the datamaps for prune.
Hence array out of index exception is thrown in this scenario.
There is no issues with higher number of datamaps.
Solution: In this scenario launch threads based on the distribution value,
not on the hardcoded value
This closes #3336
---
.../org/apache/carbondata/core/datamap/TableDataMap.java | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java b/core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java
index 33fc3b1..ecdd586 100644
--- a/core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java
+++ b/core/src/main/java/org/apache/carbondata/core/datamap/TableDataMap.java
@@ -207,9 +207,6 @@ public final class TableDataMap extends OperationEventListener {
*/
int numOfThreadsForPruning = CarbonProperties.getNumOfThreadsForPruning();
- LOG.info(
- "Number of threads selected for multi-thread block pruning is " + numOfThreadsForPruning
- + ". total files: " + totalFiles + ". total segments: " + segments.size());
int filesPerEachThread = totalFiles / numOfThreadsForPruning;
int prev;
int filesCount = 0;
@@ -254,6 +251,15 @@ public final class TableDataMap extends OperationEventListener {
// this should not happen
throw new RuntimeException(" not all the files processed ");
}
+ if (datamapListForEachThread.size() < numOfThreadsForPruning) {
+ // If the total datamaps fitted in lesser number of threads than numOfThreadsForPruning.
+ // Launch only that many threads where datamaps are fitted while grouping.
+ LOG.info("Datamaps is distributed in " + datamapListForEachThread.size() + " threads");
+ numOfThreadsForPruning = datamapListForEachThread.size();
+ }
+ LOG.info(
+ "Number of threads selected for multi-thread block pruning is " + numOfThreadsForPruning
+ + ". total files: " + totalFiles + ". total segments: " + segments.size());
List<Future<Void>> results = new ArrayList<>(numOfThreadsForPruning);
final Map<Segment, List<ExtendedBlocklet>> prunedBlockletMap =
new ConcurrentHashMap<>(segments.size());