You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by kumarvishal09 <gi...@git.apache.org> on 2016/10/27 17:34:38 UTC
[GitHub] incubator-carbondata pull request #265: [WIP]Improve first time query perfor...
GitHub user kumarvishal09 opened a pull request:
https://github.com/apache/incubator-carbondata/pull/265
[WIP]Improve first time query performance
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[CARBONDATA-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- What manual testing you have done?
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
---
Improve first time query performance
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kumarvishal09/incubator-carbondata FirstTimeQueryPerformanceImprovement
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/265.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #265
----
commit 6af76c5d2d9022740624429a31d657b81db83126
Author: kumarvishal <ku...@gmail.com>
Date: 2016-10-27T17:24:49Z
Improve first time query performance
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #265: [CARBONDATA-458]Improving First time query ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/incubator-carbondata/pull/265
please add testcase for V2 write and read
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/265
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90395469
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -814,85 +810,86 @@
* Rocord size in case of compaction.
*/
public static final int COMPACTION_INMEMORY_RECORD_SIZE = 120000;
-
- /**
- * If the level 2 compaction is done in minor then new compacted segment will end with .2
- */
- public static String LEVEL2_COMPACTION_INDEX = ".2";
-
- /**
- * Indicates compaction
- */
- public static String COMPACTION_KEY_WORD = "COMPACTION";
-
/**
* hdfs temporary directory key
*/
public static final String HDFS_TEMP_LOCATION = "hadoop.tmp.dir";
-
/**
* zookeeper url key
*/
public static final String ZOOKEEPER_URL = "spark.deploy.zookeeper.url";
-
/**
* configure the minimum blocklet size eligible for blocklet distribution
*/
public static final String CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE =
"carbon.blockletdistribution.min.blocklet.size";
-
/**
* default blocklet size eligible for blocklet distribution
*/
public static final int DEFAULT_CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE = 2;
-
+ /**
+ * This batch size is used to send rows from load step to another step in batches.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE = "DATA_LOAD_BATCH_SIZE";
+ /**
+ * Default size of data load batch size.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE_DEFAULT = "1000";
+ /**
+ * carbon data file version property
+ */
+ public static final String CARBON_DATA_FILE_VERSION = "carbon.data.file.version";
+ /**
+ * current data file version
+ */
+ public static final short CARBON_DATA_FILE_CURRENT_VERSION = 2;
+ /**
+ * data file version header
+ */
+ public static final String CARBON_DATA_VERSION_HEADER = "CARBONDATAVERSION#";
+ /**
+ * If the level 2 compaction is done in minor then new compacted segment will end with .2
+ */
+ public static String LEVEL2_COMPACTION_INDEX = ".2";
--- End diff --
By minstake i have checked in, will remove this constant
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90395264
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/measure/CompressedMeasureChunkFileReader2.java ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.carbon.datastore.chunk.reader.measure;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+
+import org.apache.carbondata.core.carbon.datastore.chunk.MeasureColumnDataChunk;
+import org.apache.carbondata.core.carbon.metadata.blocklet.BlockletInfo;
+import org.apache.carbondata.core.carbon.metadata.blocklet.datachunk.PresenceMeta;
+import org.apache.carbondata.core.datastorage.store.FileHolder;
+import org.apache.carbondata.core.datastorage.store.compression.SnappyCompression.SnappyByteCompression;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressionModel;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder.UnCompressValue;
+import org.apache.carbondata.core.datastorage.store.dataholder.CarbonReadDataHolder;
+import org.apache.carbondata.core.metadata.ValueEncoderMeta;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.format.DataChunk2;
+
+/**
+ * Class to read the measure column data for version 2
+ */
+public class CompressedMeasureChunkFileReader2 extends AbstractMeasureChunkReader {
+
+ /**
+ * measure column chunks offset
+ */
+ private List<Long> measureColumnChunkOffsets;
+
+ /**
+ * measure column chunks length
+ */
+ private List<Short> measureColumnChunkLength;
+
+ /**
+ * Constructor to get minimum parameter to create instance of this class
+ *
+ * @param blockletInfo BlockletInfo
+ * @param filePath file from which data will be read
+ */
+ public CompressedMeasureChunkFileReader2(final BlockletInfo blockletInfo, final String filePath) {
+ super(filePath);
+ this.measureColumnChunkOffsets = blockletInfo.getMeasureChunkOffsets();
+ this.measureColumnChunkLength = blockletInfo.getMeasureChunksLength();
+ }
+
+ /**
+ * Below method will be used to convert the thrift presence meta to wrapper
+ * presence meta
+ *
+ * @param presentMetadataThrift
+ * @return wrapper presence meta
+ */
+ private static PresenceMeta getPresenceMeta(
+ org.apache.carbondata.format.PresenceMeta presentMetadataThrift) {
+ PresenceMeta presenceMeta = new PresenceMeta();
+ presenceMeta.setRepresentNullValues(presentMetadataThrift.isRepresents_presence());
+ presenceMeta.setBitSet(BitSet.valueOf(
+ SnappyByteCompression.INSTANCE.unCompress(presentMetadataThrift.getPresent_bit_stream())));
+ return presenceMeta;
+ }
+
+ /**
+ * Below method will be used to read the chunk based on block indexes
+ * Reading logic of below method is:
+ * Except last column all the column chunk can be read in group
+ * if not last column then read data of all the column present in block index
+ * together then process it.
+ * For last column read is separately and process
+ *
+ * @param fileReader file reader to read the blocks from file
+ * @param blockIndexes blocks range to be read
+ * @return measure column chunks
+ */
+ public MeasureColumnDataChunk[] readMeasureChunks(FileHolder fileReader, int[][] blockIndexes) {
+ // read the column chunk based on block index and add
+ MeasureColumnDataChunk[] dataChunks =
+ new MeasureColumnDataChunk[measureColumnChunkOffsets.size()];
+ if (blockIndexes.length == 0) {
+ return dataChunks;
+ }
+ MeasureColumnDataChunk[] groupChunk = null;
+ int index = 0;
+ for (int i = 0; i < blockIndexes.length - 1; i++) {
+ index = 0;
+ groupChunk = readMeasureChunksInGroup(fileReader, blockIndexes[i][0], blockIndexes[i][1]);
+ for (int j = blockIndexes[i][0]; j <= blockIndexes[i][1]; j++) {
+ dataChunks[j] = groupChunk[index++];
+ }
+ }
+ if (blockIndexes[blockIndexes.length - 1][0] == measureColumnChunkOffsets.size() - 1) {
+ dataChunks[blockIndexes[blockIndexes.length - 1][0]] =
+ readMeasureChunk(fileReader, blockIndexes[blockIndexes.length - 1][0]);
+ } else {
+ groupChunk = readMeasureChunksInGroup(fileReader, blockIndexes[blockIndexes.length - 1][0],
+ blockIndexes[blockIndexes.length - 1][1]);
+ index = 0;
+ for (int j = blockIndexes[blockIndexes.length - 1][0];
+ j <= blockIndexes[blockIndexes.length - 1][1]; j++) {
+ dataChunks[j] = groupChunk[index++];
+ }
+ }
+ return dataChunks;
+ }
+
+ /**
+ * Method to read the blocks data based on block index
+ *
+ * @param fileReader file reader to read the blocks
+ * @param blockIndex block to be read
+ * @return measure data chunk
+ */
+ @Override public MeasureColumnDataChunk readMeasureChunk(FileHolder fileReader, int blockIndex) {
+ MeasureColumnDataChunk datChunk = new MeasureColumnDataChunk();
+ DataChunk2 measureColumnChunk = null;
+ byte[] measureDataChunk = null;
+ byte[] data = null;
+ byte[] dataPage = null;
+ if (measureColumnChunkOffsets.size() - 1 == blockIndex) {
+ measureDataChunk = fileReader
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #265: [CARBONDATA-458]Improving First time query ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/incubator-carbondata/pull/265
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90400739
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
@@ -340,17 +341,17 @@ private Expression getFilterPredicates(Configuration configuration) {
}
resultFilterredBlocks.addAll(filterredBlocks);
}
- statistic.addStatistics(QueryStatisticsConstants.LOAD_BLOCKS_DRIVER,
- System.currentTimeMillis());
+ statistic
+ .addStatistics(QueryStatisticsConstants.LOAD_BLOCKS_DRIVER, System.currentTimeMillis());
--- End diff --
no need to
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90395320
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/metadata/blocklet/DataFileFooter.java ---
@@ -38,7 +38,7 @@
/**
* version used for data compatibility
*/
- private int versionId;
+ private short versionId;
--- End diff --
short range is enough to store version. Please let me know if need to store in int is required
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90400632
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java ---
@@ -193,8 +203,7 @@ public static void setSegmentsToAccess(Configuration configuration, List<String>
* @return List<InputSplit> list of CarbonInputSplit
* @throws IOException
*/
- @Override
- public List<InputSplit> getSplits(JobContext job) throws IOException {
+ @Override public List<InputSplit> getSplits(JobContext job) throws IOException {
--- End diff --
move `Override` to previous line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90399780
--- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -236,29 +236,57 @@ private void validateHighCardinalityThreshold() {
}
private void validateHighCardinalityInRowCountPercentage() {
- String highcardPercentageStr = carbonProperties.getProperty(
- CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE,
- CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE_DEFAULT);
+ String highcardPercentageStr = carbonProperties
+ .getProperty(CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE,
+ CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE_DEFAULT);
try {
double highcardPercentage = Double.parseDouble(highcardPercentageStr);
- if(highcardPercentage <= 0){
- LOGGER.info("The percentage of high cardinality in row count value \""
- + highcardPercentageStr + "\" is invalid. Using the default value \""
- + CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE_DEFAULT);
- carbonProperties.setProperty(
- CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE,
+ if (highcardPercentage <= 0) {
+ LOGGER.info(
+ "The percentage of high cardinality in row count value \"" + highcardPercentageStr
+ + "\" is invalid. Using the default value \""
+ + CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE_DEFAULT);
+ carbonProperties.setProperty(CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE,
CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE_DEFAULT);
}
} catch (NumberFormatException e) {
- LOGGER.info("The percentage of high cardinality in row count value \""
- + highcardPercentageStr + "\" is invalid. Using the default value \""
+ LOGGER.info("The percentage of high cardinality in row count value \"" + highcardPercentageStr
+ + "\" is invalid. Using the default value \""
+ CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE_DEFAULT);
carbonProperties.setProperty(CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE,
CarbonCommonConstants.HIGH_CARDINALITY_IN_ROW_COUNT_PERCENTAGE_DEFAULT);
}
}
/**
+ * Below method will be used to validate the data file version parameter
+ * if parameter is invalid current version will be set
+ */
+ private void validateCarbonDataFileVersion() {
+ short carbondataFileVersion = CarbonCommonConstants.CARBON_DATA_FILE_CURRENT_VERSION;
--- End diff --
this initial value is of no use
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90395412
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -814,85 +810,86 @@
* Rocord size in case of compaction.
*/
public static final int COMPACTION_INMEMORY_RECORD_SIZE = 120000;
-
- /**
- * If the level 2 compaction is done in minor then new compacted segment will end with .2
- */
- public static String LEVEL2_COMPACTION_INDEX = ".2";
-
- /**
- * Indicates compaction
- */
- public static String COMPACTION_KEY_WORD = "COMPACTION";
-
/**
* hdfs temporary directory key
*/
public static final String HDFS_TEMP_LOCATION = "hadoop.tmp.dir";
-
/**
* zookeeper url key
*/
public static final String ZOOKEEPER_URL = "spark.deploy.zookeeper.url";
-
/**
* configure the minimum blocklet size eligible for blocklet distribution
*/
public static final String CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE =
"carbon.blockletdistribution.min.blocklet.size";
-
/**
* default blocklet size eligible for blocklet distribution
*/
public static final int DEFAULT_CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE = 2;
-
+ /**
+ * This batch size is used to send rows from load step to another step in batches.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE = "DATA_LOAD_BATCH_SIZE";
+ /**
+ * Default size of data load batch size.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE_DEFAULT = "1000";
+ /**
+ * carbon data file version property
+ */
+ public static final String CARBON_DATA_FILE_VERSION = "carbon.data.file.version";
+ /**
+ * current data file version
+ */
+ public static final short CARBON_DATA_FILE_CURRENT_VERSION = 2;
--- End diff --
ok i will change it to some meaning full name
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90395252
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/measure/CompressedMeasureChunkFileReader2.java ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.carbon.datastore.chunk.reader.measure;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+
+import org.apache.carbondata.core.carbon.datastore.chunk.MeasureColumnDataChunk;
+import org.apache.carbondata.core.carbon.metadata.blocklet.BlockletInfo;
+import org.apache.carbondata.core.carbon.metadata.blocklet.datachunk.PresenceMeta;
+import org.apache.carbondata.core.datastorage.store.FileHolder;
+import org.apache.carbondata.core.datastorage.store.compression.SnappyCompression.SnappyByteCompression;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressionModel;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder.UnCompressValue;
+import org.apache.carbondata.core.datastorage.store.dataholder.CarbonReadDataHolder;
+import org.apache.carbondata.core.metadata.ValueEncoderMeta;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.format.DataChunk2;
+
+/**
+ * Class to read the measure column data for version 2
+ */
+public class CompressedMeasureChunkFileReader2 extends AbstractMeasureChunkReader {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90249468
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/metadata/blocklet/DataFileFooter.java ---
@@ -38,7 +38,7 @@
/**
* version used for data compatibility
*/
- private int versionId;
+ private short versionId;
--- End diff --
why change version from int to short?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90249260
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/measure/CompressedMeasureChunkFileReader2.java ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.carbon.datastore.chunk.reader.measure;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+
+import org.apache.carbondata.core.carbon.datastore.chunk.MeasureColumnDataChunk;
+import org.apache.carbondata.core.carbon.metadata.blocklet.BlockletInfo;
+import org.apache.carbondata.core.carbon.metadata.blocklet.datachunk.PresenceMeta;
+import org.apache.carbondata.core.datastorage.store.FileHolder;
+import org.apache.carbondata.core.datastorage.store.compression.SnappyCompression.SnappyByteCompression;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressionModel;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder.UnCompressValue;
+import org.apache.carbondata.core.datastorage.store.dataholder.CarbonReadDataHolder;
+import org.apache.carbondata.core.metadata.ValueEncoderMeta;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.format.DataChunk2;
+
+/**
+ * Class to read the measure column data for version 2
+ */
+public class CompressedMeasureChunkFileReader2 extends AbstractMeasureChunkReader {
+
+ /**
+ * measure column chunks offset
+ */
+ private List<Long> measureColumnChunkOffsets;
+
+ /**
+ * measure column chunks length
+ */
+ private List<Short> measureColumnChunkLength;
+
+ /**
+ * Constructor to get minimum parameter to create instance of this class
+ *
+ * @param blockletInfo BlockletInfo
+ * @param filePath file from which data will be read
+ */
+ public CompressedMeasureChunkFileReader2(final BlockletInfo blockletInfo, final String filePath) {
+ super(filePath);
+ this.measureColumnChunkOffsets = blockletInfo.getMeasureChunkOffsets();
+ this.measureColumnChunkLength = blockletInfo.getMeasureChunksLength();
+ }
+
+ /**
+ * Below method will be used to convert the thrift presence meta to wrapper
+ * presence meta
+ *
+ * @param presentMetadataThrift
+ * @return wrapper presence meta
+ */
+ private static PresenceMeta getPresenceMeta(
+ org.apache.carbondata.format.PresenceMeta presentMetadataThrift) {
+ PresenceMeta presenceMeta = new PresenceMeta();
+ presenceMeta.setRepresentNullValues(presentMetadataThrift.isRepresents_presence());
+ presenceMeta.setBitSet(BitSet.valueOf(
+ SnappyByteCompression.INSTANCE.unCompress(presentMetadataThrift.getPresent_bit_stream())));
+ return presenceMeta;
+ }
+
+ /**
+ * Below method will be used to read the chunk based on block indexes
+ * Reading logic of below method is:
+ * Except last column all the column chunk can be read in group
+ * if not last column then read data of all the column present in block index
+ * together then process it.
+ * For last column read is separately and process
+ *
+ * @param fileReader file reader to read the blocks from file
+ * @param blockIndexes blocks range to be read
+ * @return measure column chunks
+ */
+ public MeasureColumnDataChunk[] readMeasureChunks(FileHolder fileReader, int[][] blockIndexes) {
+ // read the column chunk based on block index and add
+ MeasureColumnDataChunk[] dataChunks =
+ new MeasureColumnDataChunk[measureColumnChunkOffsets.size()];
+ if (blockIndexes.length == 0) {
+ return dataChunks;
+ }
+ MeasureColumnDataChunk[] groupChunk = null;
+ int index = 0;
+ for (int i = 0; i < blockIndexes.length - 1; i++) {
+ index = 0;
+ groupChunk = readMeasureChunksInGroup(fileReader, blockIndexes[i][0], blockIndexes[i][1]);
+ for (int j = blockIndexes[i][0]; j <= blockIndexes[i][1]; j++) {
+ dataChunks[j] = groupChunk[index++];
+ }
+ }
+ if (blockIndexes[blockIndexes.length - 1][0] == measureColumnChunkOffsets.size() - 1) {
+ dataChunks[blockIndexes[blockIndexes.length - 1][0]] =
+ readMeasureChunk(fileReader, blockIndexes[blockIndexes.length - 1][0]);
+ } else {
+ groupChunk = readMeasureChunksInGroup(fileReader, blockIndexes[blockIndexes.length - 1][0],
+ blockIndexes[blockIndexes.length - 1][1]);
+ index = 0;
+ for (int j = blockIndexes[blockIndexes.length - 1][0];
+ j <= blockIndexes[blockIndexes.length - 1][1]; j++) {
+ dataChunks[j] = groupChunk[index++];
+ }
+ }
+ return dataChunks;
+ }
+
+ /**
+ * Method to read the blocks data based on block index
+ *
+ * @param fileReader file reader to read the blocks
+ * @param blockIndex block to be read
+ * @return measure data chunk
+ */
+ @Override public MeasureColumnDataChunk readMeasureChunk(FileHolder fileReader, int blockIndex) {
+ MeasureColumnDataChunk datChunk = new MeasureColumnDataChunk();
+ DataChunk2 measureColumnChunk = null;
+ byte[] measureDataChunk = null;
+ byte[] data = null;
+ byte[] dataPage = null;
+ if (measureColumnChunkOffsets.size() - 1 == blockIndex) {
+ measureDataChunk = fileReader
--- End diff --
move fileReader to next line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90245110
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/CarbonDataReaderFactory.java ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.carbon.datastore.chunk.reader;
+
+import org.apache.carbondata.core.carbon.datastore.chunk.reader.dimension.CompressedDimensionChunkFileBasedReader;
+import org.apache.carbondata.core.carbon.datastore.chunk.reader.dimension.CompressedDimensionChunkFileBasedReader2;
+import org.apache.carbondata.core.carbon.datastore.chunk.reader.measure.CompressedMeasureChunkFileBasedReader;
+import org.apache.carbondata.core.carbon.datastore.chunk.reader.measure.CompressedMeasureChunkFileReader2;
+import org.apache.carbondata.core.carbon.metadata.blocklet.BlockletInfo;
+
+/**
+ * Factory class to get the data reader instance based on version
+ */
+public class CarbonDataReaderFactory {
+
+ /**
+ * static instance
+ */
+ private static final CarbonDataReaderFactory CARBON_DATA_READER_FACTORY =
+ new CarbonDataReaderFactory();
+
+ /**
+ * private constructor
+ */
+ private CarbonDataReaderFactory() {
+
+ }
+
+ /**
+ * To get the instance of the reader factor
+ *
+ * @return reader factory
+ */
+ public static CarbonDataReaderFactory getInstance() {
+ return CARBON_DATA_READER_FACTORY;
+ }
+
+ /**
+ * Below method will be used to get the dimension column chunk reader based on version number
+ *
+ * @param version reader version
+ * @param blockletInfo blocklet info
+ * @param eachColumnValueSize size of each dimension column
+ * @param filePath carbon data file path
+ * @return dimension column data reader based on version number
+ */
+ public DimensionColumnChunkReader getDimensionColumnChunkReader(short version,
+ BlockletInfo blockletInfo, int[] eachColumnValueSize, String filePath) {
+ switch (version) {
+ case 2:
--- End diff --
should have enum for version
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #265: [CARBONDATA-458]Improving First time query ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/265
@jackylk all the existing test cases are running with V2 and some I have updated with V1 version. ok I will add test cases for reading V1 version file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90250147
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -814,85 +810,86 @@
* Rocord size in case of compaction.
*/
public static final int COMPACTION_INMEMORY_RECORD_SIZE = 120000;
-
- /**
- * If the level 2 compaction is done in minor then new compacted segment will end with .2
- */
- public static String LEVEL2_COMPACTION_INDEX = ".2";
-
- /**
- * Indicates compaction
- */
- public static String COMPACTION_KEY_WORD = "COMPACTION";
-
/**
* hdfs temporary directory key
*/
public static final String HDFS_TEMP_LOCATION = "hadoop.tmp.dir";
-
/**
* zookeeper url key
*/
public static final String ZOOKEEPER_URL = "spark.deploy.zookeeper.url";
-
/**
* configure the minimum blocklet size eligible for blocklet distribution
*/
public static final String CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE =
"carbon.blockletdistribution.min.blocklet.size";
-
/**
* default blocklet size eligible for blocklet distribution
*/
public static final int DEFAULT_CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE = 2;
-
+ /**
+ * This batch size is used to send rows from load step to another step in batches.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE = "DATA_LOAD_BATCH_SIZE";
+ /**
+ * Default size of data load batch size.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE_DEFAULT = "1000";
+ /**
+ * carbon data file version property
+ */
+ public static final String CARBON_DATA_FILE_VERSION = "carbon.data.file.version";
+ /**
+ * current data file version
+ */
+ public static final short CARBON_DATA_FILE_CURRENT_VERSION = 2;
+ /**
+ * data file version header
+ */
+ public static final String CARBON_DATA_VERSION_HEADER = "CARBONDATAVERSION#";
+ /**
+ * If the level 2 compaction is done in minor then new compacted segment will end with .2
+ */
+ public static String LEVEL2_COMPACTION_INDEX = ".2";
--- End diff --
why need this in this PR, is't it for compaction?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #265: [CARBONDATA-458]Improving First time query ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/265
http://136.243.101.176:8080/job/ApacheCarbonManualPRBuilder/org.apache.carbondata$carbondata-spark/719/testReport/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #265: [CARBONDATA-458]Improving First time query ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/265
**Note**
Since there is change in schema.thrift and hence carbon-format jar has to be build and deployed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90386030
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/block/TableBlockInfo.java ---
@@ -81,13 +84,14 @@ public TableBlockInfo(String filePath, long blockOffset, String segmentId, Strin
* @param blockletInfos
*/
public TableBlockInfo(String filePath, long blockOffset, String segmentId, String[] locations,
- long blockLength, BlockletInfos blockletInfos) {
+ long blockLength, BlockletInfos blockletInfos, short version) {
this.filePath = FileFactory.getUpdatedFilePath(filePath);
this.blockOffset = blockOffset;
this.segmentId = segmentId;
this.locations = locations;
this.blockLength = blockLength;
this.blockletInfos = blockletInfos;
+ this.version = version;
--- End diff --
validateCarbonDataFileVersion method is CrabonProperties I am validating the version
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90386123
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/dimension/CompressedDimensionChunkFileBasedReader2.java ---
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.carbon.datastore.chunk.reader.dimension;
--- End diff --
Ok I will move and rename all the classes as per your suggestion
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #265: [CARBONDATA-458]Improving First time query ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/265
http://136.243.101.176:8080/job/ApacheCarbonManualPRBuilder/711/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90399963
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -814,85 +810,86 @@
* Rocord size in case of compaction.
*/
public static final int COMPACTION_INMEMORY_RECORD_SIZE = 120000;
-
- /**
- * If the level 2 compaction is done in minor then new compacted segment will end with .2
- */
- public static String LEVEL2_COMPACTION_INDEX = ".2";
-
- /**
- * Indicates compaction
- */
- public static String COMPACTION_KEY_WORD = "COMPACTION";
-
/**
* hdfs temporary directory key
*/
public static final String HDFS_TEMP_LOCATION = "hadoop.tmp.dir";
-
/**
* zookeeper url key
*/
public static final String ZOOKEEPER_URL = "spark.deploy.zookeeper.url";
-
/**
* configure the minimum blocklet size eligible for blocklet distribution
*/
public static final String CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE =
"carbon.blockletdistribution.min.blocklet.size";
-
/**
* default blocklet size eligible for blocklet distribution
*/
public static final int DEFAULT_CARBON_BLOCKLETDISTRIBUTION_MIN_REQUIRED_SIZE = 2;
-
+ /**
+ * This batch size is used to send rows from load step to another step in batches.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE = "DATA_LOAD_BATCH_SIZE";
+ /**
+ * Default size of data load batch size.
+ */
+ public static final String DATA_LOAD_BATCH_SIZE_DEFAULT = "1000";
+ /**
+ * carbon data file version property
+ */
+ public static final String CARBON_DATA_FILE_VERSION = "carbon.data.file.version";
+ /**
+ * current data file version
+ */
+ public static final short CARBON_DATA_FILE_CURRENT_VERSION = 2;
--- End diff --
change 2 to enum also
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90246051
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/dimension/CompressedDimensionChunkFileBasedReader2.java ---
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.carbon.datastore.chunk.reader.dimension;
+
+import java.util.List;
+
+import org.apache.carbondata.core.carbon.datastore.chunk.DimensionChunkAttributes;
+import org.apache.carbondata.core.carbon.datastore.chunk.DimensionColumnDataChunk;
+import org.apache.carbondata.core.carbon.datastore.chunk.impl.ColumnGroupDimensionDataChunk;
+import org.apache.carbondata.core.carbon.datastore.chunk.impl.FixedLengthDimensionDataChunk;
+import org.apache.carbondata.core.carbon.datastore.chunk.impl.VariableLengthDimensionDataChunk;
+import org.apache.carbondata.core.carbon.metadata.blocklet.BlockletInfo;
+import org.apache.carbondata.core.datastorage.store.FileHolder;
+import org.apache.carbondata.core.datastorage.store.columnar.UnBlockIndexer;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.format.DataChunk2;
+import org.apache.carbondata.format.Encoding;
+
+/**
+ * Compressed dimension chunk reader class for version 2
+ */
+public class CompressedDimensionChunkFileBasedReader2 extends AbstractChunkReader {
--- End diff --
rename to `CompressedDimensionChunkFileBasedReaderV2`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90395244
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/measure/AbstractMeasureChunkReader.java ---
@@ -18,58 +18,27 @@
*/
package org.apache.carbondata.core.carbon.datastore.chunk.reader.measure;
-import java.util.List;
-
import org.apache.carbondata.core.carbon.datastore.chunk.reader.MeasureColumnChunkReader;
-import org.apache.carbondata.core.carbon.metadata.blocklet.datachunk.DataChunk;
-import org.apache.carbondata.core.datastorage.store.compression.ValueCompressionModel;
-import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder;
-import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder.UnCompressValue;
/**
* Measure block reader abstract class
*/
public abstract class AbstractMeasureChunkReader implements MeasureColumnChunkReader {
--- End diff --
Currently it is not implementing any method but in future it may have some common method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90249159
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/measure/CompressedMeasureChunkFileReader2.java ---
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.carbondata.core.carbon.datastore.chunk.reader.measure;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+
+import org.apache.carbondata.core.carbon.datastore.chunk.MeasureColumnDataChunk;
+import org.apache.carbondata.core.carbon.metadata.blocklet.BlockletInfo;
+import org.apache.carbondata.core.carbon.metadata.blocklet.datachunk.PresenceMeta;
+import org.apache.carbondata.core.datastorage.store.FileHolder;
+import org.apache.carbondata.core.datastorage.store.compression.SnappyCompression.SnappyByteCompression;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressionModel;
+import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder.UnCompressValue;
+import org.apache.carbondata.core.datastorage.store.dataholder.CarbonReadDataHolder;
+import org.apache.carbondata.core.metadata.ValueEncoderMeta;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.format.DataChunk2;
+
+/**
+ * Class to read the measure column data for version 2
+ */
+public class CompressedMeasureChunkFileReader2 extends AbstractMeasureChunkReader {
--- End diff --
please rename all V2 class to end with V2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90244822
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/block/TableBlockInfo.java ---
@@ -81,13 +84,14 @@ public TableBlockInfo(String filePath, long blockOffset, String segmentId, Strin
* @param blockletInfos
*/
public TableBlockInfo(String filePath, long blockOffset, String segmentId, String[] locations,
- long blockLength, BlockletInfos blockletInfos) {
+ long blockLength, BlockletInfos blockletInfos, short version) {
this.filePath = FileFactory.getUpdatedFilePath(filePath);
this.blockOffset = blockOffset;
this.segmentId = segmentId;
this.locations = locations;
this.blockLength = blockLength;
this.blockletInfos = blockletInfos;
+ this.version = version;
--- End diff --
somewhere should check the validity of the version, where it is checking?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #265: [CARBONDATA-458]Improving First time...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/265#discussion_r90249003
--- Diff: core/src/main/java/org/apache/carbondata/core/carbon/datastore/chunk/reader/measure/AbstractMeasureChunkReader.java ---
@@ -18,58 +18,27 @@
*/
package org.apache.carbondata.core.carbon.datastore.chunk.reader.measure;
-import java.util.List;
-
import org.apache.carbondata.core.carbon.datastore.chunk.reader.MeasureColumnChunkReader;
-import org.apache.carbondata.core.carbon.metadata.blocklet.datachunk.DataChunk;
-import org.apache.carbondata.core.datastorage.store.compression.ValueCompressionModel;
-import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder;
-import org.apache.carbondata.core.datastorage.store.compression.ValueCompressonHolder.UnCompressValue;
/**
* Measure block reader abstract class
*/
public abstract class AbstractMeasureChunkReader implements MeasureColumnChunkReader {
--- End diff --
It seems this class is not implementing any interface function of `MeasureColumnChunkReader`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---