You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ra...@apache.org on 2019/10/04 02:20:22 UTC
[carbondata] branch branch-1.6 updated (72169e5 -> be9580b)
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a change to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git.
from 72169e5 [maven-release-plugin] prepare for next development iteration
new 56879f7 [CARBONDATA-3480] Fixed unnecessary refresh for table by removing modified mdt file
new 7571b99 [CARBONDATA-3494]Fix NullPointerException in drop table and Correct the document formatting
new 90b6c64 [CARBONDATA-3493] Initialize Profiler in CarbonEnv
new 019c777 [CARBONDATA-3466] Fix NPE for carboncli command
new 6c9bbfe [CARBONDATA-3491] Return updated/deleted rows count when execute update/delete sql
new 129b163 [CARBONDATA-3509] Support disable query prefetch by configuration
new dff8ab3 [HOTFIX] Remove hive-service from carbondata assembly jar
new 1943dda [CARBONDATA-3499] Fix insert failure with customFileProvider
new 75e207c [CARBONDATA-3502] Select query with UDF having Match expression inside IN expression Fails
new 2328707 [CARBONDATA-3505] Drop database cascade fix
new 99e0c7c [CARBONDATA-3497] Support to write long string for streaming table
new 41ae280 [CARBONDATA-3513] fix 'taskNo' exceeding Long.MAX_VALUE issue when execute major compaction
new 6f90b28 [CARBONDATA-3452] dictionary include udf handle all the scenarios
new 3aea196 [CARBONDATA-3495] Fix Insert into Complex data type of Binary failure with Carbon & SparkFileFormat
new d509cd1 [CARBONDATA-3507] Fix Create Table As Select Failure in Spark-2.3
new 0e2d3e2 [CARBONDATA-3508] Support CG datamap pruning fallback while querying
new 631941b [HOTFIX] Fix NPE on windows
new ef26a4a [CARBONDATA-3506]Fix alter table failures on parition table with hive.metastore.disallow.incompatible.col.type.changes as true
new f750b6f [CARBONDATA-3515] Limit local dictionary size to 16MB and allow configuration.
new 204f290 [HOTFIX] fix missing quotation marks in datamap doc
new 43a8086 [CARBONDATA-3454] optimized index server output for count(*)
new 8ffbc1d [HOTFIX] fix incorrect word in index-server doc
new 9e46647 [CARBONDATA-3489] Optimized the comparator instances in sort
new 3368483 [HOTFIX] Remove duplicate case for BYTE_ARRAY
new ab86705 [DOC] Update doc for alter sort_columns
new de81b38 [HOTFIX] Fix wrong min/max index of measure
new 57309d7 [CARBONDATA-3473] Fix data size calcution of the last column in CarbonCli
new 21bbc4a [CARBONDATA-3520] CTAS should fail if select query contains duplicate columns
new 85dc030 [HOTFIX]Update Documentation for MV datamap
new 49e9ea3 [CARBONDATA-3523] Store data file size into index file
new 9342545 [CARBONDATA-3527] Fix 'String length cannot exceed 32000 characters' issue when load data with 'GLOBAL_SORT' from csv files which include big complex type data
new 00d2fe9 [CARBONDATA-3501] Fix update table with varchar column
new be9580b [CARBONDATA-3526]Fix cache issue during update and query
The 33 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
README.md | 1 +
assembly/pom.xml | 1 +
.../core/constants/CarbonCommonConstants.java | 20 +
.../carbondata/core/datamap/DataMapFilter.java | 47 ++
.../apache/carbondata/core/datamap/DataMapJob.java | 2 +
.../core/datamap/DataMapStoreManager.java | 21 +-
.../carbondata/core/datamap/DataMapUtil.java | 13 +-
.../core/datamap/DistributableDataMapFormat.java | 34 +-
.../core/datastore/block/TableBlockInfo.java | 13 +
.../datastore/impl/DefaultFileTypeProvider.java | 31 +-
.../core/datastore/page/ComplexColumnPage.java | 1 +
.../core/indexstore/BlockletDataMapIndexStore.java | 2 +-
.../core/indexstore/ExtendedBlocklet.java | 68 ++-
.../core/indexstore/ExtendedBlockletWrapper.java | 27 +-
.../ExtendedBlockletWrapperContainer.java | 19 +-
.../dictionaryholder/MapBasedDictionaryStore.java | 16 +-
.../carbondata/core/metadata/CarbonMetadata.java | 9 +
.../core/metadata/index/BlockIndexInfo.java | 18 +
.../core/metadata/schema/table/CarbonTable.java | 4 +-
.../core/metadata/schema/table/TableSchema.java | 4 +
.../scan/executor/impl/AbstractQueryExecutor.java | 2 +
.../carbondata/core/scan/model/QueryModel.java | 4 +-
.../statusmanager/SegmentUpdateStatusManager.java | 32 +-
.../core/util/AbstractDataFileFooterConverter.java | 3 +
.../carbondata/core/util/BlockletDataMapUtil.java | 17 +-
.../carbondata/core/util/CarbonMetadataUtil.java | 65 +--
.../carbondata/core/util/CarbonProperties.java | 59 ++
.../apache/carbondata/core/util/CarbonUtil.java | 1 -
.../apache/carbondata/core/util/DataTypeUtil.java | 7 +-
.../core/util/comparator/Comparator.java | 49 +-
.../core/metadata/CarbonMetadataTest.java | 7 +-
.../ThriftWrapperSchemaConverterImplTest.java | 4 +-
.../metadata/schema/table/CarbonTableTest.java | 8 +-
.../table/CarbonTableWithComplexTypesTest.java | 6 +-
.../datamap/bloom/BloomCoarseGrainDataMap.java | 2 +-
docs/configuration-parameters.md | 2 +
docs/datamap/datamap-management.md | 2 +-
docs/datamap/mv-datamap-guide.md | 2 +
docs/ddl-of-carbondata.md | 2 +-
docs/index-server.md | 8 +-
.../carbondata/hadoop/api/CarbonInputFormat.java | 82 ++-
.../hadoop/api/CarbonTableInputFormat.java | 22 +-
.../hadoop/api/CarbonTableOutputFormat.java | 5 +-
.../hadoop/stream/StreamRecordReader.java | 19 +-
.../src/test/resources/complexbinary.csv | 3 +
.../src/test/resources/complexdata3.csv | 10 +
.../resources/streamSample_with_long_string.csv | 6 +
.../complexType/TestComplexDataType.scala | 166 ++++++
.../TestAlterTableSortColumnsProperty.scala | 1 -
.../createTable/TestCreateTableAsSelect.scala | 37 ++
.../dblocation/DBLocationCarbonTableTestCase.scala | 25 -
.../testsuite/iud/DeleteCarbonTableTestCase.scala | 19 +
.../testsuite/iud/UpdateCarbonTableTestCase.scala | 33 ++
.../longstring/VarcharDataTypesBasicTestCase.scala | 10 +
.../StandardPartitionTableQueryTestCase.scala | 29 +-
.../scala/org/apache/carbondata/spark/KeyVal.scala | 10 +-
.../spark/rdd/NewCarbonDataLoadRDD.scala | 6 +-
.../carbondata/spark/util/CarbonScalaUtil.scala | 4 +-
.../spark/sql/catalyst/CarbonDDLSqlParser.scala | 2 +
.../streaming/CarbonAppendableStreamSink.scala | 19 +-
.../apache/spark/util/CarbonReflectionUtils.scala | 16 +
.../converter/SparkDataTypeConverterImpl.java | 6 +-
.../SparkCarbonDataSourceBinaryTest.scala | 88 +++
.../apache/spark/sql/hive/CarbonSessionUtil.scala | 6 +-
.../carbondata/indexserver/DataMapJobs.scala | 15 +-
.../indexserver/DistributedCountRDD.scala | 111 ++++
.../indexserver/DistributedPruneRDD.scala | 29 +-
.../indexserver/DistributedRDDUtils.scala | 13 +
.../carbondata/indexserver/IndexServer.scala | 27 +-
.../apache/spark/sql/CarbonCatalystOperators.scala | 6 +-
.../scala/org/apache/spark/sql/CarbonEnv.scala | 72 ++-
.../scala/org/apache/spark/sql/CarbonSession.scala | 2 -
.../command/cache/CarbonDropCacheCommand.scala | 8 +-
.../command/cache/CarbonShowCacheCommand.scala | 4 +-
.../command/datamap/CarbonDropDataMapCommand.scala | 1 -
.../command/management/CarbonCliCommand.scala | 2 +-
.../command/management/CarbonLoadDataCommand.scala | 2 +-
.../management/RefreshCarbonTableCommand.scala | 2 -
.../mutation/CarbonProjectForDeleteCommand.scala | 21 +-
.../mutation/CarbonProjectForUpdateCommand.scala | 19 +-
.../command/mutation/DeleteExecution.scala | 27 +-
.../command/mutation/HorizontalCompaction.scala | 9 +-
.../CarbonAlterTableDropPartitionCommand.scala | 12 +-
.../CarbonAlterTableSplitPartitionCommand.scala | 3 -
.../command/preaaggregate/PreAggregateUtil.scala | 19 +-
.../schema/CarbonAlterTableAddColumnCommand.scala | 20 +-
...nAlterTableColRenameDataTypeChangeCommand.scala | 36 +-
.../schema/CarbonAlterTableDropColumnCommand.scala | 35 +-
.../command/table/CarbonDropTableCommand.scala | 13 +
.../spark/sql/execution/strategy/DDLStrategy.scala | 4 +-
.../spark/sql/hive/CarbonAnalysisRules.scala | 16 +-
.../spark/sql/hive/CarbonFileMetastore.scala | 429 ++++++--------
.../spark/sql/hive/CarbonHiveMetaStore.scala | 10 +-
.../apache/spark/sql/hive/CarbonMetaStore.scala | 10 +-
.../apache/spark/sql/optimizer/CarbonFilters.scala | 2 -
.../apache/spark/sql/optimizer/CarbonIUDRule.scala | 17 +-
.../spark/sql/optimizer/CarbonLateDecodeRule.scala | 141 +++--
.../spark/sql/parser/CarbonSpark2SqlParser.scala | 11 +-
.../sql/parser/CarbonSparkSqlParserUtil.scala | 23 +-
.../scala/org/apache/spark/util/CleanFiles.scala | 3 -
.../scala/org/apache/spark/util/Compaction.scala | 2 -
.../apache/spark/util/DeleteSegmentByDate.scala | 2 -
.../org/apache/spark/util/DeleteSegmentById.scala | 2 -
.../scala/org/apache/spark/util/TableLoader.scala | 2 -
.../apache/spark/sql/hive/CarbonSessionState.scala | 31 +-
.../carbondata/spark/testsuite/TestCarbonCli.scala | 82 +++
.../TestStreamingTableWithLongString.scala | 649 +++++++++++++++++++++
.../carbondata/query/SubQueryJoinTestSuite.scala | 19 +
.../AlterTableColumnRenameTestCase.scala | 4 +-
.../processing/datatypes/PrimitiveDataType.java | 3 +
.../processing/merger/AbstractResultProcessor.java | 2 +-
.../store/writer/AbstractFactDataWriter.java | 5 +-
.../writer/v3/CarbonFactDataWriterImplV3.java | 18 +-
.../processing/util/CarbonDataProcessorUtil.java | 5 +-
.../partition/impl/RawRowComparatorTest.java | 142 +++++
.../IntermediateSortTempRowComparatorTest.java | 178 ++++++
.../sort/sortdata/NewRowComparatorTest.java | 109 ++++
.../org/apache/carbondata/sdk/file/ImageTest.java | 41 ++
.../org/apache/carbondata/sdk/file/MinMaxTest.java | 161 +++++
.../streaming/CarbonStreamRecordWriter.java | 11 +-
.../streaming/parser/CSVStreamParserImp.java | 5 +-
.../streaming/parser/CarbonStreamParser.java | 3 +-
.../streaming/parser/FieldConverter.scala | 14 +-
.../streaming/parser/RowStreamParserImp.scala | 11 +-
.../java/org/apache/carbondata/tool/CarbonCli.java | 88 +--
.../java/org/apache/carbondata/tool/DataFile.java | 32 +-
.../org/apache/carbondata/tool/CarbonCliTest.java | 6 +-
127 files changed, 3155 insertions(+), 903 deletions(-)
create mode 100644 integration/spark-common-test/src/test/resources/complexbinary.csv
create mode 100644 integration/spark-common-test/src/test/resources/complexdata3.csv
create mode 100644 integration/spark-common-test/src/test/resources/streamSample_with_long_string.csv
create mode 100644 integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedCountRDD.scala
create mode 100644 integration/spark2/src/test/scala/org/apache/carbondata/spark/testsuite/TestCarbonCli.scala
create mode 100644 integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithLongString.scala
create mode 100644 processing/src/test/java/org/apache/carbondata/processing/loading/partition/impl/RawRowComparatorTest.java
create mode 100644 processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/IntermediateSortTempRowComparatorTest.java
create mode 100644 processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/NewRowComparatorTest.java
create mode 100644 store/sdk/src/test/java/org/apache/carbondata/sdk/file/MinMaxTest.java
[carbondata] 14/33: [CARBONDATA-3495] Fix Insert into Complex data
type of Binary failure with Carbon & SparkFileFormat
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 3aea196715a9954e38dccdc50824e8f6a0a75de3
Author: Indhumathi27 <in...@gmail.com>
AuthorDate: Thu Aug 22 16:01:49 2019 +0530
[CARBONDATA-3495] Fix Insert into Complex data type of Binary
failure with Carbon & SparkFileFormat
Problem:
Insert into Complex data type(Array/Struct/Map) of binary data
type fails with Invalid data type name, because Binary with
complex data types is not handled
Solution:
Handle Binary data type to work with complex data types
This closes #3361
---
.../core/datastore/page/ComplexColumnPage.java | 1 +
.../apache/carbondata/core/util/DataTypeUtil.java | 3 +
.../src/test/resources/complexbinary.csv | 3 +
.../complexType/TestComplexDataType.scala | 114 +++++++++++++++++++++
.../spark/sql/catalyst/CarbonDDLSqlParser.scala | 2 +
.../SparkCarbonDataSourceBinaryTest.scala | 88 ++++++++++++++++
.../processing/datatypes/PrimitiveDataType.java | 3 +
.../org/apache/carbondata/sdk/file/ImageTest.java | 41 ++++++++
8 files changed, 255 insertions(+)
diff --git a/core/src/main/java/org/apache/carbondata/core/datastore/page/ComplexColumnPage.java b/core/src/main/java/org/apache/carbondata/core/datastore/page/ComplexColumnPage.java
index 921ae50..c4f8849 100644
--- a/core/src/main/java/org/apache/carbondata/core/datastore/page/ComplexColumnPage.java
+++ b/core/src/main/java/org/apache/carbondata/core/datastore/page/ComplexColumnPage.java
@@ -124,6 +124,7 @@ public class ComplexColumnPage {
DataTypes.isMapType(dataType) ||
(dataType == DataTypes.STRING) ||
(dataType == DataTypes.VARCHAR) ||
+ (dataType == DataTypes.BINARY) ||
(dataType == DataTypes.DATE) ||
DataTypes.isDecimal(dataType))))) {
// For all these above condition the ColumnPage should be Taken as BYTE_ARRAY
diff --git a/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java b/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
index 9aea579..adb63cd 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
@@ -530,6 +530,7 @@ public final class DataTypeUtil {
public static boolean isFixedSizeDataType(DataType dataType) {
if (dataType == DataTypes.STRING ||
dataType == DataTypes.VARCHAR ||
+ dataType == DataTypes.BINARY ||
DataTypes.isDecimal(dataType)) {
return false;
} else {
@@ -1019,6 +1020,8 @@ public final class DataTypeUtil {
return DataTypes.BYTE_ARRAY;
} else if (DataTypes.BYTE_ARRAY.getName().equalsIgnoreCase(name)) {
return DataTypes.BYTE_ARRAY;
+ } else if (DataTypes.BINARY.getName().equalsIgnoreCase(name)) {
+ return DataTypes.BINARY;
} else if (name.equalsIgnoreCase("decimal")) {
return DataTypes.createDefaultDecimalType();
} else if (name.equalsIgnoreCase("array")) {
diff --git a/integration/spark-common-test/src/test/resources/complexbinary.csv b/integration/spark-common-test/src/test/resources/complexbinary.csv
new file mode 100644
index 0000000..3870f5f
--- /dev/null
+++ b/integration/spark-common-test/src/test/resources/complexbinary.csv
@@ -0,0 +1,3 @@
+1,true,abc,binary1$binary2,binary1,1&binary1
+2,false,abcd,binary11$binary12,binary11,1&binary2
+3,true,abcde,binary13$binary13,binary13,1&binary3
\ No newline at end of file
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
index b5f77c2..9d6b4d1 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
@@ -1013,4 +1013,118 @@ class TestComplexDataType extends QueryTest with BeforeAndAfterAll {
checkAnswer(sql("select id,name,structField.intval,name,structField.stringval from table1"),Seq(Row(null,"aaa",23,"aaa","bb")))
}
+ test("test array of binary data type") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ sql("create table if not exists hive_table(id int, label boolean, name string," +
+ "binaryField array<binary>, autoLabel boolean) row format delimited fields terminated by ','")
+ sql("insert into hive_table values(1,true,'abc',array('binary'),false)")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField array<binary>, autoLabel boolean) stored by 'carbondata'")
+ sql("insert into carbon_table values(1,true,'abc',array('binary'),false)")
+ checkAnswer(sql("SELECT binaryField[0] FROM carbon_table"),
+ sql("SELECT binaryField[0] FROM hive_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ }
+
+ test("test struct of binary data type") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ sql("create table if not exists parquet_table(id int, label boolean, name string," +
+ "binaryField struct<b:binary>, autoLabel boolean) using parquet")
+ sql("insert into parquet_table values(1,true,'abc',named_struct('b','binary'),false)")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField struct<b:binary>, autoLabel boolean) stored by 'carbondata'")
+ sql("insert into carbon_table values(1,true,'abc',named_struct('b','binary'),false)")
+ sql("SELECT binaryField.b FROM carbon_table").show(false)
+ checkAnswer(sql("SELECT binaryField.b FROM carbon_table"),
+ sql("SELECT binaryField.b FROM parquet_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ }
+
+ test("test map of binary data type") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ sql("create table if not exists hive_table(id int, label boolean, name string," +
+ "binaryField map<int, binary>, autoLabel boolean) row format delimited fields terminated by ','")
+ sql("insert into hive_table values(1,true,'abc',map(1,'binary'),false)")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField map<int, binary>, autoLabel boolean) stored by 'carbondata'")
+ sql("insert into carbon_table values(1,true,'abc',map(1,'binary'),false)")
+ checkAnswer(sql("SELECT binaryField[1] FROM carbon_table"),
+ sql("SELECT binaryField[1] FROM hive_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ }
+
+ test("test map of array and struct binary data type") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ sql("create table if not exists parquet_table(id int, label boolean, name string," +
+ "binaryField1 map<int, array<binary>>, binaryField2 map<int, struct<b:binary>> ) " +
+ "using parquet")
+ sql("insert into parquet_table values(1,true,'abc',map(1,array('binary')),map(1," +
+ "named_struct('b','binary')))")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField1 map<int, array<binary>>, binaryField2 map<int, struct<b:binary>> ) " +
+ "stored by 'carbondata'")
+ sql("insert into carbon_table values(1,true,'abc',map(1,array('binary')),map(1," +
+ "named_struct('b','binary')))")
+ checkAnswer(sql("SELECT binaryField1[1][1] FROM carbon_table"),
+ sql("SELECT binaryField1[1][1] FROM parquet_table"))
+ checkAnswer(sql("SELECT binaryField2[1].b FROM carbon_table"),
+ sql("SELECT binaryField2[1].b FROM parquet_table"))
+ sql("drop table if exists hive_table")
+ sql("drop table if exists carbon_table")
+ }
+
+ test("test of array of struct and struct of array of binary data type") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ sql("create table if not exists hive_table(id int, label boolean, name string," +
+ "binaryField1 array<struct<b1:binary>>, binaryField2 struct<b2:array<binary>> ) " +
+ "row format delimited fields terminated by ','")
+ sql("insert into hive_table values(1,true,'abc',array(named_struct('b1','binary'))," +
+ "named_struct('b2',array('binary')))")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField1 array<struct<b1:binary>>, binaryField2 struct<b2:array<binary>> ) " +
+ "stored by 'carbondata'")
+ sql("insert into carbon_table values(1,true,'abc',array(named_struct('b1','binary'))," +
+ "named_struct('b2',array('binary')))")
+ checkAnswer(sql("SELECT binaryField1[1].b1 FROM carbon_table"),
+ sql("SELECT binaryField1[1].b1 FROM hive_table"))
+ checkAnswer(sql("SELECT binaryField2.b2[0] FROM carbon_table"),
+ sql("SELECT binaryField2.b2[0] FROM hive_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ }
+
+ test("test dataload to complex of binary type column using load ddl ") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ sql("create table if not exists hive_table(id int, label boolean, name string," +
+ "binaryField1 array<binary>, binaryField2 struct<b2:binary>, binaryField3 map<int," +
+ "binary>) row format delimited fields terminated by ','")
+ sql(
+ "insert into hive_table values(1,true,'abc',array('binary1','binary2'), named_struct('b2'," +
+ "'binary1'), map(1,'binary1'))")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField1 array<binary>, binaryField2 struct<b2:binary>, binaryField3 map<int,binary>) " +
+ "stored by 'carbondata'")
+ sql(
+ "load data inpath '" + resourcesPath + "/complexbinary.csv' into table carbon_table options" +
+ "('delimiter'=',', 'quotechar'='\\','fileheader'='id,label,name,binaryField1,binaryField2," +
+ "binaryField3','complex_delimiter_level_1'='$', 'complex_delimiter_level_2'='&')")
+ checkAnswer(sql("SELECT binaryField1[0] FROM carbon_table where id=1"),
+ sql("SELECT binaryField1[0] FROM hive_table where id=1"))
+ checkAnswer(sql("SELECT binaryField2.b2 FROM carbon_table where id=1"),
+ sql("SELECT binaryField2.b2 FROM hive_table where id=1"))
+ checkAnswer(sql("SELECT binaryField3[1] FROM carbon_table where id=1"),
+ sql("SELECT binaryField3[1] FROM hive_table where id=1"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists hive_table")
+ }
+
}
diff --git a/integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala b/integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
index 3d3b89d..c331532 100644
--- a/integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
+++ b/integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
@@ -1511,6 +1511,8 @@ abstract class CarbonDDLSqlParser extends AbstractCarbonSparkSQLParser {
field.dataType.getOrElse("NIL") match {
case "String" => Field(parentName + "." + field.column, Some("String"),
Some(parentName + "." + field.name.getOrElse(None)), Some(null), parentName)
+ case "binary" => Field(parentName + "." + field.column, Some("Binary"),
+ Some(parentName + "." + field.name.getOrElse(None)), Some(null), parentName)
case "SmallInt" => Field(parentName + "." + field.column, Some("SmallInt"),
Some(parentName + "." + field.name.getOrElse(None)), Some(null), parentName)
case "Integer" => Field(parentName + "." + field.column, Some("Integer"),
diff --git a/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceBinaryTest.scala b/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceBinaryTest.scala
index 9ebf54f..9a3a9f0 100644
--- a/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceBinaryTest.scala
+++ b/integration/spark-datasource/src/test/scala/org/apache/spark/sql/carbondata/datasource/SparkCarbonDataSourceBinaryTest.scala
@@ -17,6 +17,7 @@
package org.apache.spark.sql.carbondata.datasource
import java.io.File
+
import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.util.CarbonProperties
import org.apache.carbondata.sdk.util.BinaryUtil
@@ -24,6 +25,7 @@ import org.apache.commons.codec.binary.{Base64, Hex}
import org.apache.commons.io.FileUtils
import org.apache.spark.sql.Row
import org.apache.spark.sql.carbondata.datasource.TestUtil._
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
import org.apache.spark.util.SparkUtil
import org.scalatest.{BeforeAndAfterAll, FunSuite}
@@ -635,4 +637,90 @@ class SparkCarbonDataSourceBinaryTest extends FunSuite with BeforeAndAfterAll {
assert(exception.getMessage.contains("Operation not allowed: DELETE FROM"))
}
+ test("test array of binary data type with sparkfileformat ") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField array<binary>, autoLabel boolean) using carbon")
+ sql("insert into carbon_table values(1,true,'abc',array('binary'),false)")
+ sql("create table if not exists parquet_table(id int, label boolean, name string," +
+ "binaryField array<binary>, autoLabel boolean) using parquet")
+ sql("insert into parquet_table values(1,true,'abc',array('binary'),false)")
+ checkAnswer(sql("SELECT binaryField[0] FROM carbon_table"),
+ sql("SELECT binaryField[0] FROM parquet_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ }
+
+ test("test struct of binary data type with sparkfileformat ") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField struct<b:binary>, autoLabel boolean) using carbon")
+ sql("insert into carbon_table values(1,true,'abc',named_struct('b','binary'),false)")
+ sql("create table if not exists parquet_table(id int, label boolean, name string," +
+ "binaryField struct<b:binary>, autoLabel boolean) using parquet")
+ sql("insert into parquet_table values(1,true,'abc',named_struct('b','binary'),false)")
+ checkAnswer(sql("SELECT binaryField.b FROM carbon_table"),
+ sql("SELECT binaryField.b FROM parquet_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ }
+
+ test("test map of binary data type with sparkfileformat") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ sql("create table if not exists parquet_table(id int, label boolean, name string," +
+ "binaryField map<int, binary>, autoLabel boolean) using parquet")
+ sql("insert into parquet_table values(1,true,'abc',map(1,'binary'),false)")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField map<int, binary>, autoLabel boolean) using carbon")
+ sql("insert into carbon_table values(1,true,'abc',map(1,'binary'),false)")
+ checkAnswer(sql("SELECT binaryField[1] FROM carbon_table"),
+ sql("SELECT binaryField[1] FROM parquet_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ }
+
+ test("test map of array and struct binary data type with sparkfileformat") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ sql("create table if not exists parquet_table(id int, label boolean, name string," +
+ "binaryField1 map<int, array<binary>>, binaryField2 map<int, struct<b:binary>> ) " +
+ "using parquet")
+ sql("insert into parquet_table values(1,true,'abc',map(1,array('binary')),map(1," +
+ "named_struct('b','binary')))")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField1 map<int, array<binary>>, binaryField2 map<int, struct<b:binary>> ) " +
+ "using carbon")
+ sql("insert into carbon_table values(1,true,'abc',map(1,array('binary')),map(1," +
+ "named_struct('b','binary')))")
+ checkAnswer(sql("SELECT binaryField1[1][1] FROM carbon_table"),
+ sql("SELECT binaryField1[1][1] FROM parquet_table"))
+ checkAnswer(sql("SELECT binaryField2[1].b FROM carbon_table"),
+ sql("SELECT binaryField2[1].b FROM parquet_table"))
+ sql("drop table if exists carbon_table")
+ }
+
+ test("test of array of struct and struct of array of binary data type with sparkfileformat") {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ sql("create table if not exists parquet_table(id int, label boolean, name string," +
+ "binaryField1 array<struct<b1:binary>>, binaryField2 struct<b2:array<binary>> ) " +
+ "using parquet")
+ sql("insert into parquet_table values(1,true,'abc',array(named_struct('b1','binary'))," +
+ "named_struct('b2',array('binary')))")
+ sql("create table if not exists carbon_table(id int, label boolean, name string," +
+ "binaryField1 array<struct<b1:binary>>, binaryField2 struct<b2:array<binary>> ) " +
+ "using carbon")
+ sql("insert into carbon_table values(1,true,'abc',array(named_struct('b1','binary'))," +
+ "named_struct('b2',array('binary')))")
+ checkAnswer(sql("SELECT binaryField1[1].b1 FROM carbon_table"),
+ sql("SELECT binaryField1[1].b1 FROM parquet_table"))
+ checkAnswer(sql("SELECT binaryField2.b2[0] FROM carbon_table"),
+ sql("SELECT binaryField2.b2[0] FROM parquet_table"))
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists parquet_table")
+ }
+
}
diff --git a/processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java b/processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
index 200a9f6..9504974 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/datatypes/PrimitiveDataType.java
@@ -378,6 +378,9 @@ public class PrimitiveDataType implements GenericDataType<Object> {
} else {
value = ByteUtil.toXorBytes(Long.parseLong(parsedValue));
}
+ } else if (this.carbonDimension.getDataType().equals(DataTypes.BINARY)) {
+ value = DataTypeUtil.getBytesDataDataTypeForNoDictionaryColumn(input,
+ this.carbonDimension.getDataType());
} else {
value = DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(parsedValue,
this.carbonDimension.getDataType(), dateFormat);
diff --git a/store/sdk/src/test/java/org/apache/carbondata/sdk/file/ImageTest.java b/store/sdk/src/test/java/org/apache/carbondata/sdk/file/ImageTest.java
index 30a881b..6f90155 100644
--- a/store/sdk/src/test/java/org/apache/carbondata/sdk/file/ImageTest.java
+++ b/store/sdk/src/test/java/org/apache/carbondata/sdk/file/ImageTest.java
@@ -22,6 +22,7 @@ import junit.framework.TestCase;
import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.StructField;
import org.apache.carbondata.core.scan.expression.ColumnExpression;
import org.apache.carbondata.core.scan.expression.LiteralExpression;
import org.apache.carbondata.core.scan.expression.conditional.EqualToExpression;
@@ -1127,4 +1128,44 @@ public class ImageTest extends TestCase {
}
}
+ @Test public void testBinaryWithComplexType()
+ throws IOException, InvalidLoadOptionException, InterruptedException {
+ int num = 1;
+ int rows = 1;
+ String path = "./target/binary";
+ try {
+ FileUtils.deleteDirectory(new File(path));
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ Field[] fields = new Field[4];
+ fields[0] = new Field("name", DataTypes.STRING);
+ fields[1] = new Field("age", DataTypes.INT);
+ fields[2] = new Field("arrayField", DataTypes.createArrayType(DataTypes.BINARY));
+ ArrayList<StructField> structFields = new ArrayList<>();
+ structFields.add(new StructField("b", DataTypes.BINARY));
+ fields[3] = new Field("structField", DataTypes.createStructType(structFields));
+
+ // read and write image data
+ for (int j = 0; j < num; j++) {
+ CarbonWriter writer = CarbonWriter.builder().outputPath(path).withCsvInput(new Schema(fields))
+ .writtenBy("BinaryExample").withPageSizeInMb(1).build();
+
+ for (int i = 0; i < rows; i++) {
+ // write data
+ writer.write(new String[] { "robot" + (i % 10), String.valueOf(i), "binary1", "binary2" });
+ }
+ writer.close();
+ }
+ CarbonReader reader = CarbonReader.builder(path, "_temp").build();
+ while (reader.hasNext()) {
+ Object[] row = (Object[]) reader.readNextRow();
+ Object[] arrayResult = (Object[]) row[1];
+ Object[] structResult = (Object[]) row[2];
+ assert (new String((byte[]) arrayResult[0]).equalsIgnoreCase("binary1"));
+ assert (new String((byte[]) structResult[0]).equalsIgnoreCase("binary2"));
+ }
+ reader.close();
+ }
+
}
[carbondata] 13/33: [CARBONDATA-3452] dictionary include udf handle
all the scenarios
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 6f90b28dd3d3f008f33668884807cc63cb6b5db5
Author: ajantha-bhat <aj...@gmail.com>
AuthorDate: Wed Aug 14 20:36:13 2019 +0530
[CARBONDATA-3452] dictionary include udf handle all the scenarios
Problem: select query failure when substring on dictionary column with join.
Cause: when dictionary include is present, data type is updated to int from
string in plan attribute. so substring was unresolved on int column.
Join operation try to reference this attribute which is unresolved.
Solution: Need to handle this for all the scenarios in CarbonLateDecodeRule
This closes #3358
---
.../hadoop/api/CarbonTableOutputFormat.java | 5 +-
.../spark/sql/optimizer/CarbonLateDecodeRule.scala | 141 ++++++++++++++-------
.../carbondata/query/SubQueryJoinTestSuite.scala | 19 +++
.../processing/util/CarbonDataProcessorUtil.java | 5 +-
4 files changed, 120 insertions(+), 50 deletions(-)
diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
index 9ba5e97..16703bf 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableOutputFormat.java
@@ -19,6 +19,7 @@ package org.apache.carbondata.hadoop.api;
import java.io.IOException;
import java.util.List;
+import java.util.UUID;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
@@ -221,8 +222,8 @@ public class CarbonTableOutputFormat extends FileOutputFormat<NullWritable, Obje
return (String[]) ObjectSerializationUtil.convertStringToObject(encodedString);
}
return new String[] {
- System.getProperty("java.io.tmpdir") + "/" + System.nanoTime() + "_" + taskAttemptContext
- .getTaskAttemptID().toString() };
+ System.getProperty("java.io.tmpdir") + "/" + UUID.randomUUID().toString().replace("-", "")
+ + "_" + taskAttemptContext.getTaskAttemptID().toString() };
}
@Override
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
index 961bf11..99a8e70 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala
@@ -619,6 +619,21 @@ class CarbonLateDecodeRule extends Rule[LogicalPlan] with PredicateHelper {
}
}
+ private def needDataTypeUpdate(exp: Expression): Boolean = {
+ var needChangeDatatype: Boolean = true
+ exp.transform {
+ case attr: AttributeReference => attr
+ case a@Alias(attr: AttributeReference, _) => a
+ case others =>
+ // datatype need to change for dictionary columns if only alias
+ // or attribute ref present.
+ // If anything else present, no need to change data type.
+ needChangeDatatype = false
+ others
+ }
+ needChangeDatatype
+ }
+
private def updateTempDecoder(plan: LogicalPlan,
aliasMapOriginal: CarbonAliasDecoderRelation,
attrMap: java.util.HashMap[AttributeReferenceWrapper, CarbonDecoderRelation]):
@@ -650,44 +665,71 @@ class CarbonLateDecodeRule extends Rule[LogicalPlan] with PredicateHelper {
cd
case sort: Sort =>
val sortExprs = sort.order.map { s =>
- s.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
- }.asInstanceOf[SortOrder]
+ if (needDataTypeUpdate(s)) {
+ s.transform {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }.asInstanceOf[SortOrder]
+ } else {
+ s
+ }
}
Sort(sortExprs, sort.global, sort.child)
case agg: Aggregate if !agg.child.isInstanceOf[CarbonDictionaryCatalystDecoder] =>
val aggExps = agg.aggregateExpressions.map { aggExp =>
- aggExp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ if (needDataTypeUpdate(aggExp)) {
+ aggExp.transform {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
+ } else {
+ aggExp
}
}.asInstanceOf[Seq[NamedExpression]]
-
val grpExps = agg.groupingExpressions.map { gexp =>
- gexp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ if (needDataTypeUpdate(gexp)) {
+ gexp.transform {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
+ } else {
+ gexp
}
}
Aggregate(grpExps, aggExps, agg.child)
case expand: Expand =>
- val ex = expand.transformExpressions {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ // can't use needDataTypeUpdate here as argument is of type Expand
+ var needChangeDatatype: Boolean = true
+ expand.transformExpressions {
+ case attr: AttributeReference => attr
+ case a@Alias(attr: AttributeReference, _) => a
+ case others =>
+ // datatype need to change for dictionary columns if only alias
+ // or attribute ref present.
+ // If anything else present, no need to change data type.
+ needChangeDatatype = false
+ others
}
- // Update the datatype of literal type as per the output type, otherwise codegen fails.
- val updatedProj = ex.projections.map { projs =>
- projs.zipWithIndex.map { case(p, index) =>
- p.transform {
- case l: Literal
- if l.dataType != ex.output(index).dataType &&
- !isComplexColumn(ex.output(index), ex.child.output) =>
- Literal(l.value, ex.output(index).dataType)
+ if (needChangeDatatype) {
+ val ex = expand.transformExpressions {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
+ // Update the datatype of literal type as per the output type, otherwise codegen fails.
+ val updatedProj = ex.projections.map { projs =>
+ projs.zipWithIndex.map { case (p, index) =>
+ p.transform {
+ case l: Literal
+ if l.dataType != ex.output(index).dataType &&
+ !isComplexColumn(ex.output(index), ex.child.output) =>
+ Literal(l.value, ex.output(index).dataType)
+ }
}
}
+ Expand(updatedProj, ex.output, ex.child)
+ } else {
+ expand
}
- Expand(updatedProj, ex.output, ex.child)
case filter: Filter =>
filter
case j: Join =>
@@ -698,18 +740,7 @@ class CarbonLateDecodeRule extends Rule[LogicalPlan] with PredicateHelper {
u
case p: Project if relations.nonEmpty =>
val prExps = p.projectList.map { prExp =>
- var needChangeDatatype = true
- prExp.transform {
- case attr: AttributeReference => attr
- case a@Alias(attr: AttributeReference, _) => a
- case others =>
- // datatype need to change for dictionary columns if only alias
- // or attribute ref present.
- // If anything else present, no need to change data type.
- needChangeDatatype = false
- others
- }
- if (needChangeDatatype) {
+ if (needDataTypeUpdate(prExp)) {
prExp.transform {
case attr: AttributeReference =>
updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
@@ -721,27 +752,43 @@ class CarbonLateDecodeRule extends Rule[LogicalPlan] with PredicateHelper {
Project(prExps, p.child)
case wd: Window if relations.nonEmpty =>
val prExps = wd.output.map { prExp =>
- prExp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ if (needDataTypeUpdate(prExp)) {
+ prExp.transform {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
+ } else {
+ prExp
}
}.asInstanceOf[Seq[Attribute]]
val wdExps = wd.windowExpressions.map { gexp =>
- gexp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ if (needDataTypeUpdate(gexp)) {
+ gexp.transform {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
+ } else {
+ gexp
}
}.asInstanceOf[Seq[NamedExpression]]
val partitionSpec = wd.partitionSpec.map{ exp =>
- exp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ if (needDataTypeUpdate(exp)) {
+ exp.transform {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
+ } else {
+ exp
}
}
val orderSpec = wd.orderSpec.map { exp =>
- exp.transform {
- case attr: AttributeReference =>
- updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ if (needDataTypeUpdate(exp)) {
+ exp.transform {
+ case attr: AttributeReference =>
+ updateDataType(attr, attrMap, allAttrsNotDecode, aliasMap)
+ }
+ } else {
+ exp
}
}.asInstanceOf[Seq[SortOrder]]
Window(wdExps, partitionSpec, orderSpec, wd.child)
diff --git a/integration/spark2/src/test/scala/org/apache/spark/carbondata/query/SubQueryJoinTestSuite.scala b/integration/spark2/src/test/scala/org/apache/spark/carbondata/query/SubQueryJoinTestSuite.scala
index 635445a..4552b4f 100644
--- a/integration/spark2/src/test/scala/org/apache/spark/carbondata/query/SubQueryJoinTestSuite.scala
+++ b/integration/spark2/src/test/scala/org/apache/spark/carbondata/query/SubQueryJoinTestSuite.scala
@@ -56,4 +56,23 @@ class SubQueryJoinTestSuite extends Spark2QueryTest with BeforeAndAfterAll {
sql("drop table t1")
sql("drop table t2")
}
+
+ test("test join with dictionary include with udf") {
+ sql("drop table if exists t1")
+ sql("drop table if exists t2")
+ sql(
+ "create table t1 (m_month smallint, hs_code string, country smallint, dollar_value double, " +
+ "quantity double, unit smallint, b_country smallint, imex int, y_year smallint) stored by " +
+ "'carbondata' tblproperties('dictionary_include'='m_month,hs_code,b_country,unit,y_year," +
+ "imex', 'sort_columns'='y_year,m_month,country,b_country,imex')")
+ sql(
+ "create table t2(id bigint, hs string, hs_cn string, hs_en string) stored by 'carbondata' " +
+ "tblproperties ('dictionary_include'='id,hs,hs_cn,hs_en')")
+ checkAnswer(sql(
+ "select a.hs,count(*) tb from (select substring(hs_code,1,2) as hs,count(*) v2000 from t1 " +
+ "group by substring(hs_code,1,2),y_year) a left join t2 h on (a.hs=h.hs) group by a.hs"),
+ Seq())
+ sql("drop table if exists t1")
+ sql("drop table if exists t2")
+ }
}
diff --git a/processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java b/processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
index 9ce96d4..588d4ac 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java
@@ -94,8 +94,11 @@ public final class CarbonDataProcessorUtil {
if (dir.exists()) {
LOGGER.warn("dir already exists, skip dir creation: " + loc);
} else {
- if (!dir.mkdirs()) {
+ if (!dir.mkdirs() && !dir.exists()) {
+ // concurrent scenario mkdir may fail, so checking dir
LOGGER.error("Error occurs while creating dir: " + loc);
+ } else {
+ LOGGER.info("Successfully created dir: " + loc);
}
}
}
[carbondata] 31/33: [CARBONDATA-3527] Fix 'String length cannot
exceed 32000 characters' issue when load data with 'GLOBAL_SORT' from csv
files which include big complex type data
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 93425458a93871fd18b1c3c41da396dbb06c02c8
Author: Zhang Zhichao <44...@qq.com>
AuthorDate: Wed Sep 25 15:58:35 2019 +0800
[CARBONDATA-3527] Fix 'String length cannot exceed 32000 characters' issue when load data with 'GLOBAL_SORT' from csv files which include big complex type data
Problem:
When complex type data is used more than 32000 characters to indicate in csv file, and load data with 'GLOBAL_SORT' from these csv files, it will throw 'String length cannot exceed 32000 characters' exception.
Cause:
Use 'GLOBAL_SORT' to load data from csv files, it reads files and firstly store data in StringArrayRow, the type of all data are string, when call 'CarbonScalaUtil.getString' in 'NewRddIterator.next', it will check the length of all data and throw 'String length cannot exceed 32000 characters' exception even if it's complex type data which store as more than 32000 characters in csv files.
Solution:
In 'FieldConverter.objectToString' (called in 'CarbonScalaUtil.getString'), if the data type of field is complex type, don't check the length.
This closes #3399
---
.../src/test/resources/complexdata3.csv | 10 +++++
.../complexType/TestComplexDataType.scala | 52 ++++++++++++++++++++++
.../spark/rdd/NewCarbonDataLoadRDD.scala | 6 ++-
.../carbondata/spark/util/CarbonScalaUtil.scala | 4 +-
.../streaming/parser/FieldConverter.scala | 14 +++---
5 files changed, 79 insertions(+), 7 deletions(-)
diff --git a/integration/spark-common-test/src/test/resources/complexdata3.csv b/integration/spark-common-test/src/test/resources/complexdata3.csv
new file mode 100644
index 0000000..63cd44b
--- /dev/null
+++ b/integration/spark-common-test/src/test/resources/complexdata3.csv
@@ -0,0 +1,10 @@
+e01a1773-bd37-40be-a1de-d7e74837a281 (0551)96116063 886 0031 5 (0551)46819921 853 4 0 1568220618904 50 asp fk 2745000 1 0 0 0 0 -0.19569306\0020.10781755\002-0.06963766\002-0.06576662\002-0.17820272\002-0.01949397\0020.08014756\002-0.05287997\0020.02067086\002-0.11302640\0020.07383678\0020.07296083\0020.11693181\002-0.06988186\0020.05753217\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631518\0020.05918765\0020.07385136\002-0.05143059\002-0.19158234\0020.13839211\002 [...]
+f72ce5cb-2ea6-423b-8c1f-6dadfd6f52e7 (0551)73382297 853 0031 4 (0551)73382297 49 9 0 1568275177770 1559 asp fk 5821000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.13839212\00 [...]
+e282ecb5-9be8-4a0e-8faf-d10e535ab877 13396633307 49 0031 9 13918448986 1 7 0 1568260253193 1150 asp fk 3884000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.13839212\002-0.0826 [...]
+01e36a06-b4fd-4638-862c-2785f9e4331b 13924865616 82 0031 0 0086(021)60080162 82 6 0 1568293725356 2108 asp fk 3152000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.13839212\002 [...]
+a451790d-42f8-48e5-88f4-ba21118e63e6 13326037312 81 0031 8 (0551)17198025 852 2 0 1568294179731 2116 asp fk 1127000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.13839212\002-0 [...]
+9d26e280-4e87-4cbe-a850-5965b7c36a4b 13376907227 44 0031 9 13376907227 82 3 0 1568302365552 2332 asp fk 2043000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.13839212\002-0.082 [...]
+c2eabec9-b8a7-405b-80d9-e73692d586f4 0086(021)77426829 81 0031 8 13326037312 44 0 0 1568252700180 945 asp fk 943000 1 0 0 0 0 -0.19569306\0020.10781755\002-0.06963766\002-0.06576662\002-0.17820272\002-0.01949397\0020.08014756\002-0.05287997\0020.02067086\002-0.11302640\0020.07383678\0020.07296083\0020.11693181\002-0.06988186\0020.05753217\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631518\0020.05918765\0020.07385136\002-0.05143059\002-0.19158234\0020.13839211\002-0 [...]
+04a548aa-a103-4ffd-b72c-81b6cb2ea420 0086(021)77426829 82 0031 2 13924865616 33 0 0 1568249850352 857 asp fk 2450000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.13839212\002- [...]
+45c0ded1-c608-4a49-981d-faf720442a59 13378149447 49 0031 8 13376907227 1 5 0 1568289879606 2004 asp fk 3686000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.13839212\002-0.0826 [...]
+cff43f86-ae81-4bbc-90dd-b7de39bdda1b 0086(021)77426829 82 0031 2 0086(021)60080162 886 6 0 1568230183633 329 asp fk 1615000 1 0 0 0 0 -0.19569308\0020.10781755\002-0.06963766\002-0.06576661\002-0.17820270\002-0.01949396\0020.08014755\002-0.05287996\0020.02067086\002-0.11302640\0020.07383677\0020.07296082\0020.11693182\002-0.06988187\0020.05753216\002-0.02308202\002-0.03685183\0020.05840293\0020.03959572\002-0.01631517\0020.05918765\0020.07385137\002-0.05143059\002-0.19158235\0020.138392 [...]
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
index 9d6b4d1..8ec1420 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala
@@ -1127,4 +1127,56 @@ class TestComplexDataType extends QueryTest with BeforeAndAfterAll {
sql("drop table if exists hive_table")
}
+ test("[CARBONDATA-3527] Fix 'String length cannot exceed 32000 characters' issue when load data with 'GLOBAL_SORT' from csv files which include big complex type data") {
+ val tableName = "complexdata3_table"
+ sql(s"drop table if exists ${tableName}")
+ sql(
+ s"""
+ |CREATE TABLE IF NOT EXISTS ${tableName} (
+ | begin_time LONG,
+ | id string,
+ | phone string,
+ | other_phone string,
+ | vtl LONG,
+ | gender string,
+ | lang string,
+ | lang_dec string,
+ | phone_country string,
+ | phone_province string,
+ | phone_city string,
+ | other_phone_country string,
+ | other_phone_province string,
+ | other_phone_city string,
+ | call_type INT,
+ | begin_hhmm INT,
+ | ds string,
+ | voice_flag INT,
+ | dss string,
+ | dur LONG,
+ | modela array < array < FLOAT >>, modelb array < array < FLOAT >>, modela_pk array < array < FLOAT >>, modelb_pk array < array < FLOAT >>, modela_ms array < array < FLOAT >>, modelb_ms array < array < FLOAT >>, tl LONG,
+ | lang_sc FLOAT,
+ | nlp_sc FLOAT,
+ | create_time LONG,
+ | cdr_create_time LONG,
+ | fulltext string,
+ | tag_label string,
+ | tag_memo string,
+ | tag_listen string,
+ | tag_imp string,
+ | prop string,
+ | files string
+ | )
+ | STORED AS carbondata TBLPROPERTIES (
+ | 'SORT_COLUMNS' = 'begin_time,id,phone,other_phone,vtl,gender,lang,lang_dec,phone_country,phone_province,phone_city,other_phone_country,other_phone_province,other_phone_city,call_type,begin_hhmm,ds,voice_flag',
+ | 'SORT_SCOPE' = 'GLOBAL_SORT','LONG_STRING_COLUMNS' = 'fulltext,files')""".stripMargin)
+ sql(s"""LOAD DATA inpath '${resourcesPath}/complexdata3.csv' INTO table ${tableName}
+ options('DELIMITER'='\t','QUOTECHAR'='"','COMMENTCHAR'='#','HEADER'='false',
+ 'FILEHEADER'='id,phone,phone_country,phone_province,phone_city,other_phone,other_phone_country,other_phone_province,other_phone_city,call_type,begin_time,begin_hhmm,ds,dss,dur,voice_flag,modela,modelb,modela_pk,modelb_pk,modela_ms,modelb_ms,lang,lang_dec,lang_sc,gender,nlp_sc,tl,vtl,create_time,cdr_create_time,fulltext,tag_label,tag_memo,tag_listen,tag_imp,prop,files',
+ 'MULTILINE'='true','ESCAPECHAR'='\','COMPLEX_DELIMITER_LEVEL_1'='\\001','COMPLEX_DELIMITER_LEVEL_2'='\\002',
+ 'SINGLE_PASS'='TRUE')""")
+ checkAnswer(sql(s"select count(1) from ${tableName}"), Seq(Row(10)))
+ checkAnswer(sql(s"select modela[0][0], modela_ms[0][1] from ${tableName} where id = 'e01a1773-bd37-40be-a1de-d7e74837a281'"),
+ Seq(Row(0.0, 0.10781755)))
+ sql(s"drop table if exists ${tableName}")
+ }
}
diff --git a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
index ac8224e..ce60a55 100644
--- a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
+++ b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
@@ -348,6 +348,9 @@ class NewRddIterator(rddIter: Iterator[Row],
private val isVarcharTypeMapping =
carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.getCreateOrderColumn(
carbonLoadModel.getTableName).asScala.map(_.getDataType == DataTypes.VARCHAR)
+ private val isComplexTypeMapping =
+ carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable.getCreateOrderColumn(
+ carbonLoadModel.getTableName).asScala.map(_.isComplex())
def hasNext: Boolean = rddIter.hasNext
def next: Array[AnyRef] = {
@@ -356,7 +359,8 @@ class NewRddIterator(rddIter: Iterator[Row],
for (i <- 0 until columns.length) {
columns(i) = CarbonScalaUtil.getString(row.get(i), serializationNullFormat,
complexDelimiters, timeStampFormat, dateFormat,
- isVarcharType = i < isVarcharTypeMapping.size && isVarcharTypeMapping(i))
+ isVarcharType = i < isVarcharTypeMapping.size && isVarcharTypeMapping(i),
+ isComplexType = i < isComplexTypeMapping.size && isComplexTypeMapping(i))
}
columns
}
diff --git a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
index b0af2ea..d94c5d7 100644
--- a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
+++ b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
@@ -66,9 +66,11 @@ object CarbonScalaUtil {
timeStampFormat: SimpleDateFormat,
dateFormat: SimpleDateFormat,
isVarcharType: Boolean = false,
+ isComplexType: Boolean = false,
level: Int = 0): String = {
FieldConverter.objectToString(value, serializationNullFormat, complexDelimiters,
- timeStampFormat, dateFormat, isVarcharType = isVarcharType, level)
+ timeStampFormat, dateFormat, isVarcharType = isVarcharType, isComplexType = isComplexType,
+ level)
}
/**
diff --git a/streaming/src/main/scala/org/apache/carbondata/streaming/parser/FieldConverter.scala b/streaming/src/main/scala/org/apache/carbondata/streaming/parser/FieldConverter.scala
index 5c67dfb..0cf244a 100644
--- a/streaming/src/main/scala/org/apache/carbondata/streaming/parser/FieldConverter.scala
+++ b/streaming/src/main/scala/org/apache/carbondata/streaming/parser/FieldConverter.scala
@@ -42,12 +42,13 @@ object FieldConverter {
timeStampFormat: SimpleDateFormat,
dateFormat: SimpleDateFormat,
isVarcharType: Boolean = false,
+ isComplexType: Boolean = false,
level: Int = 0): String = {
if (value == null) {
serializationNullFormat
} else {
value match {
- case s: String => if (!isVarcharType &&
+ case s: String => if (!isVarcharType && !isComplexType &&
s.length > CarbonCommonConstants.MAX_CHARS_PER_COLUMN_DEFAULT) {
throw new Exception("Dataload failed, String length cannot exceed " +
CarbonCommonConstants.MAX_CHARS_PER_COLUMN_DEFAULT + " characters")
@@ -68,23 +69,25 @@ object FieldConverter {
val delimiter = complexDelimiters.get(level)
val builder = new StringBuilder()
s.foreach { x =>
+ val nextLevel = level + 1
builder.append(objectToString(x, serializationNullFormat, complexDelimiters,
- timeStampFormat, dateFormat, isVarcharType, level + 1))
+ timeStampFormat, dateFormat, isVarcharType, level = nextLevel))
.append(delimiter)
}
builder.substring(0, builder.length - delimiter.length())
// First convert the 'key' of Map and then append the keyValueDelimiter and then convert
// the 'value of the map and append delimiter
case m: scala.collection.Map[_, _] =>
+ val nextLevel = level + 2
val delimiter = complexDelimiters.get(level)
val keyValueDelimiter = complexDelimiters.get(level + 1)
val builder = new StringBuilder()
m.foreach { x =>
builder.append(objectToString(x._1, serializationNullFormat, complexDelimiters,
- timeStampFormat, dateFormat, isVarcharType, level + 2))
+ timeStampFormat, dateFormat, isVarcharType, level = nextLevel))
.append(keyValueDelimiter)
builder.append(objectToString(x._2, serializationNullFormat, complexDelimiters,
- timeStampFormat, dateFormat, isVarcharType, level + 2))
+ timeStampFormat, dateFormat, isVarcharType, level = nextLevel))
.append(delimiter)
}
builder.substring(0, builder.length - delimiter.length())
@@ -92,8 +95,9 @@ object FieldConverter {
val delimiter = complexDelimiters.get(level)
val builder = new StringBuilder()
for (i <- 0 until r.length) {
+ val nextLevel = level + 1
builder.append(objectToString(r(i), serializationNullFormat, complexDelimiters,
- timeStampFormat, dateFormat, isVarcharType, level + 1))
+ timeStampFormat, dateFormat, isVarcharType, level = nextLevel))
.append(delimiter)
}
builder.substring(0, builder.length - delimiter.length())
[carbondata] 01/33: [CARBONDATA-3480] Fixed unnecessary refresh for
table by removing modified mdt file
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 56879f747d150f4efae0f2998a38f4297706bc5e
Author: kunal642 <ku...@gmail.com>
AuthorDate: Fri Jul 26 14:52:36 2019 +0530
[CARBONDATA-3480] Fixed unnecessary refresh for table by removing modified mdt file
This closes #3339
---
.../carbondata/core/datamap/DataMapFilter.java | 47 +++
.../core/datamap/DataMapStoreManager.java | 14 +-
.../carbondata/core/metadata/CarbonMetadata.java | 9 +
.../core/metadata/schema/table/CarbonTable.java | 4 +-
.../core/metadata/schema/table/TableSchema.java | 4 +
.../statusmanager/SegmentUpdateStatusManager.java | 26 --
.../apache/carbondata/core/util/CarbonUtil.java | 1 -
.../core/metadata/CarbonMetadataTest.java | 7 +-
.../ThriftWrapperSchemaConverterImplTest.java | 4 +-
.../metadata/schema/table/CarbonTableTest.java | 8 +-
.../table/CarbonTableWithComplexTypesTest.java | 6 +-
.../dblocation/DBLocationCarbonTableTestCase.scala | 25 --
.../apache/spark/sql/hive/CarbonSessionUtil.scala | 6 +-
.../carbondata/indexserver/IndexServer.scala | 10 +-
.../scala/org/apache/spark/sql/CarbonEnv.scala | 51 ++-
.../command/datamap/CarbonDropDataMapCommand.scala | 1 -
.../management/RefreshCarbonTableCommand.scala | 2 -
.../CarbonAlterTableDropPartitionCommand.scala | 12 +-
.../CarbonAlterTableSplitPartitionCommand.scala | 3 -
.../command/preaaggregate/PreAggregateUtil.scala | 19 +-
.../command/table/CarbonDropTableCommand.scala | 13 +
.../spark/sql/hive/CarbonFileMetastore.scala | 425 +++++++++------------
.../spark/sql/hive/CarbonHiveMetaStore.scala | 10 +-
.../apache/spark/sql/hive/CarbonMetaStore.scala | 10 +-
.../scala/org/apache/spark/util/CleanFiles.scala | 3 -
.../scala/org/apache/spark/util/Compaction.scala | 2 -
.../apache/spark/util/DeleteSegmentByDate.scala | 2 -
.../org/apache/spark/util/DeleteSegmentById.scala | 2 -
.../scala/org/apache/spark/util/TableLoader.scala | 2 -
.../apache/spark/sql/hive/CarbonSessionState.scala | 31 +-
.../AlterTableColumnRenameTestCase.scala | 4 +-
31 files changed, 322 insertions(+), 441 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapFilter.java b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapFilter.java
index c20d0d5..ac4886d 100644
--- a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapFilter.java
+++ b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapFilter.java
@@ -18,10 +18,15 @@
package org.apache.carbondata.core.datamap;
import java.io.Serializable;
+import java.util.HashSet;
+import java.util.Set;
import org.apache.carbondata.core.datastore.block.SegmentProperties;
import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
+import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension;
+import org.apache.carbondata.core.metadata.schema.table.column.CarbonMeasure;
import org.apache.carbondata.core.scan.executor.util.RestructureUtil;
+import org.apache.carbondata.core.scan.expression.ColumnExpression;
import org.apache.carbondata.core.scan.expression.Expression;
import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
@@ -39,9 +44,51 @@ public class DataMapFilter implements Serializable {
public DataMapFilter(CarbonTable table, Expression expression) {
this.table = table;
this.expression = expression;
+ if (expression != null) {
+ checkIfFilterColumnExistsInTable();
+ }
resolve();
}
+ private Set<String> extractColumnExpressions(Expression expression) {
+ Set<String> columnExpressionList = new HashSet<>();
+ for (Expression expressions: expression.getChildren()) {
+ if (expressions != null && expressions.getChildren() != null
+ && expressions.getChildren().size() > 0) {
+ columnExpressionList.addAll(extractColumnExpressions(expressions));
+ } else if (expressions instanceof ColumnExpression) {
+ columnExpressionList.add(((ColumnExpression) expressions).getColumnName());
+ }
+ }
+ return columnExpressionList;
+ }
+
+ private void checkIfFilterColumnExistsInTable() {
+ Set<String> columnExpressionList = extractColumnExpressions(expression);
+ for (String colExpression : columnExpressionList) {
+ if (colExpression.equalsIgnoreCase("positionid")) {
+ continue;
+ }
+ boolean exists = false;
+ for (CarbonMeasure carbonMeasure : table.getAllMeasures()) {
+ if (!carbonMeasure.isInvisible() && carbonMeasure.getColName()
+ .equalsIgnoreCase(colExpression)) {
+ exists = true;
+ }
+ }
+ for (CarbonDimension carbonDimension : table.getAllDimensions()) {
+ if (!carbonDimension.isInvisible() && carbonDimension.getColName()
+ .equalsIgnoreCase(colExpression)) {
+ exists = true;
+ }
+ }
+ if (!exists) {
+ throw new RuntimeException(
+ "Column " + colExpression + " not found in table " + table.getTableUniqueName());
+ }
+ }
+ }
+
public DataMapFilter(FilterResolverIntf resolver) {
this.resolver = resolver;
}
diff --git a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
index a6a2031..ce0d6a6 100644
--- a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
+++ b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
@@ -699,8 +699,13 @@ public final class DataMapStoreManager {
SegmentUpdateDetails[] updateStatusDetails = statusManager.getUpdateStatusDetails();
for (SegmentUpdateDetails updateDetails : updateStatusDetails) {
UpdateVO updateVO = statusManager.getInvalidTimestampRange(updateDetails.getSegmentName());
- segmentRefreshTime.put(updateVO.getSegmentId(),
- new SegmentRefreshInfo(updateVO.getCreatedOrUpdatedTimeStamp(), 0));
+ SegmentRefreshInfo segmentRefreshInfo;
+ if (updateVO != null && updateVO.getLatestUpdateTimestamp() != null) {
+ segmentRefreshInfo = new SegmentRefreshInfo(updateVO.getCreatedOrUpdatedTimeStamp(), 0);
+ } else {
+ segmentRefreshInfo = new SegmentRefreshInfo(0L, 0);
+ }
+ segmentRefreshTime.put(updateVO.getSegmentId(), segmentRefreshInfo);
}
}
@@ -708,8 +713,11 @@ public final class DataMapStoreManager {
SegmentRefreshInfo segmentRefreshInfo =
seg.getSegmentRefreshInfo(updateVo);
String segmentId = seg.getSegmentNo();
+ if (segmentRefreshInfo.getSegmentUpdatedTimestamp() == null) {
+ return false;
+ }
if (segmentRefreshTime.get(segmentId) == null
- && segmentRefreshInfo.getSegmentUpdatedTimestamp() != null) {
+ && segmentRefreshInfo.getSegmentUpdatedTimestamp() != 0) {
segmentRefreshTime.put(segmentId, segmentRefreshInfo);
return true;
}
diff --git a/core/src/main/java/org/apache/carbondata/core/metadata/CarbonMetadata.java b/core/src/main/java/org/apache/carbondata/core/metadata/CarbonMetadata.java
index e44092e..9c10a03 100644
--- a/core/src/main/java/org/apache/carbondata/core/metadata/CarbonMetadata.java
+++ b/core/src/main/java/org/apache/carbondata/core/metadata/CarbonMetadata.java
@@ -16,6 +16,7 @@
*/
package org.apache.carbondata.core.metadata;
+import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
@@ -180,4 +181,12 @@ public final class CarbonMetadata {
}
return null;
}
+
+ public List<CarbonTable> getAllTables() {
+ return new ArrayList<>(tableInfoMap.values());
+ }
+
+ public void clearAll() {
+ tableInfoMap.clear();
+ }
}
diff --git a/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java b/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
index 1be1624..aa82b64 100644
--- a/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
+++ b/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
@@ -28,6 +28,7 @@ import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import org.apache.carbondata.common.exceptions.sql.MalformedDataMapCommandException;
@@ -286,7 +287,8 @@ public class CarbonTable implements Serializable, Writable {
* @return
*/
public static String buildUniqueName(String databaseName, String tableName) {
- return databaseName + CarbonCommonConstants.UNDERSCORE + tableName;
+ return (databaseName + CarbonCommonConstants.UNDERSCORE + tableName).toLowerCase(
+ Locale.getDefault());
}
/**
diff --git a/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchema.java b/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchema.java
index 4425697..61b2987 100644
--- a/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchema.java
+++ b/core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchema.java
@@ -24,6 +24,7 @@ import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
@@ -142,6 +143,9 @@ public class TableSchema implements Serializable, Writable {
* @param tableName the tableName to set
*/
public void setTableName(String tableName) {
+ if (tableName != null) {
+ tableName = tableName.toLowerCase(Locale.getDefault());
+ }
this.tableName = tableName;
}
diff --git a/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java b/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
index eace1b7..f7083dc 100644
--- a/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
+++ b/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
@@ -775,32 +775,6 @@ public class SegmentUpdateStatusManager {
}
/**
- * Returns the invalid timestamp range of a segment.
- * @return
- */
- public List<UpdateVO> getInvalidTimestampRange() {
- List<UpdateVO> ranges = new ArrayList<UpdateVO>();
- for (LoadMetadataDetails segment : segmentDetails) {
- if ((SegmentStatus.LOAD_FAILURE == segment.getSegmentStatus()
- || SegmentStatus.COMPACTED == segment.getSegmentStatus()
- || SegmentStatus.MARKED_FOR_DELETE == segment.getSegmentStatus())) {
- UpdateVO range = new UpdateVO();
- range.setSegmentId(segment.getLoadName());
- range.setFactTimestamp(segment.getLoadStartTime());
- if (!segment.getUpdateDeltaStartTimestamp().isEmpty() &&
- !segment.getUpdateDeltaEndTimestamp().isEmpty()) {
- range.setUpdateDeltaStartTimestamp(
- CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaStartTimestamp()));
- range.setLatestUpdateTimestamp(
- CarbonUpdateUtil.getTimeStampAsLong(segment.getUpdateDeltaEndTimestamp()));
- }
- ranges.add(range);
- }
- }
- return ranges;
- }
-
- /**
*
* @param block
* @param needCompleteList
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
index a86690c..cb23e3e 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
@@ -2239,7 +2239,6 @@ public final class CarbonUtil {
org.apache.carbondata.format.TableInfo tableInfo =
new org.apache.carbondata.format.TableInfo(thriftFactTable,
new ArrayList<org.apache.carbondata.format.TableSchema>());
-
tableInfo.setDataMapSchemas(null);
return tableInfo;
}
diff --git a/core/src/test/java/org/apache/carbondata/core/metadata/CarbonMetadataTest.java b/core/src/test/java/org/apache/carbondata/core/metadata/CarbonMetadataTest.java
index 0ec19f2..6d8dac6 100644
--- a/core/src/test/java/org/apache/carbondata/core/metadata/CarbonMetadataTest.java
+++ b/core/src/test/java/org/apache/carbondata/core/metadata/CarbonMetadataTest.java
@@ -47,6 +47,7 @@ public class CarbonMetadataTest {
@BeforeClass public static void setUp() {
carbonMetadata = CarbonMetadata.getInstance();
+ carbonMetadata.clearAll();
carbonMetadata.loadTableMetadata(getTableInfo(10000));
tableUniqueName = CarbonTable.buildUniqueName("carbonTestDatabase", "carbonTestTable");
}
@@ -77,13 +78,13 @@ public class CarbonMetadataTest {
@Test public void testGetCarbonTableReturingProperTableWithProperDimensionCount() {
int expectedResult = 1;
assertEquals(expectedResult,
- carbonMetadata.getCarbonTable(tableUniqueName).getNumberOfDimensions("carbonTestTable"));
+ carbonMetadata.getCarbonTable(tableUniqueName).getNumberOfDimensions("carbontesttable"));
}
@Test public void testGetCarbonTableReturingProperTableWithProperMeasureCount() {
int expectedResult = 1;
assertEquals(expectedResult,
- carbonMetadata.getCarbonTable(tableUniqueName).getNumberOfMeasures("carbonTestTable"));
+ carbonMetadata.getCarbonTable(tableUniqueName).getNumberOfMeasures("carbontesttable"));
}
@Test public void testGetCarbonTableReturingProperTableWithProperDatabaseName() {
@@ -92,7 +93,7 @@ public class CarbonMetadataTest {
}
@Test public void testGetCarbonTableReturingProperTableWithProperFactTableName() {
- String expectedResult = "carbonTestTable";
+ String expectedResult = "carbontesttable";
assertEquals(expectedResult, carbonMetadata.getCarbonTable(tableUniqueName).getTableName());
}
diff --git a/core/src/test/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImplTest.java b/core/src/test/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImplTest.java
index 897c3cf..f03e193 100644
--- a/core/src/test/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImplTest.java
+++ b/core/src/test/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImplTest.java
@@ -1704,7 +1704,7 @@ public class ThriftWrapperSchemaConverterImplTest {
@Test public void testFromExternalToWrapperTableSchema() {
String tableId = "1";
- String tableName = "tableName";
+ String tableName = "tablename";
TableSchema actualResult =
thriftWrapperSchemaConverter.fromExternalToWrapperTableSchema(tabSchema, "tableName");
assertEquals(tableId, actualResult.getTableId());
@@ -1729,7 +1729,7 @@ public class ThriftWrapperSchemaConverterImplTest {
TableInfo actualResult = thriftWrapperSchemaConverter
.fromExternalToWrapperTableInfo(externalTableInfo, "dbName", "tableName", "/path");
assertEquals(time, actualResult.getLastUpdatedTime());
- assertEquals("dbName_tableName", actualResult.getTableUniqueName());
+ assertEquals("dbname_tablename", actualResult.getTableUniqueName());
}
}
\ No newline at end of file
diff --git a/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableTest.java b/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableTest.java
index ec1303f..0f9e252 100644
--- a/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableTest.java
+++ b/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableTest.java
@@ -44,11 +44,11 @@ public class CarbonTableTest extends TestCase {
}
@Test public void testNumberOfDimensionReturnsProperCount() {
- assertEquals(1, carbonTable.getNumberOfDimensions("carbonTestTable"));
+ assertEquals(1, carbonTable.getNumberOfDimensions("carbontesttable"));
}
@Test public void testNumberOfMeasureReturnsProperCount() {
- assertEquals(1, carbonTable.getNumberOfMeasures("carbonTestTable"));
+ assertEquals(1, carbonTable.getNumberOfMeasures("carbontesttable"));
}
@Test public void testGetDatabaseNameResturnsDatabaseName() {
@@ -56,7 +56,7 @@ public class CarbonTableTest extends TestCase {
}
@Test public void testFactTableNameReturnsProperFactTableName() {
- assertEquals("carbonTestTable", carbonTable.getTableName());
+ assertEquals("carbontesttable", carbonTable.getTableName());
}
@Test public void testTableUniqueNameIsProper() {
@@ -65,7 +65,7 @@ public class CarbonTableTest extends TestCase {
@Test public void testDimensionPresentInTableIsProper() {
CarbonDimension dimension = new CarbonDimension(getColumnarDimensionColumn(), 0, -1, -1);
- assertTrue(carbonTable.getDimensionByName("carbonTestTable", "IMEI").equals(dimension));
+ assertTrue(carbonTable.getDimensionByName("carbontesttable", "IMEI").equals(dimension));
}
static ColumnSchema getColumnarDimensionColumn() {
diff --git a/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableWithComplexTypesTest.java b/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableWithComplexTypesTest.java
index d3403d5..0d0d4df 100644
--- a/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableWithComplexTypesTest.java
+++ b/core/src/test/java/org/apache/carbondata/core/metadata/schema/table/CarbonTableWithComplexTypesTest.java
@@ -43,11 +43,11 @@ public class CarbonTableWithComplexTypesTest extends TestCase {
}
@Test public void testNumberOfDimensionReturnsProperCount() {
- assertEquals(2, carbonTable.getNumberOfDimensions("carbonTestTable"));
+ assertEquals(2, carbonTable.getNumberOfDimensions("carbontesttable"));
}
@Test public void testNumberOfMeasureReturnsProperCount() {
- assertEquals(1, carbonTable.getNumberOfMeasures("carbonTestTable"));
+ assertEquals(1, carbonTable.getNumberOfMeasures("carbontesttable"));
}
@Test public void testGetDatabaseNameResturnsDatabaseName() {
@@ -55,7 +55,7 @@ public class CarbonTableWithComplexTypesTest extends TestCase {
}
@Test public void testFactTableNameReturnsProperFactTableName() {
- assertEquals("carbonTestTable", carbonTable.getTableName());
+ assertEquals("carbontesttable", carbonTable.getTableName());
}
@Test public void testTableUniqueNameIsProper() {
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dblocation/DBLocationCarbonTableTestCase.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dblocation/DBLocationCarbonTableTestCase.scala
index 37ad08c..236dcfc 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dblocation/DBLocationCarbonTableTestCase.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dblocation/DBLocationCarbonTableTestCase.scala
@@ -261,31 +261,6 @@ class DBLocationCarbonTableTestCase extends QueryTest with BeforeAndAfterEach {
sql("drop table carbontable")
}
- test("test mdt file path with configured paths") {
- sql(s"create database carbon location '$dblocation'")
- sql("use carbon")
- CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_UPDATE_SYNC_FOLDER, "/tmp/carbondata1/carbondata2/")
- val (timestampFile, timestampFileType) = getMdtFileAndType()
- FileFactory.deleteFile(timestampFile, timestampFileType)
- sql("""create table carbon.carbontable (c1 string,c2 int,c3 string,c5 string) STORED BY 'org.apache.carbondata.format'""")
- sql("drop table carbontable")
- // perform file check
- assert(FileFactory.isFileExist(timestampFile, true) ||
- CarbonEnv.getInstance(sqlContext.sparkSession).carbonMetaStore.isReadFromHiveMetaStore)
-
- CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_UPDATE_SYNC_FOLDER,
- CarbonCommonConstants.CARBON_UPDATE_SYNC_FOLDER_DEFAULT)
- val (timestampFile2, timestampFileType2) = getMdtFileAndType()
- FileFactory.deleteFile(timestampFile2, timestampFileType2)
- sql("""create table carbon.carbontable (c1 string,c2 int,c3 string,c5 string) STORED BY 'org.apache.carbondata.format'""")
- sql("drop table carbontable")
- // perform file check
- assert(FileFactory.isFileExist(timestampFile, true) ||
- CarbonEnv.getInstance(sqlContext.sparkSession).carbonMetaStore.isReadFromHiveMetaStore)
- }
-
override def afterEach {
CarbonProperties.getInstance()
.addProperty(CarbonCommonConstants.CARBON_UPDATE_SYNC_FOLDER,
diff --git a/integration/spark2/src/main/commonTo2.2And2.3/org/apache/spark/sql/hive/CarbonSessionUtil.scala b/integration/spark2/src/main/commonTo2.2And2.3/org/apache/spark/sql/hive/CarbonSessionUtil.scala
index febba35..e3f1d3f 100644
--- a/integration/spark2/src/main/commonTo2.2And2.3/org/apache/spark/sql/hive/CarbonSessionUtil.scala
+++ b/integration/spark2/src/main/commonTo2.2And2.3/org/apache/spark/sql/hive/CarbonSessionUtil.scala
@@ -71,7 +71,7 @@ object CarbonSessionUtil {
case _ =>
}
isRelationRefreshed =
- CarbonEnv.refreshRelationFromCache(catalogTable.identifier)(sparkSession)
+ CarbonEnv.isRefreshRequired(catalogTable.identifier)(sparkSession)
case _ =>
}
}
@@ -79,12 +79,12 @@ object CarbonSessionUtil {
rtnRelation match {
case SubqueryAlias(_,
MatchLogicalRelation(_: CarbonDatasourceHadoopRelation, _, catalogTable)) =>
- isRelationRefreshed = CarbonEnv.refreshRelationFromCache(name)(sparkSession)
+ isRelationRefreshed = CarbonEnv.isRefreshRequired(name)(sparkSession)
if (catalogTable.isInstanceOf[Option[CatalogTable]]) {
catalogTable.asInstanceOf[Option[CatalogTable]].foreach(setStatsNone)
}
case MatchLogicalRelation(_: CarbonDatasourceHadoopRelation, _, catalogTable) =>
- isRelationRefreshed = CarbonEnv.refreshRelationFromCache(name)(sparkSession)
+ isRelationRefreshed = CarbonEnv.isRefreshRequired(name)(sparkSession)
if (catalogTable.isInstanceOf[Option[CatalogTable]]) {
catalogTable.asInstanceOf[Option[CatalogTable]].foreach(setStatsNone)
}
diff --git a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
index 718bb74..fdaa3d1 100644
--- a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
+++ b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
@@ -106,6 +106,11 @@ object IndexServer extends ServerInterface {
sparkSession.sparkContext
.setLocalProperty("spark.job.description", request.getTaskGroupDesc)
}
+ if (!request.getInvalidSegments.isEmpty) {
+ DistributedRDDUtils
+ .invalidateSegmentMapping(request.getCarbonTable.getTableUniqueName,
+ request.getInvalidSegments.asScala)
+ }
val splits = new DistributedPruneRDD(sparkSession, request).collect()
if (!request.isFallbackJob) {
DistributedRDDUtils.updateExecutorCacheSize(splits.map(_._1).toSet)
@@ -113,11 +118,6 @@ object IndexServer extends ServerInterface {
if (request.isJobToClearDataMaps) {
DistributedRDDUtils.invalidateTableMapping(request.getCarbonTable.getTableUniqueName)
}
- if (!request.getInvalidSegments.isEmpty) {
- DistributedRDDUtils
- .invalidateSegmentMapping(request.getCarbonTable.getTableUniqueName,
- request.getInvalidSegments.asScala)
- }
new ExtendedBlockletWrapperContainer(splits.map(_._2), request.isFallbackJob)
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
index e7a6d65..c13e7b9 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
@@ -33,7 +33,7 @@ import org.apache.carbondata.common.logging.LogServiceFactory
import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.datamap.DataMapStoreManager
import org.apache.carbondata.core.datastore.impl.FileFactory
-import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier
+import org.apache.carbondata.core.metadata.{AbsoluteTableIdentifier, CarbonMetadata}
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
import org.apache.carbondata.core.util._
import org.apache.carbondata.datamap.{TextMatchMaxDocUDF, TextMatchUDF}
@@ -215,38 +215,35 @@ object CarbonEnv {
databaseNameOp: Option[String],
tableName: String)
(sparkSession: SparkSession): CarbonTable = {
- refreshRelationFromCache(TableIdentifier(tableName, databaseNameOp))(sparkSession)
- val databaseName = getDatabaseName(databaseNameOp)(sparkSession)
val catalog = getInstance(sparkSession).carbonMetaStore
- // refresh cache
- catalog.checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, databaseNameOp))
-
- // try to get it from catch, otherwise lookup in catalog
- catalog.getTableFromMetadataCache(databaseName, tableName)
- .getOrElse(
- catalog
- .lookupRelation(databaseNameOp, tableName)(sparkSession)
- .asInstanceOf[CarbonRelation]
- .carbonTable)
+ // if relation is not refreshed of the table does not exist in cache then
+ if (isRefreshRequired(TableIdentifier(tableName, databaseNameOp))(sparkSession)) {
+ catalog
+ .lookupRelation(databaseNameOp, tableName)(sparkSession)
+ .asInstanceOf[CarbonRelation]
+ .carbonTable
+ } else {
+ CarbonMetadata.getInstance().getCarbonTable(databaseNameOp.getOrElse(sparkSession
+ .catalog.currentDatabase), tableName)
+ }
}
- def refreshRelationFromCache(identifier: TableIdentifier)(sparkSession: SparkSession): Boolean = {
- var isRefreshed = false
+ /**
+ *
+ * @return true is the relation was changes and was removed from cache. false is there is no
+ * change in the relation.
+ */
+ def isRefreshRequired(identifier: TableIdentifier)(sparkSession: SparkSession): Boolean = {
val carbonEnv = getInstance(sparkSession)
- val table = carbonEnv.carbonMetaStore.getTableFromMetadataCache(
- identifier.database.getOrElse(sparkSession.sessionState.catalog.getCurrentDatabase),
- identifier.table)
- if (carbonEnv.carbonMetaStore
- .checkSchemasModifiedTimeAndReloadTable(identifier) && table.isDefined) {
- sparkSession.sessionState.catalog.refreshTable(identifier)
- val tablePath = table.get.getTablePath
- DataMapStoreManager.getInstance().
- clearDataMaps(AbsoluteTableIdentifier.from(tablePath,
+ val databaseName = identifier.database.getOrElse(sparkSession.catalog.currentDatabase)
+ val table = CarbonMetadata.getInstance().getCarbonTable(databaseName, identifier.table)
+ if (table == null) {
+ true
+ } else {
+ carbonEnv.carbonMetaStore.isSchemaRefreshed(AbsoluteTableIdentifier.from(table.getTablePath,
identifier.database.getOrElse(sparkSession.sessionState.catalog.getCurrentDatabase),
- identifier.table, table.get.getTableInfo.getFactTable.getTableId))
- isRefreshed = true
+ identifier.table, table.getTableInfo.getFactTable.getTableId), sparkSession)
}
- isRefreshed
}
/**
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDropDataMapCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDropDataMapCommand.scala
index b4e60fb..1fa1337 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDropDataMapCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDropDataMapCommand.scala
@@ -67,7 +67,6 @@ case class CarbonDropDataMapCommand(
val carbonEnv = CarbonEnv.getInstance(sparkSession)
val catalog = carbonEnv.carbonMetaStore
val tablePath = CarbonEnv.getTablePath(databaseNameOp, tableName)(sparkSession)
- catalog.checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
if (mainTable == null) {
mainTable = try {
CarbonEnv.getCarbonTable(databaseNameOp, tableName)(sparkSession)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
index 6a9ac0a..cebf606 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/RefreshCarbonTableCommand.scala
@@ -105,8 +105,6 @@ case class RefreshCarbonTableCommand(
}
}
}
- // update the schema modified time
- metaStore.updateAndTouchSchemasUpdatedTime()
Seq.empty
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropPartitionCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropPartitionCommand.scala
index 566e44e..507fe02 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropPartitionCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropPartitionCommand.scala
@@ -60,14 +60,8 @@ case class CarbonAlterTableDropPartitionCommand(
setAuditTable(dbName, tableName)
setAuditInfo(Map("partition" -> model.partitionId))
val carbonMetaStore = CarbonEnv.getInstance(sparkSession).carbonMetaStore
- val relation = carbonMetaStore.lookupRelation(Option(dbName), tableName)(sparkSession)
- .asInstanceOf[CarbonRelation]
- val tablePath = relation.carbonTable.getTablePath
- carbonMetaStore.checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
- if (relation == null || CarbonMetadata.getInstance.getCarbonTable(dbName, tableName) == null) {
- throwMetadataException(dbName, tableName, "table not found")
- }
- val carbonTable = relation.carbonTable
+ val carbonTable = CarbonEnv.getCarbonTable(Option(dbName), tableName)(sparkSession)
+ val tablePath = carbonTable.getTablePath
val partitionInfo = carbonTable.getPartitionInfo(tableName)
if (partitionInfo == null) {
throwMetadataException(dbName, tableName, "table is not a partition table")
@@ -116,8 +110,6 @@ case class CarbonAlterTableDropPartitionCommand(
thriftTable,
null,
carbonTable.getAbsoluteTableIdentifier.getTablePath)(sparkSession)
- // update the schema modified time
- carbonMetaStore.updateAndTouchSchemasUpdatedTime()
// sparkSession.catalog.refreshTable(tableName)
Seq.empty
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableSplitPartitionCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableSplitPartitionCommand.scala
index 72c3142..36ddce4 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableSplitPartitionCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableSplitPartitionCommand.scala
@@ -69,7 +69,6 @@ case class CarbonAlterTableSplitPartitionCommand(
if (relation == null) {
throwMetadataException(dbName, tableName, "table not found")
}
- carbonMetaStore.checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
if (null == (CarbonEnv.getCarbonTable(Some(dbName), tableName)(sparkSession))) {
LOGGER.error(s"Alter table failed. table not found: $dbName.$tableName")
throwMetadataException(dbName, tableName, "table not found")
@@ -107,8 +106,6 @@ case class CarbonAlterTableSplitPartitionCommand(
thriftTable,
null,
carbonTable.getAbsoluteTableIdentifier.getTablePath)(sparkSession)
- // update the schema modified time
- carbonMetaStore.updateAndTouchSchemasUpdatedTime()
Seq.empty
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
index db52361..480d1f7 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
@@ -471,7 +471,7 @@ object PreAggregateUtil {
val dbName = carbonTable.getDatabaseName
val tableName = carbonTable.getTableName
CarbonEnv.getInstance(sparkSession).carbonMetaStore
- .updateTableSchemaForDataMap(carbonTable.getCarbonTableIdentifier,
+ .updateTableSchema(carbonTable.getCarbonTableIdentifier,
carbonTable.getCarbonTableIdentifier,
thriftTable,
carbonTable.getAbsoluteTableIdentifier.getTablePath)(sparkSession)
@@ -539,23 +539,6 @@ object PreAggregateUtil {
}
}
- def getChildCarbonTable(databaseName: String, tableName: String)
- (sparkSession: SparkSession): Option[CarbonTable] = {
- val metaStore = CarbonEnv.getInstance(sparkSession).carbonMetaStore
- val carbonTable = metaStore.getTableFromMetadataCache(databaseName, tableName)
- if (carbonTable.isEmpty) {
- try {
- Some(metaStore.lookupRelation(Some(databaseName), tableName)(sparkSession)
- .asInstanceOf[CarbonRelation].metaData.carbonTable)
- } catch {
- case _: Exception =>
- None
- }
- } else {
- carbonTable
- }
- }
-
/**
* Below method will be used to update logical plan
* this is required for creating pre aggregate tables,
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDropTableCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDropTableCommand.scala
index ff0177b..6b80bbe 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDropTableCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonDropTableCommand.scala
@@ -25,6 +25,7 @@ import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
import org.apache.spark.sql.execution.command.AtomicRunnableCommand
import org.apache.spark.sql.execution.command.datamap.CarbonDropDataMapCommand
+import org.apache.spark.sql.hive.CarbonFileMetastore
import org.apache.carbondata.common.logging.LogServiceFactory
import org.apache.carbondata.core.cache.dictionary.ManageDictionaryAndBTree
@@ -183,6 +184,18 @@ case class CarbonDropTableCommand(
ifExistsSet,
sparkSession)
OperationListenerBus.getInstance.fireEvent(dropTablePostEvent, operationContext)
+ // Remove all invalid entries of carbonTable and corresponding updated timestamp
+ // values from the cache. This case is valid when there are 2 JDBCServer and one of them
+ // drops the table, the other server would not be able to clear its cache.
+ try {
+ CarbonEnv.getInstance(sparkSession).carbonMetaStore match {
+ case metastore: CarbonFileMetastore => metastore.removeStaleTimeStampEntries(sparkSession)
+ case _ =>
+ }
+ } catch {
+ case _: Exception =>
+ // Do nothing
+ }
} catch {
case ex: NoSuchTableException =>
if (!ifExistsSet) {
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
index ea3bba8..7ab2d47 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
@@ -19,11 +19,13 @@ package org.apache.spark.sql.hive
import java.io.IOException
import java.net.URI
+import java.util.Locale
+import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.locks.{Lock, ReentrantReadWriteLock}
+import scala.collection.JavaConverters._
import scala.collection.mutable.ArrayBuffer
-import org.apache.hadoop.fs.permission.{FsAction, FsPermission}
import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, CarbonEnv, SparkSession}
import org.apache.spark.sql.CarbonExpressions.{CarbonSubqueryAlias => SubqueryAlias}
import org.apache.spark.sql.catalyst.TableIdentifier
@@ -36,7 +38,6 @@ import org.apache.spark.util.{CarbonReflectionUtils, SparkUtil}
import org.apache.carbondata.common.logging.LogServiceFactory
import org.apache.carbondata.core.cache.dictionary.ManageDictionaryAndBTree
-import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.datamap.DataMapStoreManager
import org.apache.carbondata.core.datastore.block.SegmentPropertiesAndSchemaHolder
import org.apache.carbondata.core.datastore.impl.FileFactory
@@ -87,59 +88,95 @@ object MatchLogicalRelation {
}
}
-class CarbonFileMetastore extends CarbonMetaStore {
+private object CarbonFileMetastore {
+
+ final val tableModifiedTimeStore = new ConcurrentHashMap[String, Long]()
+
+ def checkIfRefreshIsNeeded(absoluteTableIdentifier: AbsoluteTableIdentifier,
+ localTimeStamp: Long): Boolean = synchronized {
+ val schemaFilePath = CarbonTablePath.getSchemaFilePath(absoluteTableIdentifier.getTablePath)
+ val schemaCarbonFile = FileFactory.getCarbonFile(schemaFilePath)
+ if (schemaCarbonFile.exists()) {
+ val oldTime = Option(CarbonFileMetastore.getTableModifiedTime(absoluteTableIdentifier
+ .getCarbonTableIdentifier
+ .getTableId))
+ val newTime = schemaCarbonFile.getLastModifiedTime
+ val isSchemaModified = oldTime match {
+ case Some(cacheTime) =>
+ cacheTime != newTime
+ case None => true
+ }
+ if (isSchemaModified) {
+ CarbonMetadata.getInstance().removeTable(absoluteTableIdentifier
+ .getCarbonTableIdentifier.getTableUniqueName)
+ DataMapStoreManager.getInstance().clearDataMaps(absoluteTableIdentifier)
+ true
+ } else {
+ localTimeStamp != newTime
+ }
+ } else {
+ false
+ }
+ }
- @transient
- val LOGGER = LogServiceFactory.getLogService("org.apache.spark.sql.CarbonMetastoreCatalog")
+ def updateTableSchemaModifiedTime(tableUniqueId: String, timeStamp: Long): Unit = {
+ tableModifiedTimeStore.put(tableUniqueId, timeStamp)
+ }
- val tableModifiedTimeStore = new java.util.HashMap[String, Long]()
- tableModifiedTimeStore
- .put(CarbonCommonConstants.DATABASE_DEFAULT_NAME, System.currentTimeMillis())
+ def getTableModifiedTime(tableUniqueId: String): Long = {
+ tableModifiedTimeStore.get(tableUniqueId)
+ }
- def nextQueryId: String = {
- System.nanoTime() + ""
+ def removeStaleEntries(invalidTableUniqueIds: List[String]) {
+ for (invalidKey <- invalidTableUniqueIds) {
+ tableModifiedTimeStore.remove(invalidKey)
+ }
}
+}
- val metadata = MetaData(new ArrayBuffer[CarbonTable]())
+class CarbonFileMetastore extends CarbonMetaStore {
+ @transient private val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ final val tableModifiedTimeStore = new ConcurrentHashMap[String, Long]()
/**
- * Create spark session from paramters.
- *
- * @param parameters
- * @param absIdentifier
- * @param sparkSession
+ * Create Carbon Relation by reading the schema file
*/
override def createCarbonRelation(parameters: Map[String, String],
absIdentifier: AbsoluteTableIdentifier,
sparkSession: SparkSession): CarbonRelation = {
val database = absIdentifier.getCarbonTableIdentifier.getDatabaseName
val tableName = absIdentifier.getCarbonTableIdentifier.getTableName
- val tables = getTableFromMetadataCache(database, tableName)
+ val tables = Option(CarbonMetadata.getInstance.getCarbonTable(database, tableName))
tables match {
case Some(t) =>
- CarbonRelation(database, tableName, CarbonSparkUtil.createSparkMeta(t), t)
- case None =>
- readCarbonSchema(absIdentifier,
- !parameters.getOrElse("isTransactional", "true").toBoolean) match {
- case Some(meta) =>
- CarbonRelation(database, tableName,
- CarbonSparkUtil.createSparkMeta(meta), meta)
- case None =>
- throw new NoSuchTableException(database, tableName)
+ if (isSchemaRefreshed(absIdentifier, sparkSession)) {
+ readCarbonSchema(absIdentifier, parameters)
+ } else {
+ CarbonRelation(database, tableName, CarbonSparkUtil.createSparkMeta(t), t)
}
+ case None =>
+ readCarbonSchema(absIdentifier, parameters)
+ }
+ }
+
+ private def readCarbonSchema(absIdentifier: AbsoluteTableIdentifier,
+ parameters: Map[String, String]): CarbonRelation = {
+ readCarbonSchema(absIdentifier,
+ !parameters.getOrElse("isTransactional", "true").toBoolean) match {
+ case Some(meta) =>
+ CarbonRelation(absIdentifier.getDatabaseName, absIdentifier.getTableName,
+ CarbonSparkUtil.createSparkMeta(meta), meta)
+ case None =>
+ throw new NoSuchTableException(absIdentifier.getDatabaseName, absIdentifier.getTableName)
}
}
/**
- * This method will overwrite the existing schema and update it with the given details
- *
- * @param newTableIdentifier
- * @param thriftTableInfo
- * @param carbonStorePath
- * @param sparkSession
+ * This method will overwrite the existing schema and update it with the given details.
*/
- def updateTableSchemaForDataMap(newTableIdentifier: CarbonTableIdentifier,
+ def updateTableSchema(newTableIdentifier: CarbonTableIdentifier,
oldTableIdentifier: CarbonTableIdentifier,
thriftTableInfo: org.apache.carbondata.format.TableInfo,
carbonStorePath: String)(sparkSession: SparkSession): String = {
@@ -174,13 +211,17 @@ class CarbonFileMetastore extends CarbonMetaStore {
catalogTable.provider match {
case Some(name) if (name.equals("org.apache.spark.sql.CarbonSource")
|| name.equalsIgnoreCase("carbondata")) => name
- case _ => throw new NoSuchTableException(database, tableIdentifier.table)
+ case _ =>
+ CarbonMetadata.getInstance().removeTable(database, tableIdentifier.table)
+ throw new NoSuchTableException(database, tableIdentifier.table)
}
val identifier: AbsoluteTableIdentifier = AbsoluteTableIdentifier.from(
catalogTable.location.toString, database, tableIdentifier.table)
CarbonEnv.getInstance(sparkSession).carbonMetaStore.
createCarbonRelation(catalogTable.storage.properties, identifier, sparkSession)
- case _ => throw new NoSuchTableException(database, tableIdentifier.table)
+ case _ =>
+ CarbonMetadata.getInstance().removeTable(database, tableIdentifier.table)
+ throw new NoSuchTableException(database, tableIdentifier.table)
}
// fire post event after lookup relation
@@ -193,22 +234,6 @@ class CarbonFileMetastore extends CarbonMetaStore {
relation
}
- /**
- * This method will search for a table in the catalog metadata
- *
- * @param database
- * @param tableName
- * @return
- */
- def getTableFromMetadataCache(database: String, tableName: String): Option[CarbonTable] = {
- metadata.readLock.lock()
- val ret = metadata.carbonTables
- .find(table => table.getDatabaseName.equalsIgnoreCase(database) &&
- table.getTableName.equalsIgnoreCase(tableName))
- metadata.readLock.unlock()
- ret
- }
-
def tableExists(
table: String,
databaseOp: Option[String] = None)(sparkSession: SparkSession): Boolean = {
@@ -226,71 +251,60 @@ class CarbonFileMetastore extends CarbonMetaStore {
true
}
- def isTableInMetastore(identifier: AbsoluteTableIdentifier,
- sparkSession: SparkSession): Boolean = {
- sparkSession.sessionState.catalog.listTables(identifier.getDatabaseName)
- .exists(_.table.equalsIgnoreCase(identifier.getTableName))
- }
-
-
private def readCarbonSchema(identifier: AbsoluteTableIdentifier,
inferSchema: Boolean): Option[CarbonTable] = {
-
val schemaConverter = new ThriftWrapperSchemaConverterImpl
val dbName = identifier.getCarbonTableIdentifier.getDatabaseName
val tableName = identifier.getCarbonTableIdentifier.getTableName
val tableUniqueName = CarbonTable.buildUniqueName(dbName, tableName)
val tablePath = identifier.getTablePath
+ var schemaRefreshTime = System.currentTimeMillis()
val wrapperTableInfo =
- if (inferSchema) {
- val carbonTbl = CarbonMetadata.getInstance().getCarbonTable(dbName, tableName)
- val tblInfoFromCache = if (carbonTbl != null) {
- carbonTbl.getTableInfo
- } else {
- null
- }
-
- val thriftTableInfo : TableInfo = if (tblInfoFromCache != null) {
- // In case the TableInfo is present in the Carbon Metadata Cache
- // then get the tableinfo from the cache rather than infering from
- // the CarbonData file.
- schemaConverter
- .fromWrapperToExternalTableInfo(tblInfoFromCache, dbName, tableName)
- } else {
- schemaConverter
- .fromWrapperToExternalTableInfo(SchemaReader
- .inferSchema(identifier, false),
- dbName, tableName)
- }
+ if (inferSchema) {
+ val carbonTbl = CarbonMetadata.getInstance().getCarbonTable(dbName, tableName)
+ val tblInfoFromCache = if (carbonTbl != null) {
+ carbonTbl.getTableInfo
+ } else {
+ null
+ }
- val wrapperTableInfo =
- schemaConverter
- .fromExternalToWrapperTableInfo(thriftTableInfo, dbName, tableName, tablePath)
- wrapperTableInfo.getFactTable.getTableProperties.put("_external", "true")
- wrapperTableInfo.setTransactionalTable(false)
- Some(wrapperTableInfo)
- } else {
- val tableMetadataFile = CarbonTablePath.getSchemaFilePath(tablePath)
- val fileType = FileFactory.getFileType(tableMetadataFile)
- if (FileFactory.isFileExist(tableMetadataFile, fileType)) {
- val tableInfo: TableInfo = CarbonUtil.readSchemaFile(tableMetadataFile)
+ val thriftTableInfo: TableInfo = if (tblInfoFromCache != null) {
+ // In case the TableInfo is present in the Carbon Metadata Cache
+ // then get the tableinfo from the cache rather than infering from
+ // the CarbonData file.
+ schemaConverter
+ .fromWrapperToExternalTableInfo(tblInfoFromCache, dbName, tableName)
+ } else {
+ schemaConverter
+ .fromWrapperToExternalTableInfo(SchemaReader
+ .inferSchema(identifier, false),
+ dbName, tableName)
+ }
val wrapperTableInfo =
- schemaConverter.fromExternalToWrapperTableInfo(tableInfo, dbName, tableName, tablePath)
+ schemaConverter
+ .fromExternalToWrapperTableInfo(thriftTableInfo, dbName, tableName, tablePath)
+ wrapperTableInfo.getFactTable.getTableProperties.put("_external", "true")
+ wrapperTableInfo.setTransactionalTable(false)
Some(wrapperTableInfo)
} else {
- None
+ val tableMetadataFile = CarbonTablePath.getSchemaFilePath(tablePath)
+ schemaRefreshTime = FileFactory
+ .getCarbonFile(CarbonTablePath.getSchemaFilePath(tablePath)).getLastModifiedTime
+ val fileType = FileFactory.getFileType(tableMetadataFile)
+ if (FileFactory.isFileExist(tableMetadataFile, fileType)) {
+ val tableInfo: TableInfo = CarbonUtil.readSchemaFile(tableMetadataFile)
+ val wrapperTableInfo =
+ schemaConverter.fromExternalToWrapperTableInfo(tableInfo, dbName, tableName, tablePath)
+ Some(wrapperTableInfo)
+ } else {
+ None
+ }
}
- }
-
-
wrapperTableInfo.map { tableInfo =>
+ updateSchemasUpdatedTime(tableInfo.getFactTable.getTableId, schemaRefreshTime)
CarbonMetadata.getInstance().removeTable(tableUniqueName)
CarbonMetadata.getInstance().loadTableMetadata(tableInfo)
- val carbonTable = CarbonMetadata.getInstance().getCarbonTable(tableUniqueName)
- metadata.writeLock.lock()
- metadata.carbonTables += carbonTable
- metadata.writeLock.unlock()
- carbonTable
+ CarbonMetadata.getInstance().getCarbonTable(tableUniqueName)
}
}
@@ -319,17 +333,12 @@ class CarbonFileMetastore extends CarbonMetaStore {
newTableIdentifier.getTableName,
oldTableIdentifier.getTableId)
val path = createSchemaThriftFile(newAbsoluteTableIdentifier, thriftTableInfo)
- addTableCache(wrapperTableInfo, newAbsoluteTableIdentifier)
-
+ addCarbonTableToCache(wrapperTableInfo, newAbsoluteTableIdentifier)
path
}
/**
* This method will is used to remove the evolution entry in case of failure.
- *
- * @param carbonTableIdentifier
- * @param thriftTableInfo
- * @param sparkSession
*/
def revertTableSchemaInAlterFailure(carbonTableIdentifier: CarbonTableIdentifier,
thriftTableInfo: org.apache.carbondata.format.TableInfo,
@@ -343,7 +352,7 @@ class CarbonFileMetastore extends CarbonMetaStore {
val evolutionEntries = thriftTableInfo.fact_table.schema_evolution.schema_evolution_history
evolutionEntries.remove(evolutionEntries.size() - 1)
val path = createSchemaThriftFile(absoluteTableIdentifier, thriftTableInfo)
- addTableCache(wrapperTableInfo, absoluteTableIdentifier)
+ addCarbonTableToCache(wrapperTableInfo, absoluteTableIdentifier)
path
}
@@ -360,7 +369,7 @@ class CarbonFileMetastore extends CarbonMetaStore {
val childSchemaList = wrapperTableInfo.getDataMapSchemaList
childSchemaList.remove(childSchemaList.size() - 1)
val path = createSchemaThriftFile(absoluteTableIdentifier, thriftTableInfo)
- addTableCache(wrapperTableInfo, absoluteTableIdentifier)
+ addCarbonTableToCache(wrapperTableInfo, absoluteTableIdentifier)
path
}
@@ -393,7 +402,7 @@ class CarbonFileMetastore extends CarbonMetaStore {
tableInfo.getFactTable.getSchemaEvolution.getSchemaEvolutionEntryList.add(schemaEvolutionEntry)
removeTableFromMetadata(tableInfo.getDatabaseName, tableInfo.getFactTable.getTableName)
CarbonMetadata.getInstance().loadTableMetadata(tableInfo)
- addTableCache(tableInfo, absoluteTableIdentifier)
+ addCarbonTableToCache(tableInfo, absoluteTableIdentifier)
CarbonUtil.convertToMultiGsonStrings(tableInfo, " ", "", ",")
}
@@ -417,61 +426,27 @@ class CarbonFileMetastore extends CarbonMetaStore {
thriftWriter.open(FileWriteOperation.OVERWRITE)
thriftWriter.write(thriftTableInfo)
thriftWriter.close()
- updateSchemasUpdatedTime(touchSchemaFileSystemTime())
+ val modifiedTime = System.currentTimeMillis()
+ FileFactory.getCarbonFile(schemaFilePath).setLastModifiedTime(modifiedTime)
+ updateSchemasUpdatedTime(identifier.getCarbonTableIdentifier.getTableId, modifiedTime)
identifier.getTablePath
}
- protected def addTableCache(
+ protected def addCarbonTableToCache(
tableInfo: table.TableInfo,
- absoluteTableIdentifier: AbsoluteTableIdentifier): ArrayBuffer[CarbonTable] = {
+ absoluteTableIdentifier: AbsoluteTableIdentifier): Unit = {
val identifier = absoluteTableIdentifier.getCarbonTableIdentifier
- CarbonMetadata.getInstance.removeTable(tableInfo.getTableUniqueName)
removeTableFromMetadata(identifier.getDatabaseName, identifier.getTableName)
CarbonMetadata.getInstance().loadTableMetadata(tableInfo)
- metadata.writeLock.lock()
- metadata.carbonTables +=
- CarbonMetadata.getInstance().getCarbonTable(identifier.getTableUniqueName)
- metadata.writeLock.unlock()
- metadata.carbonTables
}
/**
- * This method will remove the table meta from catalog metadata array
- *
- * @param dbName
- * @param tableName
+ * This method will remove the table meta from CarbonMetadata cache.
*/
def removeTableFromMetadata(dbName: String, tableName: String): Unit = {
- val carbonTableToBeRemoved: Option[CarbonTable] = getTableFromMetadataCache(dbName, tableName)
- carbonTableToBeRemoved match {
- case Some(carbonTable) =>
- metadata.writeLock.lock()
- metadata.carbonTables -= carbonTable
- metadata.writeLock.unlock()
- case None =>
- if (LOGGER.isDebugEnabled) {
- LOGGER.debug(s"No entry for table $tableName in database $dbName")
- }
- }
CarbonMetadata.getInstance.removeTable(dbName, tableName)
}
- private def updateMetadataByWrapperTable(
- wrapperTableInfo: org.apache.carbondata.core.metadata.schema.table.TableInfo): Unit = {
-
- CarbonMetadata.getInstance().loadTableMetadata(wrapperTableInfo)
- val carbonTable = CarbonMetadata.getInstance().getCarbonTable(
- wrapperTableInfo.getTableUniqueName)
- for (i <- metadata.carbonTables.indices) {
- metadata.writeLock.lock()
- if (wrapperTableInfo.getTableUniqueName.equals(
- metadata.carbonTables(i).getTableUniqueName)) {
- metadata.carbonTables(i) = carbonTable
- }
- metadata.writeLock.unlock()
- }
- }
-
def updateMetadataByThriftTable(schemaFilePath: String,
tableInfo: TableInfo, dbName: String, tableName: String, tablePath: String): Unit = {
tableInfo.getFact_table.getSchema_evolution.getSchema_evolution_history.get(0)
@@ -479,7 +454,8 @@ class CarbonFileMetastore extends CarbonMetaStore {
val schemaConverter = new ThriftWrapperSchemaConverterImpl
val wrapperTableInfo =
schemaConverter.fromExternalToWrapperTableInfo(tableInfo, dbName, tableName, tablePath)
- updateMetadataByWrapperTable(wrapperTableInfo)
+ addCarbonTableToCache(wrapperTableInfo,
+ wrapperTableInfo.getOrCreateAbsoluteTableIdentifier())
}
@@ -496,133 +472,65 @@ class CarbonFileMetastore extends CarbonMetaStore {
}
- def dropTable(absoluteTableIdentifier: AbsoluteTableIdentifier)
- (sparkSession: SparkSession) {
+ def dropTable(absoluteTableIdentifier: AbsoluteTableIdentifier)(sparkSession: SparkSession) {
val dbName = absoluteTableIdentifier.getCarbonTableIdentifier.getDatabaseName
val tableName = absoluteTableIdentifier.getCarbonTableIdentifier.getTableName
- val metadataFilePath = CarbonTablePath.getMetadataPath(absoluteTableIdentifier.getTablePath)
val carbonTable = CarbonMetadata.getInstance.getCarbonTable(dbName, tableName)
if (null != carbonTable) {
// clear driver B-tree and dictionary cache
ManageDictionaryAndBTree.clearBTreeAndDictionaryLRUCache(carbonTable)
}
- val fileType = FileFactory.getFileType(metadataFilePath)
-
- if (FileFactory.isFileExist(metadataFilePath, fileType)) {
- // while drop we should refresh the schema modified time so that if any thing has changed
- // in the other beeline need to update.
- checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
-
- CarbonHiveMetadataUtil.invalidateAndDropTable(dbName, tableName, sparkSession)
- updateSchemasUpdatedTime(touchSchemaFileSystemTime())
- // discard cached table info in cachedDataSourceTables
- val tableIdentifier = TableIdentifier(tableName, Option(dbName))
- sparkSession.sessionState.catalog.refreshTable(tableIdentifier)
- DataMapStoreManager.getInstance().clearDataMaps(absoluteTableIdentifier)
- SegmentPropertiesAndSchemaHolder.getInstance().invalidate(absoluteTableIdentifier)
- removeTableFromMetadata(dbName, tableName)
- } else {
- if (!isTransactionalCarbonTable(absoluteTableIdentifier)) {
- removeTableFromMetadata(dbName, tableName)
- CarbonHiveMetadataUtil.invalidateAndDropTable(dbName, tableName, sparkSession)
- // discard cached table info in cachedDataSourceTables
- val tableIdentifier = TableIdentifier(tableName, Option(dbName))
- sparkSession.sessionState.catalog.refreshTable(tableIdentifier)
- DataMapStoreManager.getInstance().clearDataMaps(absoluteTableIdentifier)
- SegmentPropertiesAndSchemaHolder.getInstance().invalidate(absoluteTableIdentifier)
- removeTableFromMetadata(dbName, tableName)
- }
- }
+ CarbonHiveMetadataUtil.invalidateAndDropTable(dbName, tableName, sparkSession)
+ // discard cached table info in cachedDataSourceTables
+ val tableIdentifier = TableIdentifier(tableName, Option(dbName))
+ sparkSession.sessionState.catalog.refreshTable(tableIdentifier)
+ DataMapStoreManager.getInstance().clearDataMaps(absoluteTableIdentifier)
+ SegmentPropertiesAndSchemaHolder.getInstance().invalidate(absoluteTableIdentifier)
+ removeTableFromMetadata(dbName, tableName)
}
def isTransactionalCarbonTable(identifier: AbsoluteTableIdentifier): Boolean = {
- val table = getTableFromMetadataCache(identifier.getDatabaseName, identifier.getTableName)
- table.map(_.getTableInfo.isTransactionalTable).getOrElse(true)
- }
-
- private def getTimestampFileAndType() = {
- var basePath = CarbonProperties.getInstance()
- .getProperty(CarbonCommonConstants.CARBON_UPDATE_SYNC_FOLDER,
- CarbonCommonConstants.CARBON_UPDATE_SYNC_FOLDER_DEFAULT)
- basePath = CarbonUtil.checkAndAppendFileSystemURIScheme(basePath)
- val timestampFile = basePath + "/" + CarbonCommonConstants.SCHEMAS_MODIFIED_TIME_FILE
- val timestampFileType = FileFactory.getFileType(timestampFile)
- if (!FileFactory.isFileExist(basePath, timestampFileType)) {
- FileFactory
- .createDirectoryAndSetPermission(basePath,
- new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL))
+ val table = Option(CarbonMetadata.getInstance()
+ .getCarbonTable(identifier.getCarbonTableIdentifier.getTableUniqueName))
+ table match {
+ case Some(t) => t.isTransactionalTable
+ case None => true
}
- (timestampFile, timestampFileType)
}
/**
* This method will put the updated timestamp of schema file in the table modified time store map
- *
- * @param timeStamp
*/
- private def updateSchemasUpdatedTime(timeStamp: Long) {
- tableModifiedTimeStore.put(CarbonCommonConstants.DATABASE_DEFAULT_NAME, timeStamp)
- }
-
- def updateAndTouchSchemasUpdatedTime() {
- updateSchemasUpdatedTime(touchSchemaFileSystemTime())
+ private def updateSchemasUpdatedTime(tableUniqueId: String, timeStamp: Long) {
+ tableModifiedTimeStore.put(tableUniqueId, timeStamp)
+ CarbonFileMetastore.updateTableSchemaModifiedTime(tableUniqueId, timeStamp)
}
- /**
- * This method will check and create an empty schema timestamp file
- *
- * @return
- */
- private def touchSchemaFileSystemTime(): Long = {
- val (timestampFile, timestampFileType) = getTimestampFileAndType()
- if (!FileFactory.isFileExist(timestampFile, timestampFileType)) {
- LOGGER.info(s"Creating timestamp file for $timestampFile")
- FileFactory
- .createNewFile(timestampFile,
- timestampFileType,
- true,
- new FsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL))
- }
- FileFactory.getCarbonFile(timestampFile, timestampFileType)
- .setLastModifiedTime(System.currentTimeMillis())
- // since there is no guarantee that exact same set modified time returns when called
- // lastmodified time, so better get the time from file.
- FileFactory.getCarbonFile(timestampFile, timestampFileType)
- .getLastModifiedTime
- }
-
- def checkSchemasModifiedTimeAndReloadTable(tableIdentifier: TableIdentifier): Boolean = {
- val (timestampFile, timestampFileType) = getTimestampFileAndType()
- var isRefreshed = false
- if (FileFactory.isFileExist(timestampFile, timestampFileType)) {
- val lastModifiedTime =
- FileFactory.getCarbonFile(timestampFile, timestampFileType).getLastModifiedTime
- if (!(lastModifiedTime ==
- tableModifiedTimeStore.get(CarbonCommonConstants.DATABASE_DEFAULT_NAME))) {
- metadata.writeLock.lock()
- metadata.carbonTables = metadata.carbonTables.filterNot(
- table => table.getTableName.equalsIgnoreCase(tableIdentifier.table) &&
- table.getDatabaseName
- .equalsIgnoreCase(tableIdentifier.database
- .getOrElse(SparkSession.getActiveSession.get.sessionState.catalog
- .getCurrentDatabase)))
- metadata.writeLock.unlock()
- updateSchemasUpdatedTime(lastModifiedTime)
- isRefreshed = true
+ override def isSchemaRefreshed(absoluteTableIdentifier: AbsoluteTableIdentifier,
+ sparkSession: SparkSession): Boolean = {
+ val localTimeStamp = Option(tableModifiedTimeStore.get(absoluteTableIdentifier
+ .getCarbonTableIdentifier
+ .getTableId))
+ if (localTimeStamp.isDefined) {
+ if (CarbonFileMetastore.checkIfRefreshIsNeeded(absoluteTableIdentifier, localTimeStamp.get)) {
+ sparkSession.sessionState
+ .catalog.refreshTable(TableIdentifier(absoluteTableIdentifier.getTableName,
+ Some(absoluteTableIdentifier.getDatabaseName)))
+ true
+ } else {
+ false
}
+ } else {
+ true
}
- isRefreshed
}
override def isReadFromHiveMetaStore: Boolean = false
override def listAllTables(sparkSession: SparkSession): Seq[CarbonTable] = {
- metadata.readLock.lock
- val ret = metadata.carbonTables.clone()
- metadata.readLock.unlock
- ret
+ CarbonMetadata.getInstance().getAllTables.asScala
}
@@ -672,4 +580,21 @@ class CarbonFileMetastore extends CarbonMetaStore {
case _ => throw new NoSuchTableException(tableIdentifier.database.get, tableIdentifier.table)
}
}
+
+ def removeStaleTimeStampEntries(sparkSession: SparkSession): Unit = {
+ val tablesList = sparkSession.sessionState.catalog.listDatabases().flatMap {
+ database =>
+ sparkSession.sessionState.catalog.listTables(database)
+ .map(table => s"${ database }_${ table.table }".toLowerCase(Locale.getDefault()))
+ }
+ val invalidTableIds = CarbonMetadata.getInstance().getAllTables.asScala.collect {
+ case carbonTable if !tablesList.contains(carbonTable.getTableUniqueName
+ .toLowerCase(Locale.getDefault())) =>
+ CarbonMetadata.getInstance()
+ .removeTable(carbonTable.getTableUniqueName.toLowerCase(Locale.getDefault()))
+ carbonTable.getTableId
+ }
+ CarbonFileMetastore.removeStaleEntries(invalidTableIds.toList)
+ }
+
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetaStore.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetaStore.scala
index c8c7d31..a0a8c0f 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetaStore.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonHiveMetaStore.scala
@@ -63,6 +63,8 @@ class CarbonHiveMetaStore extends CarbonFileMetastore {
carbonRelation
}
+ override def isSchemaRefreshed(absoluteTableIdentifier: AbsoluteTableIdentifier,
+ sparkSession: SparkSession): Boolean = true
override def isTablePathExists(tableIdentifier: TableIdentifier)
(sparkSession: SparkSession): Boolean = {
@@ -78,7 +80,6 @@ class CarbonHiveMetaStore extends CarbonFileMetastore {
// clear driver B-tree and dictionary cache
ManageDictionaryAndBTree.clearBTreeAndDictionaryLRUCache(carbonTable)
}
- checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
CarbonHiveMetadataUtil.invalidateAndDropTable(dbName, tableName, sparkSession)
// discard cached table info in cachedDataSourceTables
val tableIdentifier = TableIdentifier(tableName, Option(dbName))
@@ -88,11 +89,6 @@ class CarbonHiveMetaStore extends CarbonFileMetastore {
removeTableFromMetadata(dbName, tableName)
}
- override def checkSchemasModifiedTimeAndReloadTable(tableIdentifier: TableIdentifier): Boolean = {
- // do nothing
- false
- }
-
override def listAllTables(sparkSession: SparkSession): Seq[CarbonTable] = {
// Todo
Seq()
@@ -140,7 +136,7 @@ class CarbonHiveMetaStore extends CarbonFileMetastore {
* @param carbonTablePath
* @param sparkSession
*/
- override def updateTableSchemaForDataMap(newTableIdentifier: CarbonTableIdentifier,
+ override def updateTableSchema(newTableIdentifier: CarbonTableIdentifier,
oldTableIdentifier: CarbonTableIdentifier,
thriftTableInfo: org.apache.carbondata.format.TableInfo,
carbonTablePath: String)(sparkSession: SparkSession): String = {
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetaStore.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetaStore.scala
index f97a8ae..95efa6a 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetaStore.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonMetaStore.scala
@@ -82,7 +82,7 @@ trait CarbonMetaStore {
* @param carbonStorePath
* @param sparkSession
*/
- def updateTableSchemaForDataMap(newTableIdentifier: CarbonTableIdentifier,
+ def updateTableSchema(newTableIdentifier: CarbonTableIdentifier,
oldTableIdentifier: CarbonTableIdentifier,
thriftTableInfo: org.apache.carbondata.format.TableInfo,
carbonStorePath: String)(sparkSession: SparkSession): String
@@ -135,9 +135,8 @@ trait CarbonMetaStore {
def dropTable(tableIdentifier: AbsoluteTableIdentifier)
(sparkSession: SparkSession)
- def updateAndTouchSchemasUpdatedTime()
-
- def checkSchemasModifiedTimeAndReloadTable(tableIdentifier: TableIdentifier): Boolean
+ def isSchemaRefreshed(absoluteTableIdentifier: AbsoluteTableIdentifier,
+ sparkSession: SparkSession): Boolean
def isReadFromHiveMetaStore: Boolean
@@ -147,8 +146,6 @@ trait CarbonMetaStore {
carbonTable: CarbonTable
): org.apache.carbondata.format.TableInfo
- def getTableFromMetadataCache(database: String, tableName: String): Option[CarbonTable]
-
/**
* Method will be used to retrieve or create carbon data source relation
*
@@ -173,6 +170,7 @@ trait CarbonMetaStore {
val df: DataFrame = Dataset.ofRows(sparkSession, query)
df.schema
}
+
}
/**
* Factory for Carbon metastore
diff --git a/integration/spark2/src/main/scala/org/apache/spark/util/CleanFiles.scala b/integration/spark2/src/main/scala/org/apache/spark/util/CleanFiles.scala
index cb3ae29..a9cb652 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/util/CleanFiles.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/util/CleanFiles.scala
@@ -69,9 +69,6 @@ object CleanFiles {
forceTableClean = args(2).toBoolean
}
val spark = TableAPIUtil.spark(storePath, s"CleanFiles: $dbName.$tableName")
- CarbonEnv.getInstance(spark).carbonMetaStore.
- checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
-
cleanFiles(spark, dbName, tableName, forceTableClean)
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/util/Compaction.scala b/integration/spark2/src/main/scala/org/apache/spark/util/Compaction.scala
index 0a3a870..91203a1 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/util/Compaction.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/util/Compaction.scala
@@ -58,8 +58,6 @@ object Compaction {
val (dbName, tableName) = TableAPIUtil.parseSchemaName(TableAPIUtil.escape(args(1)))
val compactionType = TableAPIUtil.escape(args(2))
val spark = TableAPIUtil.spark(storePath, s"Compaction: $dbName.$tableName")
- CarbonEnv.getInstance(spark).carbonMetaStore.
- checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
compaction(spark, dbName, tableName, compactionType)
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentByDate.scala b/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentByDate.scala
index 90a37f6..6149421 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentByDate.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentByDate.scala
@@ -45,8 +45,6 @@ object DeleteSegmentByDate {
val (dbName, tableName) = TableAPIUtil.parseSchemaName(TableAPIUtil.escape(args(1)))
val dateValue = TableAPIUtil.escape(args(2))
val spark = TableAPIUtil.spark(storePath, s"DeleteSegmentByDate: $dbName.$tableName")
- CarbonEnv.getInstance(spark).carbonMetaStore.
- checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
deleteSegmentByDate(spark, dbName, tableName, dateValue)
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentById.scala b/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentById.scala
index 15bec02..023c7bf 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentById.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/util/DeleteSegmentById.scala
@@ -50,8 +50,6 @@ object DeleteSegmentById {
val (dbName, tableName) = TableAPIUtil.parseSchemaName(TableAPIUtil.escape(args(1)))
val segmentIds = extractSegmentIds(TableAPIUtil.escape(args(2)))
val spark = TableAPIUtil.spark(storePath, s"DeleteSegmentById: $dbName.$tableName")
- CarbonEnv.getInstance(spark).carbonMetaStore.
- checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
deleteSegmentById(spark, dbName, tableName, segmentIds)
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala b/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
index efaa191..ef9a931 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/util/TableLoader.scala
@@ -82,8 +82,6 @@ object TableLoader {
val spark = TableAPIUtil.spark(storePath, s"TableLoader: $dbName.$tableName")
- CarbonEnv.getInstance(spark).carbonMetaStore.
- checkSchemasModifiedTimeAndReloadTable(TableIdentifier(tableName, Some(dbName)))
loadTable(spark, Option(dbName), tableName, inputPaths, map)
}
diff --git a/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala b/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
index 26f778e..75e7d89 100644
--- a/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
+++ b/integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala
@@ -125,9 +125,9 @@ class CarbonHiveSessionCatalog(
rtnRelation match {
case SubqueryAlias(_,
LogicalRelation(carbonDatasourceHadoopRelation: CarbonDatasourceHadoopRelation, _, _), _) =>
- toRefreshRelation = refreshRelationFromCache(name, alias, carbonDatasourceHadoopRelation)
- case LogicalRelation(carbonDatasourceHadoopRelation: CarbonDatasourceHadoopRelation, _, _) =>
- toRefreshRelation = refreshRelationFromCache(name, alias, carbonDatasourceHadoopRelation)
+ toRefreshRelation = CarbonEnv.isRefreshRequired(name)(sparkSession)
+ case LogicalRelation(_: CarbonDatasourceHadoopRelation, _, _) =>
+ toRefreshRelation = CarbonEnv.isRefreshRequired(name)(sparkSession)
case _ =>
}
@@ -138,31 +138,6 @@ class CarbonHiveSessionCatalog(
}
}
- private def refreshRelationFromCache(identifier: TableIdentifier,
- alias: Option[String],
- carbonDatasourceHadoopRelation: CarbonDatasourceHadoopRelation): Boolean = {
- var isRefreshed = false
- val storePath = CarbonProperties.getStorePath
- carbonEnv.carbonMetaStore.
- checkSchemasModifiedTimeAndReloadTable(identifier)
-
- val table = carbonEnv.carbonMetaStore.getTableFromMetadataCache(
- carbonDatasourceHadoopRelation.carbonTable.getDatabaseName,
- carbonDatasourceHadoopRelation.carbonTable.getTableName)
- if (table.isEmpty || (table.isDefined &&
- table.get.getTableLastUpdatedTime !=
- carbonDatasourceHadoopRelation.carbonTable.getTableLastUpdatedTime)) {
- refreshTable(identifier)
- DataMapStoreManager.getInstance().
- clearDataMaps(AbsoluteTableIdentifier.from(storePath,
- identifier.database.getOrElse("default"),
- identifier.table))
- isRefreshed = true
- logInfo(s"Schema changes have been detected for table: $identifier")
- }
- isRefreshed
- }
-
/**
* returns hive client from session state
*
diff --git a/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AlterTableColumnRenameTestCase.scala b/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AlterTableColumnRenameTestCase.scala
index dd1fa0f..d368a8e 100644
--- a/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AlterTableColumnRenameTestCase.scala
+++ b/integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AlterTableColumnRenameTestCase.scala
@@ -133,10 +133,10 @@ class AlterTableColumnRenameTestCase extends Spark2QueryTest with BeforeAndAfter
sql("alter table rename change empname name string")
sql("update rename set (name) = ('joey') where workgroupcategory = 'developer'").show()
sql("insert into rename select 20,'bill','PM','01-12-2015',3,'manager',14,'Learning',928479,'01-01-2016','30-11-2016',75,94,13547")
- val df1 = sql("select * from rename where name = 'joey'")
+ val df1Count = sql("select * from rename where name = 'joey'").count
sql("alter table rename change name empname string")
val df2 = sql("select * from rename where empname = 'joey'")
- assert(df1.count() == df2.count())
+ assert(df1Count == df2.count())
sql("delete from rename where empname = 'joey'")
val df3 = sql("select empname from rename")
sql("alter table rename change empname newname string")
[carbondata] 08/33: [CARBONDATA-3499] Fix insert failure with
customFileProvider
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 1943ddae258c754ab11c466557877378fb9a748e
Author: ajantha-bhat <aj...@gmail.com>
AuthorDate: Thu Aug 22 17:42:12 2019 +0530
[CARBONDATA-3499] Fix insert failure with customFileProvider
Problem:
Below exception is thrown when the custom file system is used with
first time insert randomly. IllegalArgumentException("Path belongs to
unsupported file system") from FileFactory.getFileType()
Cause:
DefaultFileTypeProvider.initializeCustomFileProvider is called concurrently
during insert. Hence one thread got the provider and other thread didn't get
as flag is set to true. so other thread failed as it tried with default provider.
Solution:
synchronize the initialization of custom file provider.
This closes #3362
---
.../datastore/impl/DefaultFileTypeProvider.java | 31 +++++++++++++---------
1 file changed, 19 insertions(+), 12 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/datastore/impl/DefaultFileTypeProvider.java b/core/src/main/java/org/apache/carbondata/core/datastore/impl/DefaultFileTypeProvider.java
index cdb1a20..4572cc4 100644
--- a/core/src/main/java/org/apache/carbondata/core/datastore/impl/DefaultFileTypeProvider.java
+++ b/core/src/main/java/org/apache/carbondata/core/datastore/impl/DefaultFileTypeProvider.java
@@ -43,7 +43,9 @@ public class DefaultFileTypeProvider implements FileTypeInterface {
*/
protected FileTypeInterface customFileTypeProvider = null;
- protected boolean customFileTypeProviderInitialized = false;
+ protected Boolean customFileTypeProviderInitialized = false;
+
+ private final Object lock = new Object();
public DefaultFileTypeProvider() {
}
@@ -52,17 +54,22 @@ public class DefaultFileTypeProvider implements FileTypeInterface {
* This method is required apart from Constructor to handle the below circular dependency.
* CarbonProperties-->FileFactory-->DefaultTypeProvider-->CarbonProperties
*/
- private void initializeCustomFileprovider() {
+ private void initializeCustomFileProvider() {
if (!customFileTypeProviderInitialized) {
- customFileTypeProviderInitialized = true;
- String customFileProvider =
- CarbonProperties.getInstance().getProperty(CarbonCommonConstants.CUSTOM_FILE_PROVIDER);
- if (customFileProvider != null && !customFileProvider.trim().isEmpty()) {
- try {
- customFileTypeProvider =
- (FileTypeInterface) Class.forName(customFileProvider).newInstance();
- } catch (Exception e) {
- LOGGER.error("Unable load configured FileTypeInterface class. Ignored.", e);
+ // This initialization can happen in concurrent threads.
+ synchronized (lock) {
+ if (!customFileTypeProviderInitialized) {
+ String customFileProvider = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CUSTOM_FILE_PROVIDER);
+ if (customFileProvider != null && !customFileProvider.trim().isEmpty()) {
+ try {
+ customFileTypeProvider =
+ (FileTypeInterface) Class.forName(customFileProvider).newInstance();
+ } catch (Exception e) {
+ LOGGER.error("Unable load configured FileTypeInterface class. Ignored.", e);
+ }
+ customFileTypeProviderInitialized = true;
+ }
}
}
}
@@ -77,7 +84,7 @@ public class DefaultFileTypeProvider implements FileTypeInterface {
* @return true if supported by the custom
*/
@Override public boolean isPathSupported(String path) {
- initializeCustomFileprovider();
+ initializeCustomFileProvider();
if (customFileTypeProvider != null) {
return customFileTypeProvider.isPathSupported(path);
}
[carbondata] 23/33: [CARBONDATA-3489] Optimized the comparator
instances in sort
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 9e4664742f04b4995bd8a2ecd465e8797ce7c2d2
Author: Vikram Ahuja <vi...@gmail.com>
AuthorDate: Tue Aug 6 12:21:11 2019 +0530
[CARBONDATA-3489] Optimized the comparator instances in sort
Root cause: In case of sorting in the comparator classes(NewRowComparator,
RawRowComparator, IntermediateSortTempRowComparator and UnsafeRowComparator)
a new SerializableComparator object is been created in the compare method
everytime two objects are passed for comparison.
Solution: We can reduce the number of SerializeableComparator objects
that are been created by storing the SerializeableComparators of
primitive datatypes in a map and getting it from the map instead of
creating a new SerializeableComparator everytime.
This closes #3354
---
.../core/util/comparator/Comparator.java | 49 +++---
.../partition/impl/RawRowComparatorTest.java | 142 ++++++++++++++++
.../IntermediateSortTempRowComparatorTest.java | 178 +++++++++++++++++++++
.../sort/sortdata/NewRowComparatorTest.java | 109 +++++++++++++
4 files changed, 453 insertions(+), 25 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/util/comparator/Comparator.java b/core/src/main/java/org/apache/carbondata/core/util/comparator/Comparator.java
index 6981405..d7e8f80 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/comparator/Comparator.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/comparator/Comparator.java
@@ -25,24 +25,23 @@ import org.apache.carbondata.core.util.ByteUtil;
public final class Comparator {
+ //Comparators are made static so that only one instance is generated
+ private static final SerializableComparator BOOLEAN = new BooleanSerializableComparator();
+ private static final SerializableComparator INT = new IntSerializableComparator();
+ private static final SerializableComparator SHORT = new ShortSerializableComparator();
+ private static final SerializableComparator DOUBLE = new DoubleSerializableComparator();
+ private static final SerializableComparator FLOAT = new FloatSerializableComparator();
+ private static final SerializableComparator LONG = new LongSerializableComparator();
+ private static final SerializableComparator DECIMAL = new BigDecimalSerializableComparator();
+ private static final SerializableComparator BYTE = new ByteArraySerializableComparator();
+
public static SerializableComparator getComparator(DataType dataType) {
- if (dataType == DataTypes.BOOLEAN) {
- return new BooleanSerializableComparator();
- } else if (dataType == DataTypes.INT) {
- return new IntSerializableComparator();
- } else if (dataType == DataTypes.SHORT) {
- return new ShortSerializableComparator();
- } else if (dataType == DataTypes.DOUBLE) {
- return new DoubleSerializableComparator();
- } else if (dataType == DataTypes.FLOAT) {
- return new FloatSerializableComparator();
- } else if (dataType == DataTypes.LONG || dataType == DataTypes.DATE
- || dataType == DataTypes.TIMESTAMP) {
- return new LongSerializableComparator();
- } else if (DataTypes.isDecimal(dataType)) {
- return new BigDecimalSerializableComparator();
+ if (dataType == DataTypes.DATE || dataType == DataTypes.TIMESTAMP) {
+ return LONG;
+ } else if (dataType == DataTypes.STRING) {
+ return BYTE;
} else {
- return new ByteArraySerializableComparator();
+ return getComparatorByDataTypeForMeasure(dataType);
}
}
@@ -54,21 +53,21 @@ public final class Comparator {
*/
public static SerializableComparator getComparatorByDataTypeForMeasure(DataType dataType) {
if (dataType == DataTypes.BOOLEAN) {
- return new BooleanSerializableComparator();
+ return BOOLEAN;
} else if (dataType == DataTypes.INT) {
- return new IntSerializableComparator();
+ return INT;
} else if (dataType == DataTypes.SHORT) {
- return new ShortSerializableComparator();
+ return SHORT;
} else if (dataType == DataTypes.LONG) {
- return new LongSerializableComparator();
+ return LONG;
} else if (dataType == DataTypes.DOUBLE) {
- return new DoubleSerializableComparator();
+ return DOUBLE;
} else if (dataType == DataTypes.FLOAT) {
- return new FloatSerializableComparator();
+ return FLOAT;
} else if (DataTypes.isDecimal(dataType)) {
- return new BigDecimalSerializableComparator();
+ return DECIMAL;
} else if (dataType == DataTypes.BYTE) {
- return new ByteArraySerializableComparator();
+ return BYTE;
} else {
throw new IllegalArgumentException("Unsupported data type: " + dataType.getName());
}
@@ -198,4 +197,4 @@ class BigDecimalSerializableComparator implements SerializableComparator {
}
return ((BigDecimal) key1).compareTo((BigDecimal) key2);
}
-}
\ No newline at end of file
+}
diff --git a/processing/src/test/java/org/apache/carbondata/processing/loading/partition/impl/RawRowComparatorTest.java b/processing/src/test/java/org/apache/carbondata/processing/loading/partition/impl/RawRowComparatorTest.java
new file mode 100644
index 0000000..ba0fce0
--- /dev/null
+++ b/processing/src/test/java/org/apache/carbondata/processing/loading/partition/impl/RawRowComparatorTest.java
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.processing.loading.partition.impl;
+
+import org.apache.carbondata.core.datastore.row.CarbonRow;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.util.comparator.SerializableComparator;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class RawRowComparatorTest {
+ private RawRowComparator new_comparator;
+
+ @Test public void checkTypes() {
+ DataType noDicDataTypes[] =
+ { DataTypes.INT, DataTypes.SHORT, DataTypes.LONG, DataTypes.BOOLEAN, DataTypes.BYTE,
+ DataTypes.FLOAT };
+ for (int i = 0; i < noDicDataTypes.length; i++) {
+ SerializableComparator comparator = org.apache.carbondata.core.util.comparator.Comparator
+ .getComparator(noDicDataTypes[i]);
+ SerializableComparator comparator1 = org.apache.carbondata.core.util.comparator.Comparator
+ .getComparator(noDicDataTypes[i]);
+ Assert.assertTrue(comparator1==comparator);
+ }
+ }
+
+ @Test public void compareint() {
+ DataType noDicDataTypes[] = { DataTypes.INT };
+ boolean noDicSortColumnMapping[] = { true };
+ int sortColumnIndices[] = { 1 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+
+ Integer arr[] = { 1, 7, 5 };
+ Integer arr1[] = { 2, 4, 6 };
+ int res = new_comparator.compare(new CarbonRow(arr), new CarbonRow(arr1));
+ Assert.assertTrue(res > 0);
+ }
+
+ @Test public void compareintreverse() {
+ DataType noDicDataTypes[] = { DataTypes.INT };
+ boolean noDicSortColumnMapping[] = { true };
+ int sortColumnIndices[] = { 1 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+
+ Integer arr[] = { 2, 7, 5 };
+ Integer arr1[] = { 1, 4, 6 };
+ int res = new_comparator.compare(new CarbonRow(arr1), new CarbonRow(arr));
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void compareshort() {
+ DataType noDicDataTypes[] = { DataTypes.SHORT };
+ boolean noDicSortColumnMapping[] = { true };
+ int sortColumnIndices[] = { 1 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+
+ Short arr[] = { 1, 7, 5 };
+ Short arr1[] = { 2, 4, 6 };
+ int res = new_comparator.compare(new CarbonRow(arr), new CarbonRow(arr1));
+ Assert.assertTrue(res > 0);
+ }
+
+ @Test public void comparelong() {
+ DataType noDicDataTypes[] = { DataTypes.LONG };
+ boolean noDicSortColumnMapping[] = { true };
+ int sortColumnIndices[] = { 1 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+ Long arr[] = { 1L, 7L, 5L };
+ Long arr1[] = { 2L, 4L, 6L };
+ int res = new_comparator.compare(new CarbonRow(arr), new CarbonRow(arr1));
+ Assert.assertTrue(res > 0);
+ }
+
+ @Test public void comparefloat() {
+ DataType noDicDataTypes[] = { DataTypes.FLOAT };
+ boolean noDicSortColumnMapping[] = { true };
+ int sortColumnIndices[] = { 1 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+ Float arr[] = { 1F, 7F, 5F };
+ Float arr1[] = { 2F, 4F, 6F };
+ int res = new_comparator.compare(new CarbonRow(arr), new CarbonRow(arr1));
+ Assert.assertTrue(res > 0);
+ }
+
+ @Test public void compareboolean() {
+ DataType noDicDataTypes[] = { DataTypes.BOOLEAN };
+ boolean noDicSortColumnMapping[] = { true };
+ int sortColumnIndices[] = { 1 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+ Boolean arr[] = { false, false };
+ Boolean arr1[] = { true, true };
+ int res = new_comparator.compare(new CarbonRow(arr), new CarbonRow(arr1));
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparebyte() {
+ DataType noDicDataTypes[] = { DataTypes.BYTE };
+ boolean noDicSortColumnMapping[] = { true };
+ int sortColumnIndices[] = { 1 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+ Byte arr[] = { 1, 2, 3 };
+ Byte arr1[] = { 4, 5, 6 };
+ int res = new_comparator.compare(new CarbonRow(arr), new CarbonRow(arr1));
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparemixed() {
+ DataType noDicDataTypes[] = { DataTypes.INT, DataTypes.SHORT, DataTypes.BOOLEAN };
+ boolean noDicSortColumnMapping[] = { true, true, true };
+ int sortColumnIndices[] = { 1, 2, 3 };
+ new_comparator =
+ new RawRowComparator(sortColumnIndices, noDicSortColumnMapping, noDicDataTypes);
+ Object arr[] = { 1, 2, false };
+ Object arr1[] = { 4, 5, true };
+ int res = new_comparator.compare(new CarbonRow(arr), new CarbonRow(arr1));
+ Assert.assertTrue(res < 0);
+ }
+}
diff --git a/processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/IntermediateSortTempRowComparatorTest.java b/processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/IntermediateSortTempRowComparatorTest.java
new file mode 100644
index 0000000..a0d1137
--- /dev/null
+++ b/processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/IntermediateSortTempRowComparatorTest.java
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.processing.sort.sortdata;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.processing.loading.row.IntermediateSortTempRow;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class IntermediateSortTempRowComparatorTest {
+ private IntermediateSortTempRowComparator new_comparator;
+
+ @Test public void compareint() {
+ DataType noDicDataTypes[] = { DataTypes.INT };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Object[] noDictSortDims1 = {1,2,3};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Object[] noDictSortDims = {4,5,6};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+
+ int res = new_comparator.compare( a, a1);
+ Assert.assertTrue(res > 0);
+ }
+
+ @Test public void compareintreverse() {
+ DataType noDicDataTypes[] = { DataTypes.INT };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Object[] noDictSortDims1 = {1,2,3};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Object[] noDictSortDims = {4,5,6};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+
+ int res = new_comparator.compare( a1, a);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparelong() {
+ DataType noDicDataTypes[] = { DataTypes.LONG };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Object[] noDictSortDims1 = {1L,2L,3L};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Object[] noDictSortDims = {4L,5L,6L};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+
+ int res = new_comparator.compare( a1, a);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparefloat() {
+ DataType noDicDataTypes[] = { DataTypes.FLOAT };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Object[] noDictSortDims1 = {1F,2F,3F};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Object[] noDictSortDims = {4F,5F,6F};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+
+ int res = new_comparator.compare( a1, a);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void compareboolean() {
+ DataType noDicDataTypes[] = { DataTypes.BOOLEAN};
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Object[] noDictSortDims1 = {true};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Object[] noDictSortDims = {false};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+ int res = new_comparator.compare( a1, a);
+ Assert.assertTrue(res> 0);
+ }
+
+ @Test public void comparebyte() {
+ DataType noDicDataTypes[] = { DataTypes.BYTE};
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Byte[] noDictSortDims1 = {1,2,3};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Byte[] noDictSortDims = {3,4,5};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+ int res = new_comparator.compare( a1, a);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparemixed() {
+ DataType noDicDataTypes[] = { DataTypes.BOOLEAN, DataTypes.INT, DataTypes.LONG};
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Object[] noDictSortDims1 = {true, 1, 2L};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Object[] noDictSortDims = {false, 3, 5L};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+ int res = new_comparator.compare( a1, a);
+ Assert.assertTrue(res > 0);
+ }
+
+ @Test public void compareshort() {
+ DataType noDicDataTypes[] = { DataTypes.SHORT};
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new IntermediateSortTempRowComparator(noDicSortColumnMapping, noDicDataTypes);
+
+ int[] dictSortDims1 = {1,2,3};
+ Short[] noDictSortDims1 = {2};
+ byte[] noSortDimsAndMeasures1 = {1,2,3};
+ IntermediateSortTempRow a1 = new IntermediateSortTempRow(dictSortDims1, noDictSortDims1, noSortDimsAndMeasures1);
+
+ int[] dictSortDims = {1,2,3};
+ Short[] noDictSortDims = {1};
+ byte[] noSortDimsAndMeasures = {1,2,3};
+ IntermediateSortTempRow a = new IntermediateSortTempRow(dictSortDims, noDictSortDims, noSortDimsAndMeasures);
+
+ int res = new_comparator.compare( a1, a);
+ Assert.assertTrue(res > 0);
+ }
+}
diff --git a/processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/NewRowComparatorTest.java b/processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/NewRowComparatorTest.java
new file mode 100644
index 0000000..c2bd860
--- /dev/null
+++ b/processing/src/test/java/org/apache/carbondata/processing/sort/sortdata/NewRowComparatorTest.java
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.processing.sort.sortdata;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.util.comparator.SerializableComparator;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class NewRowComparatorTest {
+ private NewRowComparator new_comparator;
+
+ @Test public void compareint() {
+ DataType noDicDataTypes[] = { DataTypes.INT };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Integer arr[] = { 1, 7, 5 };
+ Integer arr1[] = { 2, 4, 6 };
+ int res = new_comparator.compare(arr, arr1);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void compareintreverse() {
+ DataType noDicDataTypes[] = { DataTypes.INT };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Integer arr[] = { 1, 2, 3 };
+ Integer arr1[] = { 4, 5, 6 };
+ int res = new_comparator.compare(arr1, arr);
+ Assert.assertTrue(res > 0);
+ }
+
+ @Test public void compareshort() {
+ DataType noDicDataTypes[] = { DataTypes.SHORT };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Short arr[] = { 1, 2, 3 };
+ Short arr1[] = { 4, 5, 6 };
+ int res = new_comparator.compare(arr, arr1);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparelong() {
+ DataType noDicDataTypes[] = { DataTypes.LONG };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Long arr[] = { 1L, 2L, 3L };
+ Long arr1[] = { 4L, 5L, 6L };
+ int res = new_comparator.compare(arr, arr1);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparefloat() {
+ DataType noDicDataTypes[] = { DataTypes.FLOAT };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Float arr[] = { 1F, 2F, 3F };
+ Float arr1[] = { 4F, 5F, 6F };
+ int res = new_comparator.compare(arr, arr1);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void compareboolean() {
+ DataType noDicDataTypes[] = { DataTypes.BOOLEAN };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Boolean arr[] = { false };
+ Boolean arr1[] = { true };
+ int res = new_comparator.compare(arr, arr1);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparebyte() {
+ DataType noDicDataTypes[] = { DataTypes.BYTE };
+ boolean noDicSortColumnMapping[] = { true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Byte arr[] = { 1, 2, 3 };
+ Byte arr1[] = { 4, 5, 6 };
+ int res = new_comparator.compare(arr, arr1);
+ Assert.assertTrue(res < 0);
+ }
+
+ @Test public void comparemixed() {
+ DataType noDicDataTypes[] = { DataTypes.INT, DataTypes.FLOAT, DataTypes.BOOLEAN };
+ boolean noDicSortColumnMapping[] = { true, true, true };
+ new_comparator = new NewRowComparator(noDicSortColumnMapping, noDicDataTypes);
+ Object arr[] = { 1, 2F, false };
+ Object arr1[] = { 4, 5F, true };
+ int res = new_comparator.compare(arr, arr1);
+ Assert.assertTrue(res < 0);
+ }
+}
[carbondata] 21/33: [CARBONDATA-3454] optimized index server output
for count(*)
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 43a8086e15ec34648c02f9022975e683fc139f25
Author: kunal642 <ku...@gmail.com>
AuthorDate: Thu Jun 27 14:32:11 2019 +0530
[CARBONDATA-3454] optimized index server output for count(*)
Optimised the output for count(*) queries so that only a long is send back to the driver to reduce the network transfer cost for index server
This closes #3308
---
.../apache/carbondata/core/datamap/DataMapJob.java | 2 +
.../carbondata/core/datamap/DataMapUtil.java | 13 ++-
.../core/datamap/DistributableDataMapFormat.java | 34 +++++--
.../core/indexstore/ExtendedBlocklet.java | 68 ++++++++-----
.../core/indexstore/ExtendedBlockletWrapper.java | 27 +++--
.../ExtendedBlockletWrapperContainer.java | 19 ++--
.../carbondata/hadoop/api/CarbonInputFormat.java | 52 ++++++++--
.../hadoop/api/CarbonTableInputFormat.java | 22 ++--
.../carbondata/indexserver/DataMapJobs.scala | 15 ++-
.../indexserver/DistributedCountRDD.scala | 111 +++++++++++++++++++++
.../indexserver/DistributedPruneRDD.scala | 29 ++----
.../indexserver/DistributedRDDUtils.scala | 13 +++
.../carbondata/indexserver/IndexServer.scala | 19 ++++
13 files changed, 319 insertions(+), 105 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapJob.java b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapJob.java
index 9eafe7c..326282d 100644
--- a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapJob.java
+++ b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapJob.java
@@ -35,4 +35,6 @@ public interface DataMapJob extends Serializable {
List<ExtendedBlocklet> execute(DistributableDataMapFormat dataMapFormat);
+ Long executeCountJob(DistributableDataMapFormat dataMapFormat);
+
}
diff --git a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapUtil.java b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapUtil.java
index dd9debc..bca7409 100644
--- a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapUtil.java
@@ -230,7 +230,7 @@ public class DataMapUtil {
List<Segment> validSegments, List<Segment> invalidSegments, DataMapLevel level,
List<String> segmentsToBeRefreshed) throws IOException {
return executeDataMapJob(carbonTable, resolver, dataMapJob, partitionsToPrune, validSegments,
- invalidSegments, level, false, segmentsToBeRefreshed);
+ invalidSegments, level, false, segmentsToBeRefreshed, false);
}
/**
@@ -241,7 +241,8 @@ public class DataMapUtil {
public static List<ExtendedBlocklet> executeDataMapJob(CarbonTable carbonTable,
FilterResolverIntf resolver, DataMapJob dataMapJob, List<PartitionSpec> partitionsToPrune,
List<Segment> validSegments, List<Segment> invalidSegments, DataMapLevel level,
- Boolean isFallbackJob, List<String> segmentsToBeRefreshed) throws IOException {
+ Boolean isFallbackJob, List<String> segmentsToBeRefreshed, boolean isCountJob)
+ throws IOException {
List<String> invalidSegmentNo = new ArrayList<>();
for (Segment segment : invalidSegments) {
invalidSegmentNo.add(segment.getSegmentNo());
@@ -250,9 +251,11 @@ public class DataMapUtil {
DistributableDataMapFormat dataMapFormat =
new DistributableDataMapFormat(carbonTable, resolver, validSegments, invalidSegmentNo,
partitionsToPrune, false, level, isFallbackJob);
- List<ExtendedBlocklet> prunedBlocklets = dataMapJob.execute(dataMapFormat);
- // Apply expression on the blocklets.
- return prunedBlocklets;
+ if (isCountJob) {
+ dataMapFormat.setCountStarJob();
+ dataMapFormat.setIsWriteToFile(false);
+ }
+ return dataMapJob.execute(dataMapFormat);
}
public static SegmentStatusManager.ValidAndInvalidSegmentsInfo getValidAndInvalidSegments(
diff --git a/core/src/main/java/org/apache/carbondata/core/datamap/DistributableDataMapFormat.java b/core/src/main/java/org/apache/carbondata/core/datamap/DistributableDataMapFormat.java
index 8426fcb..b430c5d 100644
--- a/core/src/main/java/org/apache/carbondata/core/datamap/DistributableDataMapFormat.java
+++ b/core/src/main/java/org/apache/carbondata/core/datamap/DistributableDataMapFormat.java
@@ -28,7 +28,6 @@ import java.util.UUID;
import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datamap.dev.DataMap;
import org.apache.carbondata.core.datamap.dev.expr.DataMapDistributableWrapper;
import org.apache.carbondata.core.datastore.impl.FileFactory;
import org.apache.carbondata.core.indexstore.ExtendedBlocklet;
@@ -91,6 +90,8 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
private boolean isWriteToFile = true;
+ private boolean isCountStarJob = false;
+
DistributableDataMapFormat() {
}
@@ -103,7 +104,7 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
this.dataMapToClear = dataMapToClear;
}
- DistributableDataMapFormat(CarbonTable table, FilterResolverIntf filterResolverIntf,
+ public DistributableDataMapFormat(CarbonTable table, FilterResolverIntf filterResolverIntf,
List<Segment> validSegments, List<String> invalidSegments, List<PartitionSpec> partitions,
boolean isJobToClearDataMaps, DataMapLevel dataMapLevel, boolean isFallbackJob)
throws IOException {
@@ -136,7 +137,6 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
return new RecordReader<Void, ExtendedBlocklet>() {
private Iterator<ExtendedBlocklet> blockletIterator;
private ExtendedBlocklet currBlocklet;
- private List<DataMap> dataMaps;
@Override
public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptContext)
@@ -149,7 +149,6 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
if (dataMapLevel == null) {
TableDataMap defaultDataMap = DataMapStoreManager.getInstance()
.getDataMap(table, distributable.getDistributable().getDataMapSchema());
- dataMaps = defaultDataMap.getTableDataMaps(distributable.getDistributable());
blocklets = defaultDataMap
.prune(segmentsToLoad, new DataMapFilter(filterResolverIntf), partitions);
blocklets = DataMapUtil
@@ -192,11 +191,6 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
@Override
public void close() throws IOException {
- if (null != dataMaps) {
- for (DataMap dataMap : dataMaps) {
- dataMap.finish();
- }
- }
}
};
}
@@ -247,6 +241,7 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
out.writeUTF(taskGroupDesc);
out.writeUTF(queryId);
out.writeBoolean(isWriteToFile);
+ out.writeBoolean(isCountStarJob);
}
@Override
@@ -292,6 +287,7 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
this.taskGroupDesc = in.readUTF();
this.queryId = in.readUTF();
this.isWriteToFile = in.readBoolean();
+ this.isCountStarJob = in.readBoolean();
}
private void initReadCommittedScope() throws IOException {
@@ -398,9 +394,29 @@ public class DistributableDataMapFormat extends FileInputFormat<Void, ExtendedBl
return validSegments;
}
+ public List<Segment> getValidSegments() {
+ return validSegments;
+ }
+
public void createDataMapChooser() throws IOException {
if (null != filterResolverIntf) {
this.dataMapChooser = new DataMapChooser(table);
}
}
+
+ public void setCountStarJob() {
+ this.isCountStarJob = true;
+ }
+
+ public boolean isCountStarJob() {
+ return this.isCountStarJob;
+ }
+
+ public List<PartitionSpec> getPartitions() {
+ return partitions;
+ }
+
+ public ReadCommittedScope getReadCommittedScope() {
+ return readCommittedScope;
+ }
}
diff --git a/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlocklet.java b/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlocklet.java
index a85423b..611e969 100644
--- a/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlocklet.java
+++ b/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlocklet.java
@@ -39,6 +39,10 @@ public class ExtendedBlocklet extends Blocklet {
private CarbonInputSplit inputSplit;
+ private Long count;
+
+ private String segmentNo;
+
public ExtendedBlocklet() {
}
@@ -78,6 +82,9 @@ public class ExtendedBlocklet extends Blocklet {
}
public String getSegmentId() {
+ if (segmentNo != null) {
+ return segmentNo;
+ }
return this.inputSplit.getSegmentId();
}
@@ -92,8 +99,12 @@ public class ExtendedBlocklet extends Blocklet {
return getFilePath();
}
- public String getDataMapWriterPath() {
- return this.inputSplit.getDataMapWritePath();
+ public Long getRowCount() {
+ if (count != null) {
+ return count;
+ } else {
+ return (long) inputSplit.getRowCount();
+ }
}
public void setDataMapWriterPath(String dataMapWriterPath) {
@@ -161,30 +172,35 @@ public class ExtendedBlocklet extends Blocklet {
* @param uniqueLocation
* @throws IOException
*/
- public void serializeData(DataOutput out, Map<String, Short> uniqueLocation)
+ public void serializeData(DataOutput out, Map<String, Short> uniqueLocation, boolean isCountJob)
throws IOException {
super.write(out);
- if (dataMapUniqueId == null) {
- out.writeBoolean(false);
+ if (isCountJob) {
+ out.writeLong(inputSplit.getRowCount());
+ out.writeUTF(inputSplit.getSegmentId());
} else {
- out.writeBoolean(true);
- out.writeUTF(dataMapUniqueId);
- }
- out.writeBoolean(inputSplit != null);
- if (inputSplit != null) {
- // creating byte array output stream to get the size of input split serializeData size
- ExtendedByteArrayOutputStream ebos = new ExtendedByteArrayOutputStream();
- DataOutputStream dos = new DataOutputStream(ebos);
- inputSplit.setFilePath(null);
- inputSplit.setBucketId(null);
- if (inputSplit.isBlockCache()) {
- inputSplit.updateFooteroffset();
- inputSplit.updateBlockLength();
- inputSplit.setWriteDetailInfo(false);
+ if (dataMapUniqueId == null) {
+ out.writeBoolean(false);
+ } else {
+ out.writeBoolean(true);
+ out.writeUTF(dataMapUniqueId);
+ }
+ out.writeBoolean(inputSplit != null);
+ if (inputSplit != null) {
+ // creating byte array output stream to get the size of input split serializeData size
+ ExtendedByteArrayOutputStream ebos = new ExtendedByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(ebos);
+ inputSplit.setFilePath(null);
+ inputSplit.setBucketId(null);
+ if (inputSplit.isBlockCache()) {
+ inputSplit.updateFooteroffset();
+ inputSplit.updateBlockLength();
+ inputSplit.setWriteDetailInfo(false);
+ }
+ inputSplit.serializeFields(dos, uniqueLocation);
+ out.writeInt(ebos.size());
+ out.write(ebos.getBuffer(), 0, ebos.size());
}
- inputSplit.serializeFields(dos, uniqueLocation);
- out.writeInt(ebos.size());
- out.write(ebos.getBuffer(), 0 , ebos.size());
}
}
@@ -195,9 +211,15 @@ public class ExtendedBlocklet extends Blocklet {
* @param tablePath
* @throws IOException
*/
- public void deserializeFields(DataInput in, String[] locations, String tablePath)
+ public void deserializeFields(DataInput in, String[] locations, String tablePath,
+ boolean isCountJob)
throws IOException {
super.readFields(in);
+ if (isCountJob) {
+ count = in.readLong();
+ segmentNo = in.readUTF();
+ return;
+ }
if (in.readBoolean()) {
dataMapUniqueId = in.readUTF();
}
diff --git a/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapper.java b/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapper.java
index ab051ea..f722f32 100644
--- a/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapper.java
+++ b/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapper.java
@@ -56,25 +56,20 @@ public class ExtendedBlockletWrapper implements Writable, Serializable {
private static final Logger LOGGER =
LogServiceFactory.getLogService(ExtendedBlockletWrapper.class.getName());
-
+ private static final int BUFFER_SIZE = 8 * 1024 * 1024;
+ private static final int BLOCK_SIZE = 256 * 1024 * 1024;
private boolean isWrittenToFile;
-
private int dataSize;
-
private byte[] bytes;
- private static final int BUFFER_SIZE = 8 * 1024 * 1024;
-
- private static final int BLOCK_SIZE = 256 * 1024 * 1024;
-
public ExtendedBlockletWrapper() {
}
public ExtendedBlockletWrapper(List<ExtendedBlocklet> extendedBlockletList, String tablePath,
- String queryId, boolean isWriteToFile) {
+ String queryId, boolean isWriteToFile, boolean isCountJob) {
Map<String, Short> uniqueLocations = new HashMap<>();
- byte[] bytes = convertToBytes(tablePath, uniqueLocations, extendedBlockletList);
+ byte[] bytes = convertToBytes(tablePath, uniqueLocations, extendedBlockletList, isCountJob);
int serializeAllowedSize = Integer.parseInt(CarbonProperties.getInstance()
.getProperty(CarbonCommonConstants.CARBON_INDEX_SERVER_SERIALIZATION_THRESHOLD,
CarbonCommonConstants.CARBON_INDEX_SERVER_SERIALIZATION_THRESHOLD_DEFAULT)) * 1024;
@@ -122,13 +117,13 @@ public class ExtendedBlockletWrapper implements Writable, Serializable {
}
private byte[] convertToBytes(String tablePath, Map<String, Short> uniqueLocations,
- List<ExtendedBlocklet> extendedBlockletList) {
+ List<ExtendedBlocklet> extendedBlockletList, boolean isCountJob) {
ByteArrayOutputStream bos = new ExtendedByteArrayOutputStream();
DataOutputStream stream = new DataOutputStream(bos);
try {
for (ExtendedBlocklet extendedBlocklet : extendedBlockletList) {
extendedBlocklet.setFilePath(extendedBlocklet.getFilePath().replace(tablePath, ""));
- extendedBlocklet.serializeData(stream, uniqueLocations);
+ extendedBlocklet.serializeData(stream, uniqueLocations, isCountJob);
}
return new SnappyCompressor().compressByte(bos.toByteArray());
} catch (IOException e) {
@@ -142,6 +137,7 @@ public class ExtendedBlockletWrapper implements Writable, Serializable {
* Below method will be used to write the data to stream[file/memory]
* Data Format
* <number of splits><number of unique location[short]><locations><serialize data len><data>
+ *
* @param stream
* @param data
* @param uniqueLocation
@@ -158,7 +154,7 @@ public class ExtendedBlockletWrapper implements Writable, Serializable {
final Map.Entry<String, Short> next = iterator.next();
uniqueLoc[next.getValue()] = next.getKey();
}
- stream.writeShort((short)uniqueLoc.length);
+ stream.writeShort((short) uniqueLoc.length);
for (String loc : uniqueLoc) {
stream.writeUTF(loc);
}
@@ -170,12 +166,14 @@ public class ExtendedBlockletWrapper implements Writable, Serializable {
* deseralize the blocklet data from file or stream
* data format
* <number of splits><number of unique location[short]><locations><serialize data len><data>
+ *
* @param tablePath
* @param queryId
* @return
* @throws IOException
*/
- public List<ExtendedBlocklet> readBlocklet(String tablePath, String queryId) throws IOException {
+ public List<ExtendedBlocklet> readBlocklet(String tablePath, String queryId, boolean isCountJob)
+ throws IOException {
byte[] data;
if (bytes != null) {
if (isWrittenToFile) {
@@ -218,7 +216,7 @@ public class ExtendedBlockletWrapper implements Writable, Serializable {
try {
for (int i = 0; i < numberOfBlocklet; i++) {
ExtendedBlocklet extendedBlocklet = new ExtendedBlocklet();
- extendedBlocklet.deserializeFields(eDIS, locations, tablePath);
+ extendedBlocklet.deserializeFields(eDIS, locations, tablePath, isCountJob);
extendedBlockletList.add(extendedBlocklet);
}
} finally {
@@ -248,4 +246,5 @@ public class ExtendedBlockletWrapper implements Writable, Serializable {
}
this.dataSize = in.readInt();
}
+
}
diff --git a/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapperContainer.java b/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapperContainer.java
index 0c52297..40acf9e 100644
--- a/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapperContainer.java
+++ b/core/src/main/java/org/apache/carbondata/core/indexstore/ExtendedBlockletWrapperContainer.java
@@ -62,8 +62,8 @@ public class ExtendedBlockletWrapperContainer implements Writable {
this.isFallbackJob = isFallbackJob;
}
- public List<ExtendedBlocklet> getExtendedBlockets(String tablePath, String queryId)
- throws IOException {
+ public List<ExtendedBlocklet> getExtendedBlockets(String tablePath, String queryId,
+ boolean isCountJob) throws IOException {
if (!isFallbackJob) {
int numOfThreads = CarbonProperties.getNumOfThreadsForPruning();
ExecutorService executorService = Executors
@@ -85,8 +85,8 @@ public class ExtendedBlockletWrapperContainer implements Writable {
List<Future<List<ExtendedBlocklet>>> futures = new ArrayList<>();
for (int i = 0; i < split.length; i++) {
end += split[i];
- futures.add(executorService
- .submit(new ExtendedBlockletDeserializerThread(start, end, tablePath, queryId)));
+ futures.add(executorService.submit(
+ new ExtendedBlockletDeserializerThread(start, end, tablePath, queryId, isCountJob)));
start += split[i];
}
executorService.shutdown();
@@ -109,7 +109,8 @@ public class ExtendedBlockletWrapperContainer implements Writable {
} else {
List<ExtendedBlocklet> extendedBlocklets = new ArrayList<>();
for (ExtendedBlockletWrapper extendedBlockletWrapper: extendedBlockletWrappers) {
- extendedBlocklets.addAll(extendedBlockletWrapper.readBlocklet(tablePath, queryId));
+ extendedBlocklets
+ .addAll(extendedBlockletWrapper.readBlocklet(tablePath, queryId, isCountJob));
}
return extendedBlocklets;
}
@@ -125,18 +126,22 @@ public class ExtendedBlockletWrapperContainer implements Writable {
private String queryId;
+ private boolean isCountJob;
+
public ExtendedBlockletDeserializerThread(int start, int end, String tablePath,
- String queryId) {
+ String queryId, boolean isCountJob) {
this.start = start;
this.end = end;
this.tablePath = tablePath;
this.queryId = queryId;
+ this.isCountJob = isCountJob;
}
@Override public List<ExtendedBlocklet> call() throws Exception {
List<ExtendedBlocklet> extendedBlocklets = new ArrayList<>();
for (int i = start; i < end; i++) {
- extendedBlocklets.addAll(extendedBlockletWrappers[i].readBlocklet(tablePath, queryId));
+ extendedBlocklets.addAll(extendedBlockletWrappers[i].readBlocklet(tablePath, queryId,
+ isCountJob));
}
return extendedBlocklets;
}
diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
index 45041e4..56ccabc 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
@@ -39,6 +39,7 @@ import org.apache.carbondata.core.datamap.DataMapJob;
import org.apache.carbondata.core.datamap.DataMapLevel;
import org.apache.carbondata.core.datamap.DataMapStoreManager;
import org.apache.carbondata.core.datamap.DataMapUtil;
+import org.apache.carbondata.core.datamap.DistributableDataMapFormat;
import org.apache.carbondata.core.datamap.Segment;
import org.apache.carbondata.core.datamap.TableDataMap;
import org.apache.carbondata.core.datamap.dev.expr.DataMapExprWrapper;
@@ -412,10 +413,42 @@ m filterExpression
*/
@Override public abstract List<InputSplit> getSplits(JobContext job) throws IOException;
- List<ExtendedBlocklet> getDistributedSplit(CarbonTable table,
+ /**
+ * This method will execute a distributed job(DistributedDataMapJob) to get the count for the
+ * table. If the DistributedDataMapJob fails for some reason then an embedded job is fired to
+ * get the count.
+ */
+ Long getDistributedCount(CarbonTable table,
+ List<PartitionSpec> partitionNames, List<Segment> validSegments) throws IOException {
+ DistributableDataMapFormat dataMapFormat =
+ new DistributableDataMapFormat(table, null, validSegments, new ArrayList<String>(),
+ partitionNames, false, null, false);
+ dataMapFormat.setIsWriteToFile(false);
+ try {
+ DataMapJob dataMapJob =
+ (DataMapJob) DataMapUtil.createDataMapJob(DataMapUtil.DISTRIBUTED_JOB_NAME);
+ if (dataMapJob == null) {
+ throw new ExceptionInInitializerError("Unable to create DistributedDataMapJob");
+ }
+ return dataMapJob.executeCountJob(dataMapFormat);
+ } catch (Exception e) {
+ LOG.error("Failed to get count from index server. Initializing fallback", e);
+ DataMapJob dataMapJob = DataMapUtil.getEmbeddedJob();
+ return dataMapJob.executeCountJob(dataMapFormat);
+ }
+ }
+
+ List<ExtendedBlocklet> getDistributedBlockRowCount(CarbonTable table,
+ List<PartitionSpec> partitionNames, List<Segment> validSegments,
+ List<Segment> invalidSegments, List<String> segmentsToBeRefreshed) throws IOException {
+ return getDistributedSplit(table, null, partitionNames, validSegments, invalidSegments,
+ segmentsToBeRefreshed, true);
+ }
+
+ private List<ExtendedBlocklet> getDistributedSplit(CarbonTable table,
FilterResolverIntf filterResolverIntf, List<PartitionSpec> partitionNames,
List<Segment> validSegments, List<Segment> invalidSegments,
- List<String> segmentsToBeRefreshed) throws IOException {
+ List<String> segmentsToBeRefreshed, boolean isCountJob) throws IOException {
try {
DataMapJob dataMapJob =
(DataMapJob) DataMapUtil.createDataMapJob(DataMapUtil.DISTRIBUTED_JOB_NAME);
@@ -424,7 +457,7 @@ m filterExpression
}
return DataMapUtil
.executeDataMapJob(table, filterResolverIntf, dataMapJob, partitionNames, validSegments,
- invalidSegments, null, segmentsToBeRefreshed);
+ invalidSegments, null, false, segmentsToBeRefreshed, isCountJob);
} catch (Exception e) {
// Check if fallback is disabled for testing purposes then directly throw exception.
if (CarbonProperties.getInstance().isFallBackDisabled()) {
@@ -432,10 +465,9 @@ m filterExpression
}
LOG.error("Exception occurred while getting splits using index server. Initiating Fall "
+ "back to embedded mode", e);
- return DataMapUtil
- .executeDataMapJob(table, filterResolverIntf,
- DataMapUtil.getEmbeddedJob(), partitionNames, validSegments, invalidSegments, null,
- true, segmentsToBeRefreshed);
+ return DataMapUtil.executeDataMapJob(table, filterResolverIntf,
+ DataMapUtil.getEmbeddedJob(), partitionNames, validSegments,
+ invalidSegments, null, true, segmentsToBeRefreshed, isCountJob);
}
}
@@ -545,7 +577,7 @@ m filterExpression
try {
prunedBlocklets =
getDistributedSplit(carbonTable, filter.getResolver(), partitionsToPrune, segmentIds,
- invalidSegments, segmentsToBeRefreshed);
+ invalidSegments, segmentsToBeRefreshed, false);
} catch (Exception e) {
// Check if fallback is disabled then directly throw exception otherwise try driver
// pruning.
@@ -580,7 +612,7 @@ m filterExpression
if (distributedCG && dataMapJob != null) {
cgPrunedBlocklets = DataMapUtil
.executeDataMapJob(carbonTable, filter.getResolver(), dataMapJob, partitionsToPrune,
- segmentIds, invalidSegments, DataMapLevel.CG, true, new ArrayList<String>());
+ segmentIds, invalidSegments, DataMapLevel.CG, new ArrayList<String>());
} else {
cgPrunedBlocklets = cgDataMapExprWrapper.prune(segmentIds, partitionsToPrune);
}
@@ -616,7 +648,7 @@ m filterExpression
// Prune segments from already pruned blocklets
fgPrunedBlocklets = DataMapUtil
.executeDataMapJob(carbonTable, filter.getResolver(), dataMapJob, partitionsToPrune,
- segmentIds, invalidSegments, fgDataMapExprWrapper.getDataMapLevel(), true,
+ segmentIds, invalidSegments, fgDataMapExprWrapper.getDataMapLevel(),
new ArrayList<String>());
// note that the 'fgPrunedBlocklets' has extra datamap related info compared with
// 'prunedBlocklets', so the intersection should keep the elements in 'fgPrunedBlocklets'
diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
index 3b7a800..74a4d6e 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonTableInputFormat.java
@@ -33,6 +33,7 @@ import org.apache.carbondata.core.datamap.DataMapStoreManager;
import org.apache.carbondata.core.datamap.Segment;
import org.apache.carbondata.core.datamap.TableDataMap;
import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.indexstore.ExtendedBlocklet;
import org.apache.carbondata.core.indexstore.PartitionSpec;
import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
import org.apache.carbondata.core.metadata.schema.PartitionInfo;
@@ -562,15 +563,14 @@ public class CarbonTableInputFormat<T> extends CarbonInputFormat<T> {
if (CarbonProperties.getInstance()
.isDistributedPruningEnabled(table.getDatabaseName(), table.getTableName())) {
try {
- List<InputSplit> extendedBlocklets = CarbonTableInputFormat.convertToCarbonInputSplit(
- getDistributedSplit(table, null, partitions, filteredSegment,
- allSegments.getInvalidSegments(), toBeCleanedSegments));
- for (InputSplit extendedBlocklet : extendedBlocklets) {
- CarbonInputSplit blocklet = (CarbonInputSplit) extendedBlocklet;
+ List<ExtendedBlocklet> extendedBlocklets =
+ getDistributedBlockRowCount(table, partitions, filteredSegment,
+ allSegments.getInvalidSegments(), toBeCleanedSegments);
+ for (ExtendedBlocklet blocklet : extendedBlocklets) {
String filePath = blocklet.getFilePath().replace("\\", "/");
String blockName = filePath.substring(filePath.lastIndexOf("/") + 1);
blockletToRowCountMap.put(blocklet.getSegmentId() + "," + blockName,
- (long) blocklet.getRowCount());
+ blocklet.getRowCount());
}
} catch (Exception e) {
// Check if fallback is disabled then directly throw exception otherwise try driver
@@ -615,15 +615,11 @@ public class CarbonTableInputFormat<T> extends CarbonInputFormat<T> {
}
}
} else {
- long totalRowCount = 0L;
+ long totalRowCount;
if (CarbonProperties.getInstance()
.isDistributedPruningEnabled(table.getDatabaseName(), table.getTableName())) {
- List<InputSplit> extendedBlocklets = CarbonTableInputFormat.convertToCarbonInputSplit(
- getDistributedSplit(table, null, partitions, filteredSegment,
- allSegments.getInvalidSegments(), new ArrayList<String>()));
- for (InputSplit extendedBlocklet : extendedBlocklets) {
- totalRowCount += ((CarbonInputSplit) extendedBlocklet).getRowCount();
- }
+ totalRowCount =
+ getDistributedCount(table, partitions, filteredSegment);
} else {
TableDataMap defaultDataMap = DataMapStoreManager.getInstance().getDefaultDataMap(table);
totalRowCount = defaultDataMap.getRowCount(filteredSegment, partitions, defaultDataMap);
diff --git a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DataMapJobs.scala b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DataMapJobs.scala
index 01b8824..1fee051 100644
--- a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DataMapJobs.scala
+++ b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DataMapJobs.scala
@@ -65,7 +65,8 @@ class DistributedDataMapJob extends AbstractDataMapJob {
dataMapFormat.getCarbonTable.getAbsoluteTableIdentifier, filterProcessor)
dataMapFormat.setFilterResolverIntf(filterInf)
IndexServer.getClient.getSplits(dataMapFormat)
- .getExtendedBlockets(dataMapFormat.getCarbonTable.getTablePath, dataMapFormat.getQueryId)
+ .getExtendedBlockets(dataMapFormat.getCarbonTable.getTablePath, dataMapFormat
+ .getQueryId, dataMapFormat.isCountStarJob)
} finally {
val tmpPath = CarbonUtil
.getIndexServerTempPath(dataMapFormat.getCarbonTable.getTablePath,
@@ -106,7 +107,11 @@ class DistributedDataMapJob extends AbstractDataMapJob {
filterInf.getFilterExpression.getFilterExpressionType == ExpressionType.UNKNOWN) {
return filterProcessor.changeUnknownResloverToTrue(tableIdentifer)
}
- return filterInf;
+ filterInf
+ }
+
+ override def executeCountJob(dataMapFormat: DistributableDataMapFormat): java.lang.Long = {
+ IndexServer.getClient.getCount(dataMapFormat).get()
}
}
@@ -122,7 +127,7 @@ class EmbeddedDataMapJob extends AbstractDataMapJob {
dataMapFormat.setIsWriteToFile(false)
dataMapFormat.setFallbackJob()
val splits = IndexServer.getSplits(dataMapFormat).getExtendedBlockets(dataMapFormat
- .getCarbonTable.getTablePath, dataMapFormat.getQueryId)
+ .getCarbonTable.getTablePath, dataMapFormat.getQueryId, dataMapFormat.isCountStarJob)
// Fire a job to clear the cache from executors as Embedded mode does not maintain the cache.
IndexServer.invalidateSegmentCache(dataMapFormat.getCarbonTable, dataMapFormat
.getValidSegmentIds.asScala.toArray)
@@ -130,4 +135,8 @@ class EmbeddedDataMapJob extends AbstractDataMapJob {
splits
}
+ override def executeCountJob(dataMapFormat: DistributableDataMapFormat): java.lang.Long = {
+ IndexServer.getCount(dataMapFormat).get()
+ }
+
}
diff --git a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedCountRDD.scala b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedCountRDD.scala
new file mode 100644
index 0000000..4a080fa
--- /dev/null
+++ b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedCountRDD.scala
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.indexserver
+
+import java.util.concurrent.Executors
+
+import scala.collection.JavaConverters._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutor, Future}
+import scala.concurrent.duration.Duration
+
+import org.apache.hadoop.mapred.TaskAttemptID
+import org.apache.hadoop.mapreduce.{InputSplit, TaskType}
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.spark.{Partition, SparkEnv, TaskContext}
+import org.apache.spark.sql.SparkSession
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.cache.CacheProvider
+import org.apache.carbondata.core.datamap.{DataMapStoreManager, DistributableDataMapFormat}
+import org.apache.carbondata.core.datamap.dev.expr.DataMapDistributableWrapper
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.util.{CarbonProperties, CarbonThreadFactory}
+import org.apache.carbondata.spark.rdd.CarbonRDD
+
+/**
+ * An RDD which will get the count for the table.
+ */
+class DistributedCountRDD(@transient ss: SparkSession, dataMapFormat: DistributableDataMapFormat)
+ extends CarbonRDD[(String, String)](ss, Nil) {
+
+ @transient private val LOGGER = LogServiceFactory.getLogService(classOf[DistributedPruneRDD]
+ .getName)
+
+ override protected def getPreferredLocations(split: Partition): Seq[String] = {
+ if (split.asInstanceOf[DataMapRDDPartition].getLocations != null) {
+ split.asInstanceOf[DataMapRDDPartition].getLocations.toSeq
+ } else {
+ Seq()
+ }
+ }
+
+ override def internalCompute(split: Partition,
+ context: TaskContext): Iterator[(String, String)] = {
+ val attemptId = new TaskAttemptID(DistributedRDDUtils.generateTrackerId,
+ id, TaskType.MAP, split.index, 0)
+ val attemptContext = new TaskAttemptContextImpl(FileFactory.getConfiguration, attemptId)
+ val inputSplits = split.asInstanceOf[DataMapRDDPartition].inputSplit
+ val numOfThreads = CarbonProperties.getInstance().getNumOfThreadsForExecutorPruning
+ val service = Executors
+ .newFixedThreadPool(numOfThreads, new CarbonThreadFactory("IndexPruningPool", true))
+ implicit val ec: ExecutionContextExecutor = ExecutionContext
+ .fromExecutor(service)
+ val futures = if (inputSplits.length <= numOfThreads) {
+ inputSplits.map {
+ split => generateFuture(Seq(split))
+ }
+ } else {
+ DistributedRDDUtils.groupSplits(inputSplits, numOfThreads).map {
+ splits => generateFuture(splits)
+ }
+ }
+ // scalastyle:off awaitresult
+ val results = Await.result(Future.sequence(futures), Duration.Inf).flatten
+ // scalastyle:on awaitresult
+ val executorIP = s"${ SparkEnv.get.blockManager.blockManagerId.host }_${
+ SparkEnv.get.blockManager.blockManagerId.executorId
+ }"
+ val cacheSize = if (CacheProvider.getInstance().getCarbonCache != null) {
+ CacheProvider.getInstance().getCarbonCache.getCurrentSize
+ } else {
+ 0L
+ }
+ Iterator((executorIP + "_" + cacheSize.toString, results.map(_._2.toLong).sum.toString))
+ }
+
+ override protected def internalGetPartitions: Array[Partition] = {
+ new DistributedPruneRDD(ss, dataMapFormat).partitions
+ }
+
+ private def generateFuture(split: Seq[InputSplit])
+ (implicit executionContext: ExecutionContext) = {
+ Future {
+ val segments = split.map { inputSplit =>
+ val distributable = inputSplit.asInstanceOf[DataMapDistributableWrapper]
+ distributable.getDistributable.getSegment
+ .setReadCommittedScope(dataMapFormat.getReadCommittedScope)
+ distributable.getDistributable.getSegment
+ }
+ val defaultDataMap = DataMapStoreManager.getInstance
+ .getDataMap(dataMapFormat.getCarbonTable, split.head
+ .asInstanceOf[DataMapDistributableWrapper].getDistributable.getDataMapSchema)
+ defaultDataMap.getBlockRowCount(segments.toList.asJava, dataMapFormat
+ .getPartitions, defaultDataMap).asScala
+ }
+ }
+
+}
diff --git a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala
index d8b9c19..76d33b4 100644
--- a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala
+++ b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedPruneRDD.scala
@@ -17,15 +17,12 @@
package org.apache.carbondata.indexserver
-import java.text.SimpleDateFormat
-import java.util.Date
import java.util.concurrent.Executors
import scala.collection.JavaConverters._
import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutor, Future}
import scala.concurrent.duration.Duration
-import org.apache.commons.lang.StringUtils
import org.apache.hadoop.mapred.{RecordReader, TaskAttemptID}
import org.apache.hadoop.mapreduce.{InputSplit, Job, TaskType}
import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
@@ -35,9 +32,8 @@ import org.apache.spark.sql.hive.DistributionUtil
import org.apache.carbondata.common.logging.LogServiceFactory
import org.apache.carbondata.core.cache.CacheProvider
-import org.apache.carbondata.core.datamap.{DataMapDistributable, DataMapStoreManager, DistributableDataMapFormat, TableDataMap}
+import org.apache.carbondata.core.datamap.{DataMapStoreManager, DistributableDataMapFormat}
import org.apache.carbondata.core.datamap.dev.expr.DataMapDistributableWrapper
-import org.apache.carbondata.core.datastore.block.SegmentPropertiesAndSchemaHolder
import org.apache.carbondata.core.datastore.impl.FileFactory
import org.apache.carbondata.core.indexstore.{ExtendedBlocklet, ExtendedBlockletWrapper}
import org.apache.carbondata.core.util.{CarbonProperties, CarbonThreadFactory}
@@ -65,21 +61,13 @@ private[indexserver] class DistributedPruneRDD(@transient private val ss: SparkS
@transient private val LOGGER = LogServiceFactory.getLogService(classOf[DistributedPruneRDD]
.getName)
- private val jobTrackerId: String = {
- val formatter = new SimpleDateFormat("yyyyMMddHHmm")
- formatter.format(new Date())
- }
- var readers: scala.collection.Iterator[RecordReader[Void, ExtendedBlocklet]] = _
- private def groupSplits(xs: Seq[InputSplit], n: Int) = {
- val (quot, rem) = (xs.size / n, xs.size % n)
- val (smaller, bigger) = xs.splitAt(xs.size - rem * (quot + 1))
- (smaller.grouped(quot) ++ bigger.grouped(quot + 1)).toList
- }
+ var readers: scala.collection.Iterator[RecordReader[Void, ExtendedBlocklet]] = _
override def internalCompute(split: Partition,
context: TaskContext): Iterator[(String, ExtendedBlockletWrapper)] = {
- val attemptId = new TaskAttemptID(jobTrackerId, id, TaskType.MAP, split.index, 0)
+ val attemptId = new TaskAttemptID(DistributedRDDUtils.generateTrackerId,
+ id, TaskType.MAP, split.index, 0)
val attemptContext = new TaskAttemptContextImpl(FileFactory.getConfiguration, attemptId)
val inputSplits = split.asInstanceOf[DataMapRDDPartition].inputSplit
if (dataMapFormat.isJobToClearDataMaps) {
@@ -118,7 +106,7 @@ private[indexserver] class DistributedPruneRDD(@transient private val ss: SparkS
split => generateFuture(Seq(split), attemptContext)
}
} else {
- groupSplits(inputSplits, numOfThreads).map {
+ DistributedRDDUtils.groupSplits(inputSplits, numOfThreads).map {
splits => generateFuture(splits, attemptContext)
}
}
@@ -139,14 +127,13 @@ private[indexserver] class DistributedPruneRDD(@transient private val ss: SparkS
}"
val value = (executorIP + "_" + cacheSize.toString, new ExtendedBlockletWrapper(f.toList
.asJava, dataMapFormat.getCarbonTable.getTablePath, dataMapFormat.getQueryId,
- dataMapFormat.isWriteToFile))
+ dataMapFormat.isWriteToFile, dataMapFormat.isCountStarJob))
Iterator(value)
}
}
- private def generateFuture(split: Seq[InputSplit],
- attemptContextImpl: TaskAttemptContextImpl)
- (implicit executionContext: ExecutionContext) = {
+ private def generateFuture(split: Seq[InputSplit], attemptContextImpl: TaskAttemptContextImpl)
+ (implicit executionContext: ExecutionContext): Future[Seq[ExtendedBlocklet]] = {
Future {
split.flatMap { inputSplit =>
val blocklets = new java.util.ArrayList[ExtendedBlocklet]()
diff --git a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedRDDUtils.scala b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedRDDUtils.scala
index 933ec15..4819779 100644
--- a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedRDDUtils.scala
+++ b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedRDDUtils.scala
@@ -16,6 +16,8 @@
*/
package org.apache.carbondata.indexserver
+import java.text.SimpleDateFormat
+import java.util.Date
import java.util.concurrent.ConcurrentHashMap
import scala.collection.JavaConverters._
@@ -325,4 +327,15 @@ object DistributedRDDUtils {
}
}
+ def groupSplits(xs: Seq[InputSplit], n: Int): List[Seq[InputSplit]] = {
+ val (quot, rem) = (xs.size / n, xs.size % n)
+ val (smaller, bigger) = xs.splitAt(xs.size - rem * (quot + 1))
+ (smaller.grouped(quot) ++ bigger.grouped(quot + 1)).toList
+ }
+
+ def generateTrackerId: String = {
+ val formatter = new SimpleDateFormat("yyyyMMddHHmm")
+ formatter.format(new Date())
+ }
+
}
diff --git a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
index fdaa3d1..abee487 100644
--- a/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
+++ b/integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
@@ -23,6 +23,7 @@ import java.util.UUID
import scala.collection.JavaConverters._
import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.io.LongWritable
import org.apache.hadoop.ipc.{ProtocolInfo, RPC}
import org.apache.hadoop.net.NetUtils
import org.apache.hadoop.security.{KerberosInfo, SecurityUtil, UserGroupInformation}
@@ -58,6 +59,9 @@ trait ServerInterface {
*/
def invalidateSegmentCache(carbonTable: CarbonTable,
segmentIds: Array[String], jobGroupId: String = ""): Unit
+
+ def getCount(request: DistributableDataMapFormat): LongWritable
+
}
/**
@@ -99,6 +103,21 @@ object IndexServer extends ServerInterface {
})
}
+ def getCount(request: DistributableDataMapFormat): LongWritable = {
+ doAs {
+ if (!request.isFallbackJob) {
+ sparkSession.sparkContext.setLocalProperty("spark.jobGroup.id", request.getTaskGroupId)
+ sparkSession.sparkContext
+ .setLocalProperty("spark.job.description", request.getTaskGroupDesc)
+ }
+ val splits = new DistributedCountRDD(sparkSession, request).collect()
+ if (!request.isFallbackJob) {
+ DistributedRDDUtils.updateExecutorCacheSize(splits.map(_._1).toSet)
+ }
+ new LongWritable(splits.map(_._2.toLong).sum)
+ }
+ }
+
def getSplits(request: DistributableDataMapFormat): ExtendedBlockletWrapperContainer = {
doAs {
if (!request.isFallbackJob) {
[carbondata] 29/33: [HOTFIX]Update Documentation for MV datamap
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 85dc0304743e74c7abc00d59d4d4b5e5f619d03e
Author: Indhumathi27 <in...@gmail.com>
AuthorDate: Thu Jul 25 17:02:26 2019 +0530
[HOTFIX]Update Documentation for MV datamap
This closes #3335
---
README.md | 1 +
docs/datamap/mv-datamap-guide.md | 2 ++
2 files changed, 3 insertions(+)
diff --git a/README.md b/README.md
index 3226a30..da5b547 100644
--- a/README.md
+++ b/README.md
@@ -60,6 +60,7 @@ CarbonData is built using Apache Maven, to [build CarbonData](https://github.com
* [CarbonData Lucene DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/lucene-datamap-guide.md)
* [CarbonData Pre-aggregate DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/preaggregate-datamap-guide.md)
* [CarbonData Timeseries DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/timeseries-datamap-guide.md)
+ * [CarbonData MV DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/mv-datamap-guide.md)
* [SDK Guide](https://github.com/apache/carbondata/blob/master/docs/sdk-guide.md)
* [C++ SDK Guide](https://github.com/apache/carbondata/blob/master/docs/csdk-guide.md)
* [Performance Tuning](https://github.com/apache/carbondata/blob/master/docs/performance-tuning.md)
diff --git a/docs/datamap/mv-datamap-guide.md b/docs/datamap/mv-datamap-guide.md
index d22357c..fc1ffd5 100644
--- a/docs/datamap/mv-datamap-guide.md
+++ b/docs/datamap/mv-datamap-guide.md
@@ -65,6 +65,7 @@ EXPLAIN SELECT a, sum(b) from maintable group by a;
CREATE DATAMAP agg_sales
ON TABLE sales
USING "MV"
+ DMPROPERTIES('TABLE_BLOCKSIZE'='256 MB','LOCAL_DICTIONARY_ENABLE'='false')
AS
SELECT country, sex, sum(quantity), avg(price)
FROM sales
@@ -97,6 +98,7 @@ EXPLAIN SELECT a, sum(b) from maintable group by a;
property is inherited from parent table, which allows user to provide different tableproperties
for child table
* MV creation with limit or union all ctas queries is unsupported
+ * MV datamap does not support Streaming
#### How MV tables are selected
[carbondata] 18/33: [CARBONDATA-3506]Fix alter table failures on
parition table with hive.metastore.disallow.incompatible.col.type.changes
as true
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit ef26a4a0a556d574cc30c74c674234a4564f34c1
Author: akashrn5 <ak...@gmail.com>
AuthorDate: Wed Aug 28 12:05:13 2019 +0530
[CARBONDATA-3506]Fix alter table failures on parition table with hive.metastore.disallow.incompatible.col.type.changes as true
Problem:
In case of spark2.2 and above and , when we call alterExternalCatalogForTableWithUpdatedSchema to update the new schema to external catalog
in case of add column, spark gets the catalog table and then it itself adds the partition columns if the table is partition table for all the
new data schema sent by carbon, so there will be duplicate partition columns, so validation fails in hive
When the table has only two columns and one of them is partition column, then dropping non partition column is invalid because,
if we allow it is like table with all columns as partition columns. So with the above property as true, drop column will fail to update the hive metastore.
in spark2.2 and above if the datatype change is done on partition column, with the above property as true, it also fails,
as we are not sending partition column for schema alter in hive
Solution:
when sending the new schema to spark to update in catalog, do not send the partition columns in case of spark2.2 and above,
as spark will take care of adding parition columns to new schema sent by us.
In the above scenario of drop, do not allow drop column, if after dropping the specific column, if table has only partition columns.
Block the operation on datatype change on partition column on spark2.2 and above.
This closes #3367
---
.../StandardPartitionTableQueryTestCase.scala | 29 +++++++++++++----
.../schema/CarbonAlterTableAddColumnCommand.scala | 20 +++++++++---
...nAlterTableColRenameDataTypeChangeCommand.scala | 36 +++++++++++++++++++---
.../schema/CarbonAlterTableDropColumnCommand.scala | 35 +++++++++++++++++----
4 files changed, 99 insertions(+), 21 deletions(-)
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableQueryTestCase.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableQueryTestCase.scala
index c19c0b9..fb4b511 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableQueryTestCase.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/standardpartition/StandardPartitionTableQueryTestCase.scala
@@ -21,8 +21,10 @@ import org.apache.spark.sql.execution.strategy.CarbonDataSourceScan
import org.apache.spark.sql.test.Spark2TestQueryExecutor
import org.apache.spark.sql.test.util.QueryTest
import org.apache.spark.sql.{DataFrame, Row}
+import org.apache.spark.util.SparkUtil
import org.scalatest.BeforeAndAfterAll
+import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.datastore.impl.FileFactory
import org.apache.carbondata.core.util.CarbonProperties
@@ -439,18 +441,32 @@ test("Creation of partition table should fail if the colname in table schema and
test("validate data in partition table after dropping and adding a column") {
sql("drop table if exists par")
- sql("create table par(name string) partitioned by (age double) stored by " +
+ sql("create table par(name string, add string) partitioned by (age double) stored by " +
"'carbondata' TBLPROPERTIES('cache_level'='blocklet')")
- sql(s"load data local inpath '$resourcesPath/uniqwithoutheader.csv' into table par options" +
- s"('header'='false')")
+ sql("insert into par select 'joey','NY',32 union all select 'chandler','NY',32")
sql("alter table par drop columns(name)")
sql("alter table par add columns(name string)")
- sql(s"load data local inpath '$resourcesPath/uniqwithoutheader.csv' into table par options" +
- s"('header'='false')")
- checkAnswer(sql("select name from par"), Seq(Row("a"),Row("b"), Row(null), Row(null)))
+ sql("insert into par select 'joey','NY',32 union all select 'joey','NY',32")
+ checkAnswer(sql("select name from par"), Seq(Row("NY"),Row("NY"), Row(null), Row(null)))
sql("drop table if exists par")
}
+ test("test drop column when after dropping only partition column remains and datatype change on partition column") {
+ sql("drop table if exists onlyPart")
+ sql("create table onlyPart(name string) partitioned by (age int) stored by " +
+ "'carbondata' TBLPROPERTIES('cache_level'='blocklet')")
+ val ex1 = intercept[MalformedCarbonCommandException] {
+ sql("alter table onlyPart drop columns(name)")
+ }
+ assert(ex1.getMessage.contains("alter table drop column is failed, cannot have the table with all columns as partition column"))
+ if (SparkUtil.isSparkVersionXandAbove("2.2")) {
+ val ex2 = intercept[MalformedCarbonCommandException] {
+ sql("alter table onlyPart change age age bigint")
+ }
+ assert(ex2.getMessage.contains("Alter datatype of the partition column age is not allowed"))
+ }
+ sql("drop table if exists onlyPart")
+ }
private def verifyPartitionInfo(frame: DataFrame, partitionNames: Seq[String]) = {
val plan = frame.queryExecution.sparkPlan
@@ -488,6 +504,7 @@ test("Creation of partition table should fail if the colname in table schema and
sql("drop table if exists staticpartitionextlocload_new")
sql("drop table if exists staticpartitionlocloadother_new")
sql("drop table if exists par")
+ sql("drop table if exists onlyPart")
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableAddColumnCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableAddColumnCommand.scala
index 3fab2fb..6b0d709 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableAddColumnCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableAddColumnCommand.scala
@@ -22,7 +22,7 @@ import scala.collection.JavaConverters._
import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
import org.apache.spark.sql.execution.command.{AlterTableAddColumnsModel, AlterTableColumnSchemaGenerator, MetadataCommand}
import org.apache.spark.sql.hive.CarbonSessionCatalog
-import org.apache.spark.util.AlterTableUtil
+import org.apache.spark.util.{AlterTableUtil, SparkUtil}
import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.common.logging.LogServiceFactory
@@ -102,12 +102,24 @@ private[sql] case class CarbonAlterTableAddColumnCommand(
carbonTable,
schemaConverter.fromWrapperToExternalSchemaEvolutionEntry(schemaEvolutionEntry),
thriftTable)(sparkSession)
+ // In case of spark2.2 and above and , when we call
+ // alterExternalCatalogForTableWithUpdatedSchema to update the new schema to external catalog
+ // in case of add column, spark gets the catalog table and then it itself adds the partition
+ // columns if the table is partition table for all the new data schema sent by carbon,
+ // so there will be duplicate partition columns, so send the columns without partition columns
+ val cols = if (SparkUtil.isSparkVersionXandAbove("2.2") && carbonTable.isHivePartitionTable) {
+ val partitionColumns = carbonTable.getPartitionInfo.getColumnSchemaList.asScala
+ val carbonColumnsWithoutPartition = carbonColumns.filterNot(col => partitionColumns.contains
+ (col))
+ Some(carbonColumnsWithoutPartition ++ sortedColsBasedActualSchemaOrder)
+ } else {
+ Some(carbonColumns ++ sortedColsBasedActualSchemaOrder)
+ }
sparkSession.sessionState.catalog.asInstanceOf[CarbonSessionCatalog].alterAddColumns(
- tableIdentifier, schemaParts, Some(carbonColumns ++ sortedColsBasedActualSchemaOrder))
+ tableIdentifier, schemaParts, cols)
sparkSession.catalog.refreshTable(tableIdentifier.quotedString)
val alterTablePostExecutionEvent: AlterTableAddColumnPostEvent =
- new AlterTableAddColumnPostEvent(sparkSession,
- carbonTable, alterTableAddColumnsModel)
+ AlterTableAddColumnPostEvent(sparkSession, carbonTable, alterTableAddColumnsModel)
OperationListenerBus.getInstance.fireEvent(alterTablePostExecutionEvent, operationContext)
LOGGER.info(s"Alter table for add columns is successful for table $dbName.$tableName")
} catch {
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
index 49f651e..d3d63eb 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
@@ -21,10 +21,9 @@ import scala.collection.JavaConverters._
import scala.collection.mutable
import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
-import org.apache.spark.sql.execution.command.{AlterTableDataTypeChangeModel, DataTypeInfo,
- MetadataCommand}
+import org.apache.spark.sql.execution.command.{AlterTableDataTypeChangeModel, DataTypeInfo, MetadataCommand}
import org.apache.spark.sql.hive.CarbonSessionCatalog
-import org.apache.spark.util.AlterTableUtil
+import org.apache.spark.util.{AlterTableUtil, SparkUtil}
import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.common.logging.LogServiceFactory
@@ -162,6 +161,18 @@ private[sql] case class CarbonAlterTableColRenameDataTypeChangeCommand(
isDataTypeChange = true
}
if (isDataTypeChange) {
+ // if column datatype change operation is on partition column, then fail the datatype change
+ // operation
+ if (SparkUtil.isSparkVersionXandAbove("2.2") && null != carbonTable.getPartitionInfo) {
+ val partitionColumns = carbonTable.getPartitionInfo.getColumnSchemaList
+ partitionColumns.asScala.foreach {
+ col =>
+ if (col.getColumnName.equalsIgnoreCase(oldColumnName)) {
+ throw new MalformedCarbonCommandException(
+ s"Alter datatype of the partition column $newColumnName is not allowed")
+ }
+ }
+ }
validateColumnDataType(alterTableColRenameAndDataTypeChangeModel.dataTypeInfo,
oldCarbonColumn.head)
}
@@ -257,7 +268,7 @@ private[sql] case class CarbonAlterTableColRenameDataTypeChangeCommand(
* @param carbonTable carbonTable
* @param tableInfo tableInfo
* @param addColumnSchema added column schema
- * @param schemaEvolutionEntryList new SchemaEvolutionEntry
+ * @param schemaEvolutionEntry new SchemaEvolutionEntry
*/
private def updateSchemaAndRefreshTable(sparkSession: SparkSession,
carbonTable: CarbonTable,
@@ -275,12 +286,27 @@ private[sql] case class CarbonAlterTableColRenameDataTypeChangeCommand(
// update the schema changed column at the specific index in carbonColumns based on schema order
carbonColumns
.update(schemaOrdinal, schemaConverter.fromExternalToWrapperColumnSchema(addColumnSchema))
+ // In case of spark2.2 and above and , when we call
+ // alterExternalCatalogForTableWithUpdatedSchema to update the new schema to external catalog
+ // in case of rename column or change datatype, spark gets the catalog table and then it itself
+ // adds the partition columns if the table is partition table for all the new data schema sent
+ // by carbon, so there will be duplicate partition columns, so send the columns without
+ // partition columns
+ val columns = if (SparkUtil.isSparkVersionXandAbove("2.2") &&
+ carbonTable.isHivePartitionTable) {
+ val partitionColumns = carbonTable.getPartitionInfo.getColumnSchemaList.asScala
+ val carbonColumnsWithoutPartition = carbonColumns.filterNot(col => partitionColumns.contains(
+ col))
+ Some(carbonColumnsWithoutPartition)
+ } else {
+ Some(carbonColumns)
+ }
val (tableIdentifier, schemaParts) = AlterTableUtil.updateSchemaInfo(
carbonTable,
schemaEvolutionEntry,
tableInfo)(sparkSession)
sparkSession.sessionState.catalog.asInstanceOf[CarbonSessionCatalog]
- .alterColumnChangeDataTypeOrRename(tableIdentifier, schemaParts, Some(carbonColumns))
+ .alterColumnChangeDataTypeOrRename(tableIdentifier, schemaParts, columns)
sparkSession.catalog.refreshTable(tableIdentifier.quotedString)
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDropColumnCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDropColumnCommand.scala
index 31cfdaf..8a2e837 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDropColumnCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDropColumnCommand.scala
@@ -23,7 +23,7 @@ import scala.collection.mutable.ListBuffer
import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
import org.apache.spark.sql.execution.command.{AlterTableDropColumnModel, MetadataCommand}
import org.apache.spark.sql.hive.CarbonSessionCatalog
-import org.apache.spark.util.AlterTableUtil
+import org.apache.spark.util.{AlterTableUtil, SparkUtil}
import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.common.logging.LogServiceFactory
@@ -63,6 +63,7 @@ private[sql] case class CarbonAlterTableDropColumnCommand(
"alter table drop column is not supported for index datamap")
}
val partitionInfo = carbonTable.getPartitionInfo(tableName)
+ val tableColumns = carbonTable.getCreateOrderColumn(tableName).asScala
if (partitionInfo != null) {
val partitionColumnSchemaList = partitionInfo.getColumnSchemaList.asScala
.map(_.getColumnName)
@@ -74,9 +75,18 @@ private[sql] case class CarbonAlterTableDropColumnCommand(
throwMetadataException(dbName, tableName, "Partition columns cannot be dropped: " +
s"$partitionColumns")
}
+
+ // this check is added because, when table have only two columns, one is partition and one
+ // is non partition, then dropping one column means, having table with only partition
+ // column, which is wrong
+ if (tableColumns.filterNot(col => alterTableDropColumnModel.columns
+ .contains(col.getColName)).map(_.getColName).equals(partitionColumnSchemaList)) {
+ throw new MalformedCarbonCommandException(
+ "alter table drop column is failed, cannot have the table with all columns as " +
+ "partition columns")
+ }
}
- val tableColumns = carbonTable.getCreateOrderColumn(tableName).asScala
var dictionaryColumns = Seq[org.apache.carbondata.core.metadata.schema.table.column
.ColumnSchema]()
// TODO: if deleted column list includes bucketted column throw an error
@@ -139,11 +149,24 @@ private[sql] case class CarbonAlterTableDropColumnCommand(
schemaEvolutionEntry,
tableInfo)(sparkSession)
// get the columns in schema order and filter the dropped column in the column set
- val cols = Some(carbonTable.getCreateOrderColumn(carbonTable.getTableName).asScala
- .collect { case carbonColumn if !carbonColumn.isInvisible => carbonColumn.getColumnSchema}
- .filterNot(column => delCols.contains(column)))
+ val cols = carbonTable.getCreateOrderColumn(carbonTable.getTableName).asScala
+ .collect { case carbonColumn if !carbonColumn.isInvisible => carbonColumn.getColumnSchema }
+ .filterNot(column => delCols.contains(column))
+ // In case of spark2.2 and above and , when we call
+ // alterExternalCatalogForTableWithUpdatedSchema to update the new schema to external catalog
+ // in case of drop column, spark gets the catalog table and then it itself adds the partition
+ // columns if the table is partition table for all the new data schema sent by carbon,
+ // so there will be duplicate partition columns, so send the columns without partition columns
+ val columns = if (SparkUtil.isSparkVersionXandAbove("2.2") &&
+ carbonTable.isHivePartitionTable) {
+ val partitionColumns = partitionInfo.getColumnSchemaList.asScala
+ val carbonColumnsWithoutPartition = cols.filterNot(col => partitionColumns.contains(col))
+ Some(carbonColumnsWithoutPartition)
+ } else {
+ Some(cols)
+ }
sparkSession.sessionState.catalog.asInstanceOf[CarbonSessionCatalog]
- .alterDropColumns(tableIdentifier, schemaParts, cols)
+ .alterDropColumns(tableIdentifier, schemaParts, columns)
sparkSession.catalog.refreshTable(tableIdentifier.quotedString)
// TODO: 1. add check for deletion of index tables
// delete dictionary files for dictionary column and clear dictionary cache from memory
[carbondata] 02/33: [CARBONDATA-3494]Fix NullPointerException in
drop table and Correct the document formatting
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 7571b99ecb257956a05528a0d2910f66f4fd7daf
Author: akashrn5 <ak...@gmail.com>
AuthorDate: Thu Aug 15 17:56:57 2019 +0530
[CARBONDATA-3494]Fix NullPointerException in drop table and Correct the document formatting
This closes #Problem:
Fix the formatting of the document in index server md file.
drop table is calling drop datamap command with force drop as true. Due to this table is removed from meta and physically. Then when processData is called for drop table, it tried to create carbonTable object by reading schema which causes NullPointerException.
Solution:
correct the formatting
Skip ProcessData if carbonTable is null
This closes #3359
---
.../org/apache/carbondata/core/datamap/DataMapStoreManager.java | 7 ++++++-
docs/index-server.md | 6 ++----
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
index ce0d6a6..f1f48fa 100644
--- a/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
+++ b/core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
@@ -598,6 +598,11 @@ public final class DataMapStoreManager {
*/
public void deleteDataMap(AbsoluteTableIdentifier identifier, String dataMapName) {
CarbonTable carbonTable = getCarbonTable(identifier);
+ if (carbonTable == null) {
+ // If carbon table is null then it means table is already deleted, therefore return without
+ // doing any further changes.
+ return;
+ }
String tableUniqueName = identifier.getCarbonTableIdentifier().getTableUniqueName();
if (CarbonProperties.getInstance()
.isDistributedPruningEnabled(identifier.getDatabaseName(), identifier.getTableName())) {
@@ -613,7 +618,7 @@ public final class DataMapStoreManager {
if (tableIndices != null) {
int i = 0;
for (TableDataMap tableDataMap : tableIndices) {
- if (carbonTable != null && tableDataMap != null && dataMapName
+ if (tableDataMap != null && dataMapName
.equalsIgnoreCase(tableDataMap.getDataMapSchema().getDataMapName())) {
try {
DataMapUtil
diff --git a/docs/index-server.md b/docs/index-server.md
index 5dd15c5..9253f2a 100644
--- a/docs/index-server.md
+++ b/docs/index-server.md
@@ -136,11 +136,9 @@ The Index Server is a long running service therefore the 'spark.yarn.keytab' and
| Name | Default Value | Description |
|:----------:|:-------------:|:------: |
| carbon.enable.index.server | false | Enable the use of index server for pruning for the whole application. |
-| carbon.index.server.ip | NA | Specify the IP/HOST on which the server is started. Better to
- specify the private IP. |
+| carbon.index.server.ip | NA | Specify the IP/HOST on which the server is started. Better to specify the private IP. |
| carbon.index.server.port | NA | The port on which the index server is started. |
-| carbon.disable.index.server.fallback | false | Whether to enable/disable fallback for index server
-. Should be used for testing purposes only. Refer: [Fallback](#Fallback)|
+| carbon.disable.index.server.fallback | false | Whether to enable/disable fallback for index server. Should be used for testing purposes only. Refer: [Fallback](#Fallback)|
|carbon.index.server.max.jobname.length|NA|The max length of the job to show in the index server service UI. For bigger queries this may impact performance as the whole string would be sent from JDBCServer to IndexServer.|
[carbondata] 05/33: [CARBONDATA-3491] Return updated/deleted rows
count when execute update/delete sql
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 6c9bbfe8ec5606262bbbbb93b101986352578b2b
Author: Zhang Zhichao <44...@qq.com>
AuthorDate: Tue Aug 13 11:00:23 2019 +0800
[CARBONDATA-3491] Return updated/deleted rows count when execute update/delete sql
Return updated/deleted rows count when execute update/delete sql
This closes #3357
---
.../testsuite/iud/DeleteCarbonTableTestCase.scala | 19 +++++++++++++
.../testsuite/iud/UpdateCarbonTableTestCase.scala | 33 ++++++++++++++++++++++
.../scala/org/apache/carbondata/spark/KeyVal.scala | 10 +++----
.../apache/spark/util/CarbonReflectionUtils.scala | 16 +++++++++++
.../apache/spark/sql/CarbonCatalystOperators.scala | 6 ++--
.../mutation/CarbonProjectForDeleteCommand.scala | 21 ++++++++++----
.../mutation/CarbonProjectForUpdateCommand.scala | 19 ++++++++-----
.../command/mutation/DeleteExecution.scala | 27 ++++++++++--------
.../spark/sql/hive/CarbonAnalysisRules.scala | 12 +++++++-
9 files changed, 129 insertions(+), 34 deletions(-)
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
index f26283b..4565d7a 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/DeleteCarbonTableTestCase.scala
@@ -361,6 +361,25 @@ class DeleteCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
sql("drop table if exists decimal_table")
}
+ test("[CARBONDATA-3491] Return updated/deleted rows count when execute update/delete sql") {
+ sql("drop table if exists test_return_row_count")
+
+ sql("create table test_return_row_count (a string, b string, c string) stored by 'carbondata'").show()
+ sql("insert into test_return_row_count select 'aaa','bbb','ccc'").show()
+ sql("insert into test_return_row_count select 'bbb','bbb','ccc'").show()
+ sql("insert into test_return_row_count select 'ccc','bbb','ccc'").show()
+ sql("insert into test_return_row_count select 'ccc','bbb','ccc'").show()
+
+ checkAnswer(sql("delete from test_return_row_count where a = 'aaa'"),
+ Seq(Row(1))
+ )
+ checkAnswer(sql("select * from test_return_row_count"),
+ Seq(Row("bbb", "bbb", "ccc"), Row("ccc", "bbb", "ccc"), Row("ccc", "bbb", "ccc"))
+ )
+
+ sql("drop table if exists test_return_row_count").show()
+ }
+
override def afterAll {
sql("use default")
sql("drop database if exists iud_db cascade")
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
index cf45600..ef18035 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
@@ -826,6 +826,39 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
sql("""drop table iud.dest11""").show
}
+ test("[CARBONDATA-3491] Return updated/deleted rows count when execute update/delete sql") {
+ sql("drop table if exists test_return_row_count")
+ sql("drop table if exists test_return_row_count_source")
+
+ sql("create table test_return_row_count (a string, b string, c string) stored by 'carbondata'").show()
+ sql("insert into test_return_row_count select 'bbb','bbb','ccc'").show()
+ sql("insert into test_return_row_count select 'ccc','bbb','ccc'").show()
+ sql("insert into test_return_row_count select 'ccc','bbb','ccc'").show()
+
+ sql("create table test_return_row_count_source (a string, b string, c string) stored by 'carbondata'").show()
+ sql("insert into test_return_row_count_source select 'aaa','eee','ccc'").show()
+ sql("insert into test_return_row_count_source select 'bbb','bbb','ccc'").show()
+ sql("insert into test_return_row_count_source select 'ccc','bbb','ccc'").show()
+ sql("insert into test_return_row_count_source select 'ccc','bbb','ccc'").show()
+
+ checkAnswer(sql("update test_return_row_count set (b) = ('ddd') where a = 'ccc'"),
+ Seq(Row(2))
+ )
+ checkAnswer(sql("select * from test_return_row_count"),
+ Seq(Row("bbb", "bbb", "ccc"), Row("ccc", "ddd", "ccc"), Row("ccc", "ddd", "ccc"))
+ )
+
+ checkAnswer(sql("update test_return_row_count t set (t.b) = (select s.b from test_return_row_count_source s where s.a = 'aaa') where t.a = 'ccc'"),
+ Seq(Row(2))
+ )
+ checkAnswer(sql("select * from test_return_row_count"),
+ Seq(Row("bbb", "bbb", "ccc"), Row("ccc", "eee", "ccc"), Row("ccc", "eee", "ccc"))
+ )
+
+ sql("drop table if exists test_return_row_count")
+ sql("drop table if exists test_return_row_count_source")
+ }
+
override def afterAll {
sql("use default")
sql("drop database if exists iud cascade")
diff --git a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/KeyVal.scala b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/KeyVal.scala
index c4b1144..9fca245 100644
--- a/integration/spark-common/src/main/scala/org/apache/carbondata/spark/KeyVal.scala
+++ b/integration/spark-common/src/main/scala/org/apache/carbondata/spark/KeyVal.scala
@@ -76,22 +76,20 @@ class updateResultImpl
}
trait DeleteDelataResult[K, V] extends Serializable {
- def getKey(key: SegmentStatus, value: (SegmentUpdateDetails, ExecutionErrors)): (K, V)
+ def getKey(key: SegmentStatus, value: (SegmentUpdateDetails, ExecutionErrors, Long)): (K, V)
}
class DeleteDelataResultImpl
- extends DeleteDelataResult[SegmentStatus, (SegmentUpdateDetails, ExecutionErrors)] {
+ extends DeleteDelataResult[SegmentStatus, (SegmentUpdateDetails, ExecutionErrors, Long)] {
override def getKey(key: SegmentStatus,
- value: (SegmentUpdateDetails, ExecutionErrors)): (SegmentStatus, (SegmentUpdateDetails,
- ExecutionErrors)) = {
+ value: (SegmentUpdateDetails, ExecutionErrors, Long)): (SegmentStatus, (SegmentUpdateDetails,
+ ExecutionErrors, Long)) = {
(key, value)
}
}
-
trait PartitionResult[K, V] extends Serializable {
def getKey(key: Int, value: Boolean): (K, V)
-
}
class PartitionResultImpl extends PartitionResult[Int, Boolean] {
diff --git a/integration/spark-common/src/main/scala/org/apache/spark/util/CarbonReflectionUtils.scala b/integration/spark-common/src/main/scala/org/apache/spark/util/CarbonReflectionUtils.scala
index 4fc30d02..46692df 100644
--- a/integration/spark-common/src/main/scala/org/apache/spark/util/CarbonReflectionUtils.scala
+++ b/integration/spark-common/src/main/scala/org/apache/spark/util/CarbonReflectionUtils.scala
@@ -26,6 +26,7 @@ import org.apache.spark.{SPARK_VERSION, SparkContext}
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
import org.apache.spark.sql.catalyst.{InternalRow, TableIdentifier}
+import org.apache.spark.sql.catalyst.analysis.Analyzer
import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
@@ -381,4 +382,19 @@ object CarbonReflectionUtils {
nameField.setAccessible(true)
nameField.set(caseObj, objToSet)
}
+
+ def invokeAnalyzerExecute(analyzer: Analyzer,
+ plan: LogicalPlan): LogicalPlan = {
+ if (SparkUtil.isSparkVersionEqualTo("2.1") || SparkUtil.isSparkVersionEqualTo("2.2")) {
+ val method: Method = analyzer.getClass
+ .getMethod("execute", classOf[LogicalPlan])
+ method.invoke(analyzer, plan).asInstanceOf[LogicalPlan]
+ } else if (SparkUtil.isSparkVersionEqualTo("2.3")) {
+ val method: Method = analyzer.getClass
+ .getMethod("executeAndCheck", classOf[LogicalPlan])
+ method.invoke(analyzer, plan).asInstanceOf[LogicalPlan]
+ } else {
+ throw new UnsupportedOperationException("Spark version not supported")
+ }
+ }
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
index 160c785..5f745d2 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonCatalystOperators.scala
@@ -68,8 +68,8 @@ case class ExcludeProfile(attributes: Seq[Attribute]) extends CarbonProfile(attr
case class ProjectForUpdate(
table: UnresolvedRelation,
columns: List[String],
- children: Seq[LogicalPlan] ) extends LogicalPlan {
- override def output: Seq[AttributeReference] = Seq.empty
+ children: Seq[LogicalPlan]) extends LogicalPlan {
+ override def output: Seq[Attribute] = Seq.empty
}
case class UpdateTable(
@@ -79,7 +79,7 @@ case class UpdateTable(
alias: Option[String] = None,
filer: String) extends LogicalPlan {
override def children: Seq[LogicalPlan] = Seq.empty
- override def output: Seq[AttributeReference] = Seq.empty
+ override def output: Seq[Attribute] = Seq.empty
}
case class DeleteRecords(
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala
index ae1d848..45c73ac 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala
@@ -20,8 +20,10 @@ package org.apache.spark.sql.execution.command.mutation
import scala.collection.JavaConverters._
import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.execution.command._
+import org.apache.spark.sql.types.LongType
import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.common.logging.LogServiceFactory
@@ -48,6 +50,10 @@ private[sql] case class CarbonProjectForDeleteCommand(
timestamp: String)
extends DataCommand {
+ override val output: Seq[Attribute] = {
+ Seq(AttributeReference("Deleted Row Count", LongType, nullable = false)())
+ }
+
override def processData(sparkSession: SparkSession): Seq[Row] = {
val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
val carbonTable = CarbonEnv.getCarbonTable(databaseNameOp, tableName)(sparkSession)
@@ -104,7 +110,7 @@ private[sql] case class CarbonProjectForDeleteCommand(
// handle the clean up of IUD.
CarbonUpdateUtil.cleanUpDeltaFiles(carbonTable, false)
- val deletedSegments = DeleteExecution.deleteDeltaExecution(
+ val (deletedSegments, deletedRowCount) = DeleteExecution.deleteDeltaExecution(
databaseNameOp,
tableName,
sparkSession,
@@ -112,16 +118,18 @@ private[sql] case class CarbonProjectForDeleteCommand(
timestamp,
isUpdateOperation = false,
executorErrors)
+
+ // Check for any failures occured during delete delta execution
+ if (executorErrors.failureCauses != FailureCauses.NONE) {
+ throw new Exception(executorErrors.errorMsg)
+ }
+
// call IUD Compaction.
HorizontalCompaction.tryHorizontalCompaction(sparkSession, carbonTable,
isUpdateOperation = false)
DeleteExecution.clearDistributedSegmentCache(carbonTable, deletedSegments)(sparkSession)
- if (executorErrors.failureCauses != FailureCauses.NONE) {
- throw new Exception(executorErrors.errorMsg)
- }
-
val allDataMapSchemas = DataMapStoreManager.getInstance
.getDataMapSchemasOfTable(carbonTable).asScala
.filter(dataMapSchema => null != dataMapSchema.getRelationIdentifier &&
@@ -134,11 +142,13 @@ private[sql] case class CarbonProjectForDeleteCommand(
val deleteFromTablePostEvent: DeleteFromTablePostEvent =
DeleteFromTablePostEvent(sparkSession, carbonTable)
OperationListenerBus.getInstance.fireEvent(deleteFromTablePostEvent, operationContext)
+ Seq(Row(deletedRowCount))
} catch {
case e: HorizontalCompactionException =>
LOGGER.error("Delete operation passed. Exception in Horizontal Compaction." +
" Please check logs. " + e.getMessage)
CarbonUpdateUtil.cleanStaleDeltaFiles(carbonTable, e.compactionTimeStamp.toString)
+ Seq(Row(0L))
case e: Exception =>
LOGGER.error("Exception in Delete data operation " + e.getMessage, e)
@@ -160,7 +170,6 @@ private[sql] case class CarbonProjectForDeleteCommand(
updateLock.unlock()
compactionLock.unlock()
}
- Seq.empty
}
override protected def opName: String = "DELETE DATA"
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala
index b620e38..686990d 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala
@@ -20,11 +20,12 @@ package org.apache.spark.sql.execution.command.mutation
import scala.collection.JavaConverters._
import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
import org.apache.spark.sql.execution.command._
import org.apache.spark.sql.execution.command.management.CarbonLoadDataCommand
import org.apache.spark.sql.execution.datasources.LogicalRelation
-import org.apache.spark.sql.types.ArrayType
+import org.apache.spark.sql.types.{ArrayType, LongType}
import org.apache.spark.storage.StorageLevel
import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
@@ -50,8 +51,13 @@ private[sql] case class CarbonProjectForUpdateCommand(
columns: List[String])
extends DataCommand {
+ override val output: Seq[Attribute] = {
+ Seq(AttributeReference("Updated Row Count", LongType, nullable = false)())
+ }
+
override def processData(sparkSession: SparkSession): Seq[Row] = {
val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+ var updatedRowCount = 0L
IUDCommonUtil.checkIfSegmentListIsSet(sparkSession, plan)
val res = plan find {
case relation: LogicalRelation if relation.relation
@@ -61,7 +67,7 @@ private[sql] case class CarbonProjectForUpdateCommand(
}
if (res.isEmpty) {
- return Seq.empty
+ return Array(Row(updatedRowCount)).toSeq
}
val carbonTable = CarbonEnv.getCarbonTable(databaseNameOp, tableName)(sparkSession)
if (carbonTable.getPartitionInfo != null &&
@@ -71,6 +77,7 @@ private[sql] case class CarbonProjectForUpdateCommand(
throw new UnsupportedOperationException("Unsupported update operation for range/" +
"hash/list partition table")
}
+
setAuditTable(carbonTable)
setAuditInfo(Map("plan" -> plan.simpleString))
columns.foreach { col =>
@@ -135,7 +142,7 @@ private[sql] case class CarbonProjectForUpdateCommand(
CarbonUpdateUtil.cleanUpDeltaFiles(carbonTable, false)
// do delete operation.
- val segmentsToBeDeleted = DeleteExecution.deleteDeltaExecution(
+ val (segmentsToBeDeleted, updatedRowCountTemp) = DeleteExecution.deleteDeltaExecution(
databaseNameOp,
tableName,
sparkSession,
@@ -148,6 +155,7 @@ private[sql] case class CarbonProjectForUpdateCommand(
throw new Exception(executionErrors.errorMsg)
}
+ updatedRowCount = updatedRowCountTemp
// do update operation.
performUpdate(dataSet,
databaseNameOp,
@@ -217,7 +225,7 @@ private[sql] case class CarbonProjectForUpdateCommand(
CarbonLockUtil.fileUnlock(metadataLock, LockUsage.METADATA_LOCK)
}
}
- Seq.empty
+ Seq(Row(updatedRowCount))
}
private def performUpdate(
@@ -304,9 +312,6 @@ private[sql] case class CarbonProjectForUpdateCommand(
executorErrors.errorMsg = updateTableModel.executorErrors.errorMsg
executorErrors.failureCauses = updateTableModel.executorErrors.failureCauses
-
- Seq.empty
-
}
override protected def opName: String = "UPDATE DATA"
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala
index f9428a2..cb86cb5 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/DeleteExecution.scala
@@ -64,14 +64,15 @@ object DeleteExecution {
dataRdd: RDD[Row],
timestamp: String,
isUpdateOperation: Boolean,
- executorErrors: ExecutionErrors): Seq[Segment] = {
+ executorErrors: ExecutionErrors): (Seq[Segment], Long) = {
- var res: Array[List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors))]] = null
+ var res: Array[List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors, Long))]] = null
val database = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
val carbonTable = CarbonEnv.getCarbonTable(databaseNameOp, tableName)(sparkSession)
val absoluteTableIdentifier = carbonTable.getAbsoluteTableIdentifier
val tablePath = absoluteTableIdentifier.getTablePath
var segmentsTobeDeleted = Seq.empty[Segment]
+ var operatedRowCount = 0L
val deleteRdd = if (isUpdateOperation) {
val schema =
@@ -97,7 +98,7 @@ object DeleteExecution {
// if no loads are present then no need to do anything.
if (keyRdd.partitions.length == 0) {
- return segmentsTobeDeleted
+ return (segmentsTobeDeleted, operatedRowCount)
}
val blockMappingVO =
carbonInputFormat.getBlockRowCount(
@@ -124,9 +125,9 @@ object DeleteExecution {
val rdd = rowContRdd.join(keyRdd)
res = rdd.mapPartitionsWithIndex(
(index: Int, records: Iterator[((String), (RowCountDetailsVO, Iterable[Row]))]) =>
- Iterator[List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors))]] {
+ Iterator[List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors, Long))]] {
ThreadLocalSessionInfo.setConfigurationToCurrentThread(conf.value.value)
- var result = List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors))]()
+ var result = List[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors, Long))]()
while (records.hasNext) {
val ((key), (rowCountDetailsVO, groupedRows)) = records.next
val segmentId = key.substring(0, key.indexOf(CarbonCommonConstants.FILE_SEPARATOR))
@@ -143,7 +144,7 @@ object DeleteExecution {
// if no loads are present then no need to do anything.
if (res.flatten.isEmpty) {
- return segmentsTobeDeleted
+ return (segmentsTobeDeleted, operatedRowCount)
}
// update new status file
@@ -217,7 +218,7 @@ object DeleteExecution {
timestamp: String,
rowCountDetailsVO: RowCountDetailsVO,
isStandardTable: Boolean
- ): Iterator[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors))] = {
+ ): Iterator[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors, Long))] = {
val result = new DeleteDelataResultImpl()
var deleteStatus = SegmentStatus.LOAD_FAILURE
@@ -228,7 +229,8 @@ object DeleteExecution {
CarbonTablePath.addDataPartPrefix(key.split(CarbonCommonConstants.FILE_SEPARATOR)(1)))
val segmentId = key.split(CarbonCommonConstants.FILE_SEPARATOR)(0)
val deleteDeltaBlockDetails: DeleteDeltaBlockDetails = new DeleteDeltaBlockDetails(blockName)
- val resultIter = new Iterator[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors))] {
+ val resultIter =
+ new Iterator[(SegmentStatus, (SegmentUpdateDetails, ExecutionErrors, Long))] {
val segmentUpdateDetails = new SegmentUpdateDetails()
var TID = ""
var countOfRows = 0
@@ -305,15 +307,18 @@ object DeleteExecution {
}
}
- override def next(): (SegmentStatus, (SegmentUpdateDetails, ExecutionErrors)) = {
+ override def next(): (SegmentStatus, (SegmentUpdateDetails, ExecutionErrors, Long)) = {
finished = true
- result.getKey(deleteStatus, (segmentUpdateDetails, executorErrors))
+ result.getKey(deleteStatus, (segmentUpdateDetails, executorErrors, countOfRows.toLong))
}
}
resultIter
}
- segmentsTobeDeleted
+ if (executorErrors.failureCauses == FailureCauses.NONE) {
+ operatedRowCount = res.flatten.map(_._2._3).sum
+ }
+ (segmentsTobeDeleted, operatedRowCount)
}
def clearDistributedSegmentCache(carbonTable: CarbonTable,
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala
index 9ba2301..9b923b0 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala
@@ -40,6 +40,8 @@ import org.apache.carbondata.core.util.CarbonUtil
case class CarbonIUDAnalysisRule(sparkSession: SparkSession) extends Rule[LogicalPlan] {
private lazy val parser = sparkSession.sessionState.sqlParser
+ private lazy val optimizer = sparkSession.sessionState.optimizer
+ private lazy val analyzer = sparkSession.sessionState.analyzer
private def processUpdateQuery(
table: UnresolvedRelation,
@@ -181,7 +183,15 @@ case class CarbonIUDAnalysisRule(sparkSession: SparkSession) extends Rule[Logica
}
val destinationTable = CarbonReflectionUtils.getUnresolvedRelation(table.tableIdentifier, alias)
- ProjectForUpdate(destinationTable, columns, Seq(finalPlan))
+ // In Spark 2.1 and 2.2, it uses Analyzer.execute method to transform LogicalPlan
+ // but in Spark 2.3, it uses Analyzer.executeAndCheck method
+ val analyzedPlan = CarbonReflectionUtils.invokeAnalyzerExecute(
+ analyzer, ProjectForUpdate(destinationTable, columns, Seq(finalPlan)))
+ // For all commands, they execute eagerly, and will be transformed to
+ // logical plan 'LocalRelation' in analyze phase(please see the code in 'Dataset.logicalPlan'),
+ // so it needs to return logical plan 'CarbonProjectForUpdateCommand' here
+ // instead of 'ProjectForUpdate'
+ optimizer.execute(analyzedPlan)
}
[carbondata] 19/33: [CARBONDATA-3515] Limit local dictionary size
to 16MB and allow configuration.
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit f750b6f210ba87923793631f6b4a2cc4f7dbdd3d
Author: ajantha-bhat <aj...@gmail.com>
AuthorDate: Tue Sep 10 10:48:26 2019 +0530
[CARBONDATA-3515] Limit local dictionary size to 16MB and allow configuration.
problem: currently local dictionary max size is 2GB, because of this, for varchar columns or long string columns,
local dictionary can be of 2GB size. so, as local dictionary is stored in blocklet. blocklet size will exceed 2 GB,
even though configured maximum blocklet size is 64MB. some places inter overflow happens during casting.
solution: Limit local dictionary size to 16MB and allow configuration. default size is 4MB
This closes #3380
---
.../core/constants/CarbonCommonConstants.java | 11 ++++++
.../dictionaryholder/MapBasedDictionaryStore.java | 16 ++++++--
.../carbondata/core/util/CarbonProperties.java | 43 ++++++++++++++++++++++
docs/configuration-parameters.md | 1 +
4 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
index 67fa13f..ac77582 100644
--- a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
+++ b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
@@ -1209,6 +1209,17 @@ public final class CarbonCommonConstants {
public static final String CARBON_ENABLE_RANGE_COMPACTION_DEFAULT = "true";
+ @CarbonProperty
+ /**
+ * size based threshold for local dictionary in mb.
+ */
+ public static final String CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB =
+ "carbon.local.dictionary.size.threshold.inmb";
+
+ public static final int CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB_DEFAULT = 4;
+
+ public static final int CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB_MAX = 16;
+
//////////////////////////////////////////////////////////////////////////////////////////
// Query parameter start here
//////////////////////////////////////////////////////////////////////////////////////////
diff --git a/core/src/main/java/org/apache/carbondata/core/localdictionary/dictionaryholder/MapBasedDictionaryStore.java b/core/src/main/java/org/apache/carbondata/core/localdictionary/dictionaryholder/MapBasedDictionaryStore.java
index 7b8617a..0a50451 100644
--- a/core/src/main/java/org/apache/carbondata/core/localdictionary/dictionaryholder/MapBasedDictionaryStore.java
+++ b/core/src/main/java/org/apache/carbondata/core/localdictionary/dictionaryholder/MapBasedDictionaryStore.java
@@ -20,7 +20,9 @@ import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import org.apache.carbondata.core.cache.dictionary.DictionaryByteArrayWrapper;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
import org.apache.carbondata.core.localdictionary.exception.DictionaryThresholdReachedException;
+import org.apache.carbondata.core.util.CarbonProperties;
/**
* Map based dictionary holder class, it will use map to hold
@@ -51,6 +53,11 @@ public class MapBasedDictionaryStore implements DictionaryStore {
private int dictionaryThreshold;
/**
+ * dictionary threshold size in bytes
+ */
+ private long dictionarySizeThresholdInBytes;
+
+ /**
* for checking threshold is reached or not
*/
private boolean isThresholdReached;
@@ -62,6 +69,8 @@ public class MapBasedDictionaryStore implements DictionaryStore {
public MapBasedDictionaryStore(int dictionaryThreshold) {
this.dictionaryThreshold = dictionaryThreshold;
+ this.dictionarySizeThresholdInBytes = Integer.parseInt(CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB)) << 20;
this.dictionary = new ConcurrentHashMap<>();
this.referenceDictionaryArray = new DictionaryByteArrayWrapper[dictionaryThreshold];
}
@@ -93,7 +102,7 @@ public class MapBasedDictionaryStore implements DictionaryStore {
value = ++lastAssignValue;
currentSize += data.length;
// if new value is greater than threshold
- if (value > dictionaryThreshold || currentSize >= Integer.MAX_VALUE) {
+ if (value > dictionaryThreshold || currentSize > dictionarySizeThresholdInBytes) {
// set the threshold boolean to true
isThresholdReached = true;
// throw exception
@@ -111,9 +120,10 @@ public class MapBasedDictionaryStore implements DictionaryStore {
private void checkIfThresholdReached() throws DictionaryThresholdReachedException {
if (isThresholdReached) {
- if (currentSize >= Integer.MAX_VALUE) {
+ if (currentSize > dictionarySizeThresholdInBytes) {
throw new DictionaryThresholdReachedException(
- "Unable to generate dictionary. Dictionary Size crossed 2GB limit");
+ "Unable to generate dictionary. Dictionary Size crossed bytes: "
+ + dictionarySizeThresholdInBytes);
} else {
throw new DictionaryThresholdReachedException(
"Unable to generate dictionary value. Dictionary threshold reached");
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java b/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
index adf4905..e4efc0b 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
@@ -202,6 +202,9 @@ public final class CarbonProperties {
case CarbonCommonConstants.CARBON_INDEX_SERVER_SERIALIZATION_THRESHOLD:
validateIndexServerSerializationThreshold();
break;
+ case CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB:
+ validateAndGetLocalDictionarySizeThresholdInMB();
+ break;
// TODO : Validation for carbon.lock.type should be handled for addProperty flow
default:
// none
@@ -268,6 +271,7 @@ public final class CarbonProperties {
validateStringCharacterLimit();
validateDetailQueryBatchSize();
validateIndexServerSerializationThreshold();
+ validateAndGetLocalDictionarySizeThresholdInMB();
}
/**
@@ -1789,4 +1793,43 @@ public final class CarbonProperties {
return !prefetchEnable.equalsIgnoreCase("false");
}
}
+
+ /**
+ * get local dictionary size threshold in mb.
+ */
+ private void validateAndGetLocalDictionarySizeThresholdInMB() {
+ String sizeStr = carbonProperties
+ .getProperty(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB);
+ String defaultValue = Integer
+ .toString(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB_DEFAULT);
+ if (sizeStr == null) {
+ carbonProperties
+ .setProperty(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB,
+ defaultValue);
+ } else {
+ try {
+ int size = Integer.parseInt(sizeStr);
+ if (size < 0 || size == 0
+ || size > CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB_MAX) {
+ LOGGER.info("using default value of carbon.local.dictionary.size.threshold.inmb = "
+ + defaultValue);
+ carbonProperties
+ .setProperty(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB,
+ defaultValue);
+ } else {
+ LOGGER.info("using carbon.local.dictionary.size.threshold.inmb = " + size);
+ carbonProperties
+ .setProperty(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB,
+ Integer.toString(size));
+ }
+ } catch (Exception ex) {
+ LOGGER.info(
+ "using default value of carbon.local.dictionary.size.threshold.inmb = " + defaultValue);
+ carbonProperties
+ .setProperty(CarbonCommonConstants.CARBON_LOCAL_DICTIONARY_SIZE_THRESHOLD_IN_MB,
+ defaultValue);
+ }
+ }
+ }
+
}
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index da226ec..51017fe 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -96,6 +96,7 @@ This section provides the details of all the configurations required for the Car
| carbon.minmax.allowed.byte.count | 200 | CarbonData will write the min max values for string/varchar types column using the byte count specified by this configuration. Max value is 1000 bytes(500 characters) and Min value is 10 bytes(5 characters). **NOTE:** This property is useful for reducing the store size thereby improving the query performance but can lead to query degradation if value is not configured properly. | |
| carbon.merge.index.failure.throw.exception | true | It is used to configure whether or not merge index failure should result in data load failure also. |
| carbon.binary.decoder | None | Support configurable decode for loading. Two decoders supported: base64 and hex |
+| carbon.local.dictionary.size.threshold.inmb | 4 | size based threshold for local dictionary in MB, maximum allowed size is 16 MB. |
## Compaction Configuration
[carbondata] 04/33: [CARBONDATA-3466] Fix NPE for carboncli command
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 019c777b7b25f7fd287bb49613a46e5d07fe78b7
Author: Manhua <ke...@qq.com>
AuthorDate: Mon Jul 8 10:37:50 2019 +0800
[CARBONDATA-3466] Fix NPE for carboncli command
What is changed:
require option for carboncli command when parsing
unify to fill result in array when running carbon cli , use adapter to print to stream on demand
Problem details:
if no options specified, like "carboncli for table source", NPE will occur when running commandOptions.split("\\s+") in CarbonCliCommand#processData.
Since option, as a role of command, is a must in CarbonCli, we require it when parsing command in this PR.
This closes #3318
---
.../TestAlterTableSortColumnsProperty.scala | 1 -
.../command/management/CarbonCliCommand.scala | 2 +-
.../spark/sql/parser/CarbonSpark2SqlParser.scala | 11 +--
.../carbondata/spark/testsuite/TestCarbonCli.scala | 82 ++++++++++++++++++++
.../java/org/apache/carbondata/tool/CarbonCli.java | 88 ++++++++--------------
5 files changed, 116 insertions(+), 68 deletions(-)
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableSortColumnsProperty.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableSortColumnsProperty.scala
index dab9934..2ac6889 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableSortColumnsProperty.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/alterTable/TestAlterTableSortColumnsProperty.scala
@@ -601,7 +601,6 @@ class TestAlterTableSortColumnsProperty extends QueryTest with BeforeAndAfterAll
val out: ByteArrayOutputStream = new ByteArrayOutputStream
val stream: PrintStream = new PrintStream(out)
CarbonCli.run(args, stream)
- CarbonCli.cleanOutPuts()
val output: String = new String(out.toByteArray)
if (segmentId == 2) {
assertResult(s"Input Folder: $segmentPath\nsorted by intfield,stringfield\n")(output)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonCliCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonCliCommand.scala
index d1a54d0..5dd0c12 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonCliCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonCliCommand.scala
@@ -56,7 +56,7 @@ case class CarbonCliCommand(
case x => Seq(x.trim)
}.flatten
val summaryOutput = new util.ArrayList[String]()
- CarbonCli.run(finalCommands.toArray, summaryOutput)
+ CarbonCli.run(finalCommands.toArray, summaryOutput, false)
summaryOutput.asScala.map(x =>
Row(x)
)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
index 6db6d01..22548ff 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
@@ -523,14 +523,9 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
}
protected lazy val cli: Parser[LogicalPlan] =
- (CARBONCLI ~> FOR ~> TABLE) ~> (ident <~ ".").? ~ ident ~
- (OPTIONS ~> "(" ~> commandOptions <~ ")").? <~
- opt(";") ^^ {
- case databaseName ~ tableName ~ commandList =>
- var commandOptions: String = null
- if (commandList.isDefined) {
- commandOptions = commandList.get
- }
+ CARBONCLI ~> FOR ~> TABLE ~> (ident <~ ".").? ~ ident ~
+ (OPTIONS ~> "(" ~> commandOptions <~ ")") <~ opt(";") ^^ {
+ case databaseName ~ tableName ~ commandOptions =>
CarbonCliCommand(
convertDbNameToLowerCase(databaseName),
tableName.toLowerCase(),
diff --git a/integration/spark2/src/test/scala/org/apache/carbondata/spark/testsuite/TestCarbonCli.scala b/integration/spark2/src/test/scala/org/apache/carbondata/spark/testsuite/TestCarbonCli.scala
new file mode 100644
index 0000000..ba54e8f
--- /dev/null
+++ b/integration/spark2/src/test/scala/org/apache/carbondata/spark/testsuite/TestCarbonCli.scala
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.common.util.Spark2QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+class TestCarbonCli extends Spark2QueryTest with BeforeAndAfterAll{
+
+ override protected def beforeAll(): Unit = {
+ sql("drop table if exists OneRowTable")
+ sql("create table OneRowTable(col1 string, col2 string, col3 int, col4 double) stored by 'carbondata'")
+ sql("insert into OneRowTable select '0.1', 'a.b', 1, 1.2")
+ }
+
+ test("CarbonCli table summary") {
+ checkExistence(
+ sql("carboncli for table OneRowTable options('-cmd summary -a')"),
+ true, "## Summary")
+
+ checkExistence(
+ sql("carboncli for table OneRowTable options('-cmd summary -v')"),
+ true, "## version Details")
+
+ checkExistence(
+ sql("carboncli for table OneRowTable options('-cmd summary -s')"),
+ true, "## Schema")
+
+ checkExistence(
+ sql("carboncli for table OneRowTable options('-cmd summary -t')"),
+ true, "## Table Properties")
+
+ checkExistence(
+ sql("carboncli for table OneRowTable options('-cmd summary -m')"),
+ true, "## Segment")
+ }
+
+ test("CarbonCli column details") {
+ checkExistence(
+ sql("carboncli for table OneRowTable options('-cmd summary -c col1')"),
+ true, "## Column Statistics for 'col1'")
+ }
+
+ test("CarbonCli benchmark") {
+ checkExistence(
+ sql("carboncli for table OneRowTable options('-cmd benchmark -c col1')"),
+ true, "## Benchmark")
+ }
+
+ test("CarbonCli invalid cmd"){
+
+ assert(intercept[AnalysisException] {
+ sql("carboncli for table OneRowTable").show()
+ }.getMessage().contains("mismatched input 'carboncli'"))
+
+ assert(intercept[Exception] {
+ sql("carboncli for table OneRowTable options('')")
+ }.getMessage().contains("Missing required option: cmd"))
+
+ checkExistence(sql("carboncli for table OneRowTable options('-cmd test')"),
+ true, "command test is not supported")
+ }
+
+ override protected def afterAll(): Unit = {
+ sql("drop table if exists OneRowTable")
+ }
+}
diff --git a/tools/cli/src/main/java/org/apache/carbondata/tool/CarbonCli.java b/tools/cli/src/main/java/org/apache/carbondata/tool/CarbonCli.java
index d562b8e..ef9a50e 100644
--- a/tools/cli/src/main/java/org/apache/carbondata/tool/CarbonCli.java
+++ b/tools/cli/src/main/java/org/apache/carbondata/tool/CarbonCli.java
@@ -22,7 +22,6 @@ import java.io.PrintStream;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.ArrayList;
-import java.util.List;
import org.apache.carbondata.common.annotations.InterfaceAudience;
import org.apache.carbondata.common.annotations.InterfaceStability;
@@ -44,12 +43,6 @@ import org.apache.commons.cli.PosixParser;
@InterfaceStability.Unstable
public class CarbonCli {
- // List to collect all the outputs of option details
- private static List<String> outPuts;
-
- // a boolean variable to decide whether to print the output in console or return the list,
- // by default true, and it will be set to false if the cli is trigerred via sql command
- private static boolean isPrintInConsole = true;
static class OptionsHolder {
static Options instance = buildOptions();
@@ -117,47 +110,45 @@ public class CarbonCli {
run(args, System.out);
}
- public static void run(String[] args, ArrayList<String> e) {
- // this boolean to check whether to print in console or not
- isPrintInConsole = false;
- outPuts = e;
- Options options = OptionsHolder.instance;
- CommandLineParser parser = new PosixParser();
- CommandLine line;
- try {
- line = parser.parse(options, args);
- } catch (ParseException ex) {
- throw new RuntimeException("Parsing failed. Reason: " + ex.getMessage(), ex);
+ /**
+ * adapter to run CLI and print to stream
+ * @param args input arguments
+ * @param out output stream
+ */
+ public static void run(String[] args, PrintStream out) {
+ ArrayList<String> outputs = new ArrayList<String>();
+ run(args, outputs, true);
+ for (String line: outputs) {
+ out.println(line);
}
-
- runCli(System.out, options, line);
}
- public static void run(String[] args, PrintStream out) {
+
+ /**
+ * run CLI and fill result into array
+ * @param args input arguments
+ * @param outPuts array for filling result
+ * @param isPrintInConsole flag to decide whether to print error in console or return the list
+ */
+ public static void run(String[] args, ArrayList<String> outPuts, boolean isPrintInConsole) {
Options options = OptionsHolder.instance;
CommandLineParser parser = new PosixParser();
- CommandLine line;
+ CommandLine line = null;
try {
line = parser.parse(options, args);
- } catch (ParseException exp) {
- out.println("Parsing failed. Reason: " + exp.getMessage());
- return;
+ } catch (ParseException ex) {
+ if (isPrintInConsole) {
+ outPuts.add("Parsing failed. Reason: " + ex.getMessage());
+ return;
+ } else {
+ throw new RuntimeException("Parsing failed. Reason: " + ex.getMessage(), ex);
+ }
}
- runCli(out, options, line);
- }
-
- private static void runCli(PrintStream out, Options options, CommandLine line) {
- if (outPuts == null) {
- outPuts = new ArrayList<>();
- }
if (line.hasOption("h")) {
- collectHelpInfo(options);
- for (String output : outPuts) {
- out.println(output);
- }
+ outPuts.add(collectHelpInfo(options));
return;
}
@@ -180,48 +171,29 @@ public class CarbonCli {
} catch (IOException e) {
e.printStackTrace();
}
- for (String output : outPuts) {
- out.println(output);
- }
}
return;
} else {
- out.println("command " + cmd + " is not supported");
outPuts.add("command " + cmd + " is not supported");
- collectHelpInfo(options);
- for (String output : outPuts) {
- out.println(output);
- }
+ outPuts.add(collectHelpInfo(options));
return;
}
try {
command.run(line);
- if (isPrintInConsole) {
- for (String output : outPuts) {
- out.println(output);
- }
- }
- out.flush();
} catch (IOException | MemoryException e) {
e.printStackTrace();
- } finally {
- out.close();
}
}
- private static void collectHelpInfo(Options options) {
+ private static String collectHelpInfo(Options options) {
HelpFormatter formatter = new HelpFormatter();
StringWriter stringWriter = new StringWriter();
PrintWriter printWriter = new PrintWriter(stringWriter);
formatter.printHelp(printWriter, formatter.getWidth(), "CarbonCli", null, options,
formatter.getLeftPadding(), formatter.getDescPadding(), null, false);
printWriter.flush();
- outPuts.add(stringWriter.toString());
- }
-
- public static void cleanOutPuts() {
- outPuts = null;
+ return stringWriter.toString();
}
}
[carbondata] 25/33: [DOC] Update doc for alter sort_columns
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit ab86705f64f460b639e05a110d6a4e13977cc773
Author: QiangCai <qi...@qq.com>
AuthorDate: Fri Sep 20 09:54:52 2019 +0800
[DOC] Update doc for alter sort_columns
This closes #3395
---
docs/ddl-of-carbondata.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/ddl-of-carbondata.md b/docs/ddl-of-carbondata.md
index 7ab0e5f..9c9a02f 100644
--- a/docs/ddl-of-carbondata.md
+++ b/docs/ddl-of-carbondata.md
@@ -817,7 +817,7 @@ Users can specify which columns to include and exclude for local dictionary gene
```
**NOTE:**
- * The future version will enhance "custom" compaction to sort the old segment one by one.
+ * The "custom" compaction support re-sorting the old segment one by one in version 1.6 or later.
* The streaming table is not supported for SORT_COLUMNS modification.
* If the inverted index columns are removed from the new SORT_COLUMNS, they will not
create the inverted index. But the old configuration of INVERTED_INDEX will be kept.
[carbondata] 16/33: [CARBONDATA-3508] Support CG datamap pruning
fallback while querying
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 0e2d3e20cad0e7032d959c5f9107249eaa258685
Author: shivamasn <sh...@gmail.com>
AuthorDate: Thu Aug 29 11:49:41 2019 +0530
[CARBONDATA-3508] Support CG datamap pruning fallback while querying
Problem: Select query fails when the cg datamap is dropped concurrently
while running the select query on filter column on which datamap is created.
Solution: Handle the exception from datamap blocklet pruning if
it fails and consider only the pruned blocklets from default datamap pruning.
This closes #3369
---
.../core/indexstore/BlockletDataMapIndexStore.java | 2 +-
.../statusmanager/SegmentUpdateStatusManager.java | 6 ++--
.../datamap/bloom/BloomCoarseGrainDataMap.java | 2 +-
.../carbondata/hadoop/api/CarbonInputFormat.java | 32 ++++++++++++++--------
4 files changed, 27 insertions(+), 15 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java b/core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
index 32ee9cb..fd549e0 100644
--- a/core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
+++ b/core/src/main/java/org/apache/carbondata/core/indexstore/BlockletDataMapIndexStore.java
@@ -80,7 +80,7 @@ public class BlockletDataMapIndexStore
return get(identifierWrapper, null);
}
- private BlockletDataMapIndexWrapper get(TableBlockIndexUniqueIdentifierWrapper identifierWrapper,
+ public BlockletDataMapIndexWrapper get(TableBlockIndexUniqueIdentifierWrapper identifierWrapper,
Map<String, Map<String, BlockMetaInfo>> segInfoCache) throws IOException {
TableBlockIndexUniqueIdentifier identifier =
identifierWrapper.getTableBlockIndexUniqueIdentifier();
diff --git a/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java b/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
index f7083dc..bc794f4 100644
--- a/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
+++ b/core/src/main/java/org/apache/carbondata/core/statusmanager/SegmentUpdateStatusManager.java
@@ -27,8 +27,10 @@ import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.util.ArrayList;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
@@ -790,8 +792,8 @@ public class SegmentUpdateStatusManager {
final long deltaEndTimestamp =
getEndTimeOfDeltaFile(CarbonCommonConstants.DELETE_DELTA_FILE_EXT, block);
- List<CarbonFile> files =
- new ArrayList<>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE);
+ Set<CarbonFile> files =
+ new HashSet<>(CarbonCommonConstants.DEFAULT_COLLECTION_SIZE);
for (CarbonFile eachFile : allSegmentFiles) {
String fileName = eachFile.getName();
diff --git a/datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java b/datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
index fea48c3..f931353 100644
--- a/datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
+++ b/datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
@@ -232,7 +232,7 @@ public class BloomCoarseGrainDataMap extends CoarseGrainDataMap {
LOGGER.warn(String.format("HitBlocklets is empty in bloom filter prune method. " +
"bloomQueryModels size is %d, filterShards size if %d",
bloomQueryModels.size(), filteredShard.size()));
- return null;
+ return new ArrayList<>();
}
return new ArrayList<>(hitBlocklets);
}
diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
index ac9e11e..45041e4 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java
@@ -573,19 +573,29 @@ m filterExpression
if (cgDataMapExprWrapper != null) {
// Prune segments from already pruned blocklets
DataMapUtil.pruneSegments(segmentIds, prunedBlocklets);
- List<ExtendedBlocklet> cgPrunedBlocklets;
+ List<ExtendedBlocklet> cgPrunedBlocklets = new ArrayList<>();
+ boolean isCGPruneFallback = false;
// Again prune with CG datamap.
- if (distributedCG && dataMapJob != null) {
- cgPrunedBlocklets = DataMapUtil
- .executeDataMapJob(carbonTable, filter.getResolver(), dataMapJob, partitionsToPrune,
- segmentIds, invalidSegments, DataMapLevel.CG, true, new ArrayList<String>());
- } else {
- cgPrunedBlocklets = cgDataMapExprWrapper.prune(segmentIds, partitionsToPrune);
+ try {
+ if (distributedCG && dataMapJob != null) {
+ cgPrunedBlocklets = DataMapUtil
+ .executeDataMapJob(carbonTable, filter.getResolver(), dataMapJob, partitionsToPrune,
+ segmentIds, invalidSegments, DataMapLevel.CG, true, new ArrayList<String>());
+ } else {
+ cgPrunedBlocklets = cgDataMapExprWrapper.prune(segmentIds, partitionsToPrune);
+ }
+ } catch (Exception e) {
+ isCGPruneFallback = true;
+ LOG.error("CG datamap pruning failed.", e);
+ }
+ // If isCGPruneFallback = true, it means that CG datamap pruning failed,
+ // hence no need to do intersect and simply pass the prunedBlocklets from default datamap
+ if (!isCGPruneFallback) {
+ // since index datamap prune in segment scope,
+ // the result need to intersect with previous pruned result
+ prunedBlocklets =
+ intersectFilteredBlocklets(carbonTable, prunedBlocklets, cgPrunedBlocklets);
}
- // since index datamap prune in segment scope,
- // the result need to intersect with previous pruned result
- prunedBlocklets =
- intersectFilteredBlocklets(carbonTable, prunedBlocklets, cgPrunedBlocklets);
if (ExplainCollector.enabled()) {
ExplainCollector.recordCGDataMapPruning(
DataMapWrapperSimpleInfo.fromDataMapWrapper(cgDataMapExprWrapper),
[carbondata] 32/33: [CARBONDATA-3501] Fix update table with varchar
column
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 00d2fe930e713958e052e3a738616a852f1dbfe7
Author: Manhua <ke...@qq.com>
AuthorDate: Wed Sep 25 11:47:01 2019 +0800
[CARBONDATA-3501] Fix update table with varchar column
Problem
Update on table with varchar column will throw exception
Analyse
In the loading part of update operation, it gets the isVarcharTypeMapping for each column in the order when table created. And this gives a hint for checking string length. It does not allow to exceeds 32000 char for a column which is not varchar type.
However when changing the plan for updating in CarbonIUDRule, it first deletes the old expression and appends the new one, which makes the order differ to table created. Such that the string length checking fail.
Solution
Keep the order as table created when modify update plan
This closes #3398
---
.../longstring/VarcharDataTypesBasicTestCase.scala | 10 ++++++++++
.../command/management/CarbonLoadDataCommand.scala | 2 +-
.../org/apache/spark/sql/hive/CarbonAnalysisRules.scala | 4 ++--
.../org/apache/spark/sql/optimizer/CarbonIUDRule.scala | 17 ++++++++++++++---
4 files changed, 27 insertions(+), 6 deletions(-)
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/longstring/VarcharDataTypesBasicTestCase.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/longstring/VarcharDataTypesBasicTestCase.scala
index 4fd2cc0..9719cfc 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/longstring/VarcharDataTypesBasicTestCase.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/longstring/VarcharDataTypesBasicTestCase.scala
@@ -389,6 +389,16 @@ class VarcharDataTypesBasicTestCase extends QueryTest with BeforeAndAfterEach wi
sql("DROP TABLE IF EXISTS varchar_complex_table")
}
+
+ test("update table with long string column") {
+ prepareTable()
+ // update non-varchar column
+ sql(s"update $longStringTable set(id)=(0) where name is not null").show()
+ // update varchar column
+ sql(s"update $longStringTable set(description)=('empty') where name is not null").show()
+ // update non-varchar&varchar column
+ sql(s"update $longStringTable set(description, id)=('sth.', 1) where name is not null").show()
+ }
// ignore this test in CI, because it will need at least 4GB memory to run successfully
ignore("Exceed 2GB per column page for varchar datatype") {
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
index 6a03eab..b2f9a1e 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
@@ -1060,7 +1060,7 @@ case class CarbonLoadDataCommand(
val dropAttributes = df.logicalPlan.output.dropRight(1)
val finalOutput = catalogTable.schema.map { attr =>
dropAttributes.find { d =>
- val index = d.name.lastIndexOf("-updatedColumn")
+ val index = d.name.lastIndexOf(CarbonCommonConstants.UPDATED_COL_EXTENSION)
if (index > 0) {
d.name.substring(0, index).equalsIgnoreCase(attr.name)
} else {
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala
index 9b923b0..d11bf1e 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonAnalysisRules.scala
@@ -122,9 +122,9 @@ case class CarbonIUDAnalysisRule(sparkSession: SparkSession) extends Rule[Logica
val renamedProjectList = projectList.zip(columns).map { case (attr, col) =>
attr match {
case UnresolvedAlias(child22, _) =>
- UnresolvedAlias(Alias(child22, col + "-updatedColumn")())
+ UnresolvedAlias(Alias(child22, col + CarbonCommonConstants.UPDATED_COL_EXTENSION)())
case UnresolvedAttribute(_) =>
- UnresolvedAlias(Alias(attr, col + "-updatedColumn")())
+ UnresolvedAlias(Alias(attr, col + CarbonCommonConstants.UPDATED_COL_EXTENSION)())
case _ => attr
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonIUDRule.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonIUDRule.scala
index ae5825d..da1ca55 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonIUDRule.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonIUDRule.scala
@@ -23,6 +23,8 @@ import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.execution.command.mutation.CarbonProjectForUpdateCommand
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+
/**
* Rule specific for IUD operations
*/
@@ -37,9 +39,9 @@ class CarbonIUDRule extends Rule[LogicalPlan] with PredicateHelper {
var isTransformed = false
val newPlan = updatePlan transform {
case Project(pList, child) if !isTransformed =>
- val (dest: Seq[NamedExpression], source: Seq[NamedExpression]) = pList
+ var (dest: Seq[NamedExpression], source: Seq[NamedExpression]) = pList
.splitAt(pList.size - cols.size)
- val diff = cols.diff(dest.map(_.name.toLowerCase))
+ // check complex column
cols.foreach { col =>
val complexExists = "\"name\":\"" + col + "\""
if (dest.exists(m => m.dataType.json.contains(complexExists))) {
@@ -47,11 +49,20 @@ class CarbonIUDRule extends Rule[LogicalPlan] with PredicateHelper {
"Unsupported operation on Complex data type")
}
}
+ // check updated columns exists in table
+ val diff = cols.diff(dest.map(_.name.toLowerCase))
if (diff.nonEmpty) {
sys.error(s"Unknown column(s) ${ diff.mkString(",") } in table ${ table.tableName }")
}
+ // modify plan for updated column *in place*
isTransformed = true
- Project(dest.filter(a => !cols.contains(a.name.toLowerCase)) ++ source, child)
+ source.foreach { col =>
+ val colName = col.name.substring(0,
+ col.name.lastIndexOf(CarbonCommonConstants.UPDATED_COL_EXTENSION))
+ val updateIdx = dest.indexWhere(_.name.equalsIgnoreCase(colName))
+ dest = dest.updated(updateIdx, col)
+ }
+ Project(dest, child)
}
CarbonProjectForUpdateCommand(
newPlan, table.tableIdentifier.database, table.tableIdentifier.table, cols)
[carbondata] 09/33: [CARBONDATA-3502] Select query with UDF having
Match expression inside IN expression Fails
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 75e207cf7ac3fa262ba04cc3c8d7f2d902256882
Author: manishnalla1994 <ma...@gmail.com>
AuthorDate: Mon Aug 26 17:25:34 2019 +0530
[CARBONDATA-3502] Select query with UDF having Match expression
inside IN expression Fails
Problem: Select query with UDF having Match expression inside
IN expression Fails with ArrayIndexOutOfBounds exception.
Cause: The expression should not be treated as Match expression,
instead should be treated as SparkUnknownExpression.
Solution: Removed the check for Match Expression as it was only added
for Lucene Search mode, which is no longer present.
This closes #3363
---
.../src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala | 2 --
1 file changed, 2 deletions(-)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala
index c4415f8..0fd07bb 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala
@@ -425,8 +425,6 @@ object CarbonFilters {
new AndExpression(l, r)
case strTrim: StringTrim if isStringTrimCompatibleWithCarbon(strTrim) =>
transformExpression(strTrim)
- case s: ScalaUDF =>
- new MatchExpression(s.children.head.toString())
case _ =>
new SparkUnknownExpression(expr.transform {
case AttributeReference(name, dataType, _, _) =>
[carbondata] 28/33: [CARBONDATA-3520] CTAS should fail if select
query contains duplicate columns
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 21bbc4a5306ff850fcf14b488e4cb0452415213c
Author: Indhumathi27 <in...@gmail.com>
AuthorDate: Mon Sep 16 16:25:03 2019 +0530
[CARBONDATA-3520] CTAS should fail if select query contains duplicate columns
Problem:
If Select query contains Duplicate columns, CTAS was creating
a table with only one column, which is wrong
Solution:
Throw error message if Select query contains duplicate columns.
This closes #3388
---
.../createTable/TestCreateTableAsSelect.scala | 37 ++++++++++++++++++++++
.../sql/parser/CarbonSparkSqlParserUtil.scala | 23 +++++++++++---
2 files changed, 56 insertions(+), 4 deletions(-)
diff --git a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableAsSelect.scala b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableAsSelect.scala
index 3896061..8e4d8fa 100644
--- a/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableAsSelect.scala
+++ b/integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableAsSelect.scala
@@ -407,6 +407,43 @@ class TestCreateTableAsSelect extends QueryTest with BeforeAndAfterAll {
checkAnswer(sql("SELECT * FROM target_table"), Seq(Row("shenzhen", 24.5)))
}
+ test("test duplicate columns with select query") {
+ sql("DROP TABLE IF EXISTS target_table")
+ sql("DROP TABLE IF EXISTS source_table")
+ // create carbon table and insert data
+ sql(
+ """
+ | CREATE TABLE source_table(
+ | id INT,
+ | name STRING,
+ | city STRING,
+ | age INT)
+ | STORED BY 'carbondata'
+ | """.stripMargin)
+ sql("INSERT INTO source_table SELECT 1,'bob','shenzhen',27")
+ val e = intercept[AnalysisException] {
+ sql(
+ """
+ | CREATE TABLE target_table
+ | STORED BY 'carbondata'
+ | AS
+ | SELECT t1.city, t2.city
+ | FROM source_table t1, source_table t2 where t1.city=t2.city and t1.city = 'shenzhen'
+ """.stripMargin)
+ }
+ e.getMessage().toString.contains("Duplicated column names found in table definition of " +
+ "`target_table`: [\"city\"]")
+ sql(
+ """
+ | CREATE TABLE target_table
+ | STORED BY 'carbondata'
+ | AS
+ | SELECT t1.city as a, t2.city as b
+ | FROM source_table t1, source_table t2 where t1.city=t2.city and t1.city = 'shenzhen'
+ """.stripMargin)
+ checkAnswer(sql("select * from target_table"), Seq(Row("shenzhen", "shenzhen")))
+ }
+
override def afterAll {
sql("DROP TABLE IF EXISTS carbon_ctas_test")
sql("DROP TABLE IF EXISTS parquet_ctas_test")
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSparkSqlParserUtil.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSparkSqlParserUtil.scala
index 5c008f2..4d85e88 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSparkSqlParserUtil.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSparkSqlParserUtil.scala
@@ -119,6 +119,8 @@ object CarbonSparkSqlParserUtil {
case _ =>
// ignore this case
}
+ val columnNames = fields.map(_.name.get)
+ checkIfDuplicateColumnExists(columns, tableIdentifier, columnNames)
if (partitionFields.nonEmpty && options.isStreaming) {
operationNotAllowed("Streaming is not allowed on partitioned table", partitionColumns)
}
@@ -355,16 +357,29 @@ object CarbonSparkSqlParserUtil {
// Ensuring whether no duplicate name is used in table definition
val colNames: Seq[String] = cols.map(_.name)
+ checkIfDuplicateColumnExists(columns, tableIdentifier, colNames)
+ colNames
+ }
+
+ private def checkIfDuplicateColumnExists(columns: ColTypeListContext,
+ tableIdentifier: TableIdentifier,
+ colNames: Seq[String]): Unit = {
if (colNames.length != colNames.distinct.length) {
val duplicateColumns = colNames.groupBy(identity).collect {
case (x, ys) if ys.length > 1 => "\"" + x + "\""
}
- operationNotAllowed(s"Duplicated column names found in table definition of " +
- s"$tableIdentifier: ${ duplicateColumns.mkString("[", ",", "]") }",
- columns)
+ val errorMessage = s"Duplicated column names found in table definition of " +
+ s"$tableIdentifier: ${ duplicateColumns.mkString("[", ",", "]") }"
+ // In case of create table as select, ColTypeListContext will be null. Check if
+ // duplicateColumns present in column names list, If yes, throw exception
+ if (null != columns) {
+ operationNotAllowed(errorMessage, columns)
+ } else {
+ throw new UnsupportedOperationException(errorMessage)
+ }
}
- colNames
}
+
/**
* The method return's the storage type
* @param createFileFormat
[carbondata] 07/33: [HOTFIX] Remove hive-service from carbondata
assembly jar
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit dff8ab38a986b388182a12f60dd4272659e1cbb6
Author: Zhang Zhichao <44...@qq.com>
AuthorDate: Wed Sep 4 14:36:06 2019 +0800
[HOTFIX] Remove hive-service from carbondata assembly jar
Problem: In some environments, there will occur 'No Such Method: registerCurrentOperationLog' exception while execute sql on carbon thrift server.
Cause: spark hive thrift module rewrite class 'org.apache.hive.service.cli.operation.ExecuteStatementOperation' and add method 'registerCurrentOperationLog' in it, but when start carbon thrift server, it maybe load class 'ExecuteStatementOperation' first from carbondata assembly jar (includes 'org.apache.hive:hive-service'), this class 'ExecuteStatementOperation' which is from hive-service jar doesn't have method 'registerCurrentOperationLog', so it throws NoSuchMethodException.
Solution: remove all artifacts of 'org.apache.hive' when assemble carbondata jar.
This closes #3373
---
assembly/pom.xml | 1 +
1 file changed, 1 insertion(+)
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 12d7e6e..bf729c5 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -107,6 +107,7 @@
<exclude>org.apache.spark:*</exclude>
<exclude>org.apache.zookeeper:*</exclude>
<exclude>org.apache.avro:*</exclude>
+ <exclude>org.apache.hive:*</exclude>
<exclude>com.google.guava:guava</exclude>
<exclude>org.xerial.snappy:snappy-java</exclude>
<!--add more items to be excluded from the assembly-->
[carbondata] 20/33: [HOTFIX] fix missing quotation marks in datamap
doc
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 204f29047b2685d03f27febcec24095f56053a31
Author: lamber-ken <22...@qq.com>
AuthorDate: Wed Sep 11 21:48:21 2019 +0800
[HOTFIX] fix missing quotation marks in datamap doc
This closes #3383
---
docs/datamap/datamap-management.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/datamap/datamap-management.md b/docs/datamap/datamap-management.md
index 199cd14..f910559 100644
--- a/docs/datamap/datamap-management.md
+++ b/docs/datamap/datamap-management.md
@@ -74,7 +74,7 @@ If user perform following command on the main table, system will return failure.
`ALTER TABLE RENAME`. Note that adding a new column is supported, and for dropping columns and
change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if
not, the operation is allowed, otherwise operation will be rejected by throwing exception.
-3. Partition management command: `ALTER TABLE ADD/DROP PARTITION
+3. Partition management command: `ALTER TABLE ADD/DROP PARTITION`.
If user do want to perform above operations on the main table, user can first drop the datamap, perform the operation, and re-create the datamap again.
[carbondata] 11/33: [CARBONDATA-3497] Support to write long string
for streaming table
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 99e0c7cb59fdf9d89a797b1b13923b64639dfc30
Author: Zhang Zhichao <44...@qq.com>
AuthorDate: Tue Aug 27 11:32:48 2019 +0800
[CARBONDATA-3497] Support to write long string for streaming table
This closes #3366
---
.../hadoop/stream/StreamRecordReader.java | 19 +-
.../resources/streamSample_with_long_string.csv | 6 +
.../streaming/CarbonAppendableStreamSink.scala | 19 +-
.../converter/SparkDataTypeConverterImpl.java | 6 +-
.../TestStreamingTableWithLongString.scala | 649 +++++++++++++++++++++
.../streaming/CarbonStreamRecordWriter.java | 11 +-
.../streaming/parser/CSVStreamParserImp.java | 5 +-
.../streaming/parser/CarbonStreamParser.java | 3 +-
.../streaming/parser/RowStreamParserImp.scala | 11 +-
9 files changed, 715 insertions(+), 14 deletions(-)
diff --git a/hadoop/src/main/java/org/apache/carbondata/hadoop/stream/StreamRecordReader.java b/hadoop/src/main/java/org/apache/carbondata/hadoop/stream/StreamRecordReader.java
index 75e36be..1e40baa 100644
--- a/hadoop/src/main/java/org/apache/carbondata/hadoop/stream/StreamRecordReader.java
+++ b/hadoop/src/main/java/org/apache/carbondata/hadoop/stream/StreamRecordReader.java
@@ -81,6 +81,7 @@ public class StreamRecordReader extends RecordReader<Void, Object> {
protected CarbonTable carbonTable;
private CarbonColumn[] storageColumns;
private boolean[] isRequired;
+ private boolean[] dimensionsIsVarcharTypeMap;
private DataType[] measureDataTypes;
private int dimensionCount;
private int measureCount;
@@ -163,6 +164,10 @@ public class StreamRecordReader extends RecordReader<Void, Object> {
.getDirectDictionaryGenerator(storageColumns[i].getDataType());
}
}
+ dimensionsIsVarcharTypeMap = new boolean[dimensionCount];
+ for (int i = 0; i < dimensionCount; i++) {
+ dimensionsIsVarcharTypeMap[i] = storageColumns[i].getDataType() == DataTypes.VARCHAR;
+ }
measureDataTypes = new DataType[measureCount];
for (int i = 0; i < measureCount; i++) {
measureDataTypes[i] = storageColumns[dimensionCount + i].getDataType();
@@ -387,7 +392,12 @@ public class StreamRecordReader extends RecordReader<Void, Object> {
}
} else {
if (isNoDictColumn[colCount]) {
- int v = input.readShort();
+ int v = 0;
+ if (dimensionsIsVarcharTypeMap[colCount]) {
+ v = input.readInt();
+ } else {
+ v = input.readShort();
+ }
if (isRequired[colCount]) {
byte[] b = input.readBytes(v);
if (isFilterRequired[colCount]) {
@@ -561,7 +571,12 @@ public class StreamRecordReader extends RecordReader<Void, Object> {
outputValues[colCount] = CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY;
} else {
if (isNoDictColumn[colCount]) {
- int v = input.readShort();
+ int v = 0;
+ if (dimensionsIsVarcharTypeMap[colCount]) {
+ v = input.readInt();
+ } else {
+ v = input.readShort();
+ }
outputValues[colCount] = input.readBytes(v);
} else {
outputValues[colCount] = input.readInt();
diff --git a/integration/spark-common-test/src/test/resources/streamSample_with_long_string.csv b/integration/spark-common-test/src/test/resources/streamSample_with_long_string.csv
new file mode 100644
index 0000000..b010c07
--- /dev/null
+++ b/integration/spark-common-test/src/test/resources/streamSample_with_long_string.csv
@@ -0,0 +1,6 @@
+id,name,city,salary,tax,percent,birthday,register,updated,longstr,file
+100000001,batch_1,city_1,0.1,0.01,80.01,1990-01-01,2010-01-01 10:01:01,2010-01-01 10:01:01,1abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabca [...]
+100000002,batch_2,city_2,0.2,0.02,80.02,1990-01-02,2010-01-02 10:01:01,2010-01-02 10:01:01,2abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabca [...]
+100000003,batch_3,city_3,0.3,0.03,80.03,1990-01-03,2010-01-03 10:01:01,2010-01-03 10:01:01,3abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabca [...]
+100000004,batch_4,city_4,0.4,0.04,80.04,1990-01-04,2010-01-04 10:01:01,2010-01-04 10:01:01,4abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabca [...]
+100000005,batch_5,city_5,0.5,0.05,80.05,1990-01-05,2010-01-05 10:01:01,2010-01-05 10:01:01,5abcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabcabca [...]
diff --git a/integration/spark-common/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala b/integration/spark-common/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala
index 90132ff..4440e3a 100644
--- a/integration/spark-common/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala
+++ b/integration/spark-common/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala
@@ -40,6 +40,7 @@ import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.datastore.impl.FileFactory
import org.apache.carbondata.core.dictionary.server.DictionaryServer
import org.apache.carbondata.core.metadata.datatype.DataType
+import org.apache.carbondata.core.metadata.datatype.DataTypes
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
import org.apache.carbondata.core.stats.QueryStatistic
import org.apache.carbondata.core.util.CarbonProperties
@@ -261,6 +262,15 @@ object CarbonAppendableStreamSink {
}
val rowSchema = queryExecution.analyzed.schema
+ val isVarcharTypeMapping = {
+ val col2VarcharType = carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
+ .getCreateOrderColumn(carbonLoadModel.getTableName).asScala
+ .map(c => c.getColName -> (c.getDataType == DataTypes.VARCHAR)).toMap
+ rowSchema.fieldNames.map(c => {
+ val r = col2VarcharType.get(c.toLowerCase)
+ r.isDefined && r.get
+ })
+ }
// write data file
result = sparkSession.sparkContext.runJob(queryExecution.toRdd,
(taskContext: TaskContext, iterator: Iterator[InternalRow]) => {
@@ -272,7 +282,8 @@ object CarbonAppendableStreamSink {
sparkAttemptNumber = taskContext.attemptNumber(),
committer,
iterator,
- rowSchema
+ rowSchema,
+ isVarcharTypeMapping
)
})
@@ -319,7 +330,8 @@ object CarbonAppendableStreamSink {
sparkAttemptNumber: Int,
committer: FileCommitProtocol,
iterator: Iterator[InternalRow],
- rowSchema: StructType): (TaskCommitMessage, StreamFileIndex) = {
+ rowSchema: StructType,
+ isVarcharTypeMapping: Array[Boolean]): (TaskCommitMessage, StreamFileIndex) = {
val jobId = CarbonInputFormatUtil.getJobId(new Date, sparkStageId)
val taskId = new TaskID(jobId, TaskType.MAP, sparkPartitionId)
@@ -350,7 +362,8 @@ object CarbonAppendableStreamSink {
val streamParser =
Class.forName(parserName).newInstance.asInstanceOf[CarbonStreamParser]
- streamParser.initialize(taskAttemptContext.getConfiguration, rowSchema)
+ streamParser.initialize(taskAttemptContext.getConfiguration,
+ rowSchema, isVarcharTypeMapping)
blockIndex = StreamSegment.appendBatchData(new InputIterator(iterator, streamParser),
taskAttemptContext, carbonLoadModel)
diff --git a/integration/spark-datasource/src/main/scala/org/apache/carbondata/converter/SparkDataTypeConverterImpl.java b/integration/spark-datasource/src/main/scala/org/apache/carbondata/converter/SparkDataTypeConverterImpl.java
index 41b378d..4db1154 100644
--- a/integration/spark-datasource/src/main/scala/org/apache/carbondata/converter/SparkDataTypeConverterImpl.java
+++ b/integration/spark-datasource/src/main/scala/org/apache/carbondata/converter/SparkDataTypeConverterImpl.java
@@ -114,7 +114,8 @@ public final class SparkDataTypeConverterImpl implements DataTypeConverter, Seri
private static org.apache.spark.sql.types.DataType convertCarbonToSparkDataType(
DataType carbonDataType) {
- if (carbonDataType == org.apache.carbondata.core.metadata.datatype.DataTypes.STRING) {
+ if (carbonDataType == org.apache.carbondata.core.metadata.datatype.DataTypes.STRING
+ || carbonDataType == org.apache.carbondata.core.metadata.datatype.DataTypes.VARCHAR) {
return DataTypes.StringType;
} else if (carbonDataType == org.apache.carbondata.core.metadata.datatype.DataTypes.SHORT) {
return DataTypes.ShortType;
@@ -170,7 +171,8 @@ public final class SparkDataTypeConverterImpl implements DataTypeConverter, Seri
|| dataType == org.apache.carbondata.core.metadata.datatype.DataTypes.SHORT
|| dataType == org.apache.carbondata.core.metadata.datatype.DataTypes.INT
|| dataType == org.apache.carbondata.core.metadata.datatype.DataTypes.LONG
- || dataType == org.apache.carbondata.core.metadata.datatype.DataTypes.BINARY) {
+ || dataType == org.apache.carbondata.core.metadata.datatype.DataTypes.BINARY
+ || dataType == org.apache.carbondata.core.metadata.datatype.DataTypes.VARCHAR) {
fields[i] = new StructField(carbonColumn.getColName(),
convertCarbonToSparkDataType(dataType), true, null);
} else if (org.apache.carbondata.core.metadata.datatype.DataTypes.isDecimal(dataType)) {
diff --git a/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithLongString.scala b/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithLongString.scala
new file mode 100644
index 0000000..521b241
--- /dev/null
+++ b/integration/spark2/src/test/scala/org/apache/spark/carbondata/TestStreamingTableWithLongString.scala
@@ -0,0 +1,649 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.carbondata
+
+import java.io.{File, PrintWriter}
+import java.math.BigDecimal
+import java.net.{BindException, ServerSocket}
+import java.sql.{Date, Timestamp}
+
+import scala.collection.mutable
+
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.hive.CarbonRelation
+import org.apache.spark.sql.streaming.{ProcessingTime, StreamingQuery}
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.streaming.parser.CarbonStreamParser
+
+case class StreamLongStrData(id: Integer, name: String, city: String, salary: java.lang.Float,
+ tax: BigDecimal, percent: java.lang.Double, birthday: String,
+ register: String, updated: String, longStr: String,
+ file: FileElement)
+
+class TestStreamingTableWithLongString extends QueryTest with BeforeAndAfterAll {
+
+ private val spark = sqlContext.sparkSession
+ private val dataFilePath = s"$resourcesPath/streamSample_with_long_string.csv"
+ private val csvDataDir = integrationPath + "/spark2/target/csvdata_longstr"
+ private val longStrValue = "abc" * 12000
+
+ override def beforeAll {
+ CarbonProperties.getInstance().addProperty(
+ CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
+ CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+ CarbonProperties.getInstance().addProperty(
+ CarbonCommonConstants.CARBON_DATE_FORMAT,
+ CarbonCommonConstants.CARBON_DATE_DEFAULT_FORMAT)
+ sql("DROP DATABASE IF EXISTS streaming_longstr CASCADE")
+ sql("CREATE DATABASE streaming_longstr")
+ sql("USE streaming_longstr")
+
+ dropTable()
+
+ // 1. streaming table with long string field
+ // socket source
+ createTable(tableName = "stream_table_longstr", streaming = true, withBatchLoad = true)
+
+ // 2. streaming table with long string field
+ // file source
+ createTable(tableName = "stream_table_longstr_file", streaming = true, withBatchLoad = true)
+
+ // 3. streaming table with long string and complex field
+ createTableWithComplexType(
+ tableName = "stream_table_longstr_complex", streaming = true, withBatchLoad = true)
+ }
+
+ override def afterAll {
+ dropTable()
+ sql("USE default")
+ sql("DROP DATABASE IF EXISTS streaming_longstr CASCADE")
+ new File(csvDataDir).delete()
+ }
+
+ def dropTable(): Unit = {
+ sql("drop table if exists streaming_longstr.stream_table_longstr")
+ sql("drop table if exists streaming_longstr.stream_table_longstr_file")
+ sql("drop table if exists streaming_longstr.stream_table_longstr_complex")
+ }
+
+ // input source: file
+ test("[CARBONDATA-3497] Support to write long string for streaming table: ingest from file source") {
+ val identifier = new TableIdentifier("stream_table_longstr_file", Option("streaming_longstr"))
+ val carbonTable = CarbonEnv.getInstance(spark).carbonMetaStore.lookupRelation(identifier)(spark)
+ .asInstanceOf[CarbonRelation].metaData.carbonTable
+ // streaming ingest 10 rows
+ generateCSVDataFile(spark, idStart = 10, rowNums = 10, csvDataDir)
+ val thread = createFileStreamingThread(spark, carbonTable, csvDataDir, intervalSecond = 1,
+ identifier)
+ thread.start()
+ Thread.sleep(3000)
+ generateCSVDataFile(spark, idStart = 30, rowNums = 10, csvDataDir)
+ Thread.sleep(5000)
+ thread.interrupt()
+ checkAnswer(
+ sql("select count(*) from streaming_longstr.stream_table_longstr_file"),
+ Seq(Row(25))
+ )
+
+ val row = sql("select * from streaming_longstr.stream_table_longstr_file order by id").head()
+ val exceptedRow = Row(10, "name_10", "city_10", 100000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), "10" + longStrValue)
+ assertResult(exceptedRow)(row)
+ new File(csvDataDir).delete()
+ }
+
+ test("[CARBONDATA-3497] Support to write long string for streaming table") {
+ executeStreamingIngest(
+ tableName = "stream_table_longstr",
+ batchNums = 2,
+ rowNumsEachBatch = 25,
+ intervalOfSource = 5,
+ intervalOfIngest = 5,
+ continueSeconds = 20,
+ handoffSize = 51200,
+ autoHandoff = false
+ )
+
+ var result = sql("select * from streaming_longstr.stream_table_longstr order by id, name").collect()
+ assert(result != null)
+ assert(result.length == 55)
+ // check one row of streaming data
+ assert(result(1).getString(1) == "name_2")
+ assert(result(1).getString(9) == ("2" + longStrValue))
+ // check one row of batch loading
+ assert(result(50).getInt(0) == 100000001)
+ assert(result(50).getString(1) == "batch_1")
+ assert(result(50).getString(9) == ("1" + longStrValue))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id = 1"),
+ Seq(Row(1, "name_1", "city_1", 10000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id > 49 and id < 100000002"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id between 50 and 100000001"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ sql("show segments for table streaming_longstr.stream_table_longstr").show(20, false)
+ sql("alter table streaming_longstr.stream_table_longstr finish streaming")
+ sql("alter table streaming_longstr.stream_table_longstr compact 'streaming'")
+ sql("show segments for table streaming_longstr.stream_table_longstr").show(20, false)
+ Thread.sleep(5000)
+
+ result = sql("select * from streaming_longstr.stream_table_longstr order by id, name").collect()
+ assert(result != null)
+ assert(result.length == 55)
+ // check one row of streaming data
+ assert(result(2).getString(1) == "name_3")
+ assert(result(2).getString(9) == ("3" + longStrValue))
+ // check one row of batch loading
+ assert(result(51).getInt(0) == 100000002)
+ assert(result(51).getString(1) == "batch_2")
+ assert(result(51).getString(9) == ("2" + longStrValue))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id = 1"),
+ Seq(Row(1, "name_1", "city_1", 10000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id > 49 and id < 100000002"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id between 50 and 100000001"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ sql("alter table streaming_longstr.stream_table_longstr compact 'major'")
+ sql("show segments for table streaming_longstr.stream_table_longstr").show(20, false)
+ Thread.sleep(5000)
+
+ result = sql("select * from streaming_longstr.stream_table_longstr order by id, name").collect()
+ assert(result != null)
+ assert(result.length == 55)
+ // check one row of streaming data
+ assert(result(3).getString(1) == "name_4")
+ assert(result(3).getString(9) == ("4" + longStrValue))
+ // check one row of batch loading
+ assert(result(52).getInt(0) == 100000003)
+ assert(result(52).getString(1) == "batch_3")
+ assert(result(52).getString(9) == ("3" + longStrValue))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id = 1"),
+ Seq(Row(1, "name_1", "city_1", 10000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id > 49 and id < 100000002"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr where id between 50 and 100000001"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue))))
+ }
+
+ test("[CARBONDATA-3497] Support to write long string for streaming table: include complex column") {
+ executeStreamingIngest(
+ tableName = "stream_table_longstr_complex",
+ batchNums = 2,
+ rowNumsEachBatch = 25,
+ intervalOfSource = 5,
+ intervalOfIngest = 5,
+ continueSeconds = 20,
+ handoffSize = 51200,
+ autoHandoff = false
+ )
+
+ // non-filter
+ val result = sql("select * from streaming_longstr.stream_table_longstr_complex order by id, name").collect()
+ assert(result != null)
+ assert(result.length == 55)
+ // check one row of streaming data
+ assert(result(3).getString(1) == "name_4")
+ assert(result(3).getString(9) == ("4" + longStrValue))
+ // check one row of batch loading
+ assert(result(52).getInt(0) == 100000003)
+ assert(result(52).getString(1) == "batch_3")
+ assert(result(52).getString(9) == ("3" + longStrValue))
+ assert(result(52).getStruct(10).getInt(1) == 40)
+
+ // filter
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr_complex where id = 1"),
+ Seq(Row(1, "name_1", "city_1", 10000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue), Row(wrap(Array("school_1", "school_11")), 1))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr_complex where id > 49 and id < 100000002"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue), Row(wrap(Array("school_50", "school_5050")), 50)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue), Row(wrap(Array("school_1", "school_11")), 20))))
+
+ checkAnswer(
+ sql("select * from streaming_longstr.stream_table_longstr_complex where id between 50 and 100000001"),
+ Seq(Row(50, "name_50", "city_50", 500000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("50" + longStrValue), Row(wrap(Array("school_50", "school_5050")), 50)),
+ Row(100000001, "batch_1", "city_1", 0.1, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("1" + longStrValue), Row(wrap(Array("school_1", "school_11")), 20))))
+ }
+
+ test("[CARBONDATA-3497] Support to write long string for streaming table: StreamSQL") {
+ sql("DROP TABLE IF EXISTS source")
+ sql("DROP TABLE IF EXISTS sink")
+
+ var rows = sql("SHOW STREAMS").collect()
+ assertResult(0)(rows.length)
+
+ val csvDataDir = integrationPath + "/spark2/target/streamSql_longstr"
+ // streaming ingest 10 rows
+ generateCSVDataFile(spark, idStart = 10, rowNums = 10, csvDataDir)
+
+ sql(
+ s"""
+ |CREATE TABLE source(
+ | id INT,
+ | name STRING,
+ | city STRING,
+ | salary FLOAT,
+ | tax DECIMAL(8,2),
+ | percent double,
+ | birthday DATE,
+ | register TIMESTAMP,
+ | updated TIMESTAMP,
+ | longstr STRING
+ |)
+ |STORED AS carbondata
+ |TBLPROPERTIES (
+ | 'streaming'='source',
+ | 'format'='csv',
+ | 'path'='$csvDataDir'
+ |)
+ """.stripMargin)
+
+ sql(
+ s"""
+ |CREATE TABLE sink(
+ | id INT,
+ | name STRING,
+ | city STRING,
+ | salary FLOAT,
+ | tax DECIMAL(8,2),
+ | percent double,
+ | birthday DATE,
+ | register TIMESTAMP,
+ | updated TIMESTAMP,
+ | longstr STRING
+ | )
+ |STORED AS carbondata
+ |TBLPROPERTIES('streaming'='sink', 'LONG_STRING_COLUMNS'='longstr')
+ """.stripMargin)
+
+ sql(
+ """
+ |CREATE STREAM stream123 ON TABLE sink
+ |STMPROPERTIES(
+ | 'trigger'='ProcessingTime',
+ | 'interval'='5 seconds')
+ |AS
+ | SELECT *
+ | FROM source
+ | WHERE id % 2 = 1
+ """.stripMargin).show(false)
+
+ Thread.sleep(200)
+ sql("select * from sink").show
+
+ generateCSVDataFile(spark, idStart = 30, rowNums = 10, csvDataDir, SaveMode.Append)
+ Thread.sleep(7000)
+
+ // after 2 minibatch, there should be 10 row added (filter condition: id%2=1)
+ checkAnswer(sql("select count(*) from sink"), Seq(Row(10)))
+
+ val row = sql("select * from sink order by id").head()
+ val exceptedRow = Row(11, "name_11", "city_11", 110000.0, BigDecimal.valueOf(0.01), 80.01, Date.valueOf("1990-01-01"), Timestamp.valueOf("2010-01-01 10:01:01.0"), Timestamp.valueOf("2010-01-01 10:01:01.0"), ("11" + longStrValue))
+ assertResult(exceptedRow)(row)
+
+ sql("SHOW STREAMS").show(false)
+
+ rows = sql("SHOW STREAMS").collect()
+ assertResult(1)(rows.length)
+ assertResult("stream123")(rows.head.getString(0))
+ assertResult("RUNNING")(rows.head.getString(2))
+ assertResult("streaming_longstr.source")(rows.head.getString(3))
+ assertResult("streaming_longstr.sink")(rows.head.getString(4))
+
+ rows = sql("SHOW STREAMS ON TABLE sink").collect()
+ assertResult(1)(rows.length)
+ assertResult("stream123")(rows.head.getString(0))
+ assertResult("RUNNING")(rows.head.getString(2))
+ assertResult("streaming_longstr.source")(rows.head.getString(3))
+ assertResult("streaming_longstr.sink")(rows.head.getString(4))
+
+ sql("DROP STREAM stream123")
+ sql("DROP STREAM IF EXISTS stream123")
+
+ rows = sql("SHOW STREAMS").collect()
+ assertResult(0)(rows.length)
+
+ sql("DROP TABLE IF EXISTS source")
+ sql("DROP TABLE IF EXISTS sink")
+ new File(csvDataDir).delete()
+ }
+
+ def createWriteSocketThread(
+ serverSocket: ServerSocket,
+ writeNums: Int,
+ rowNums: Int,
+ intervalSecond: Int): Thread = {
+ new Thread() {
+ override def run(): Unit = {
+ // wait for client to connection request and accept
+ val clientSocket = serverSocket.accept()
+ val socketWriter = new PrintWriter(clientSocket.getOutputStream())
+ var index = 0
+ for (_ <- 1 to writeNums) {
+ // write 5 records per iteration
+ val stringBuilder = new StringBuilder()
+ for (_ <- 1 to rowNums) {
+ index = index + 1
+ stringBuilder.append(index.toString + ",name_" + index
+ + ",city_" + index + "," + (10000.00 * index).toString + ",0.01,80.01" +
+ ",1990-01-01,2010-01-01 10:01:01,2010-01-01 10:01:01," +
+ index.toString() + ("abc" * 12000) +
+ ",school_" + index + ":school_" + index + index + "$" + index)
+ stringBuilder.append("\n")
+ }
+ socketWriter.append(stringBuilder.toString())
+ socketWriter.flush()
+ Thread.sleep(1000 * intervalSecond)
+ }
+ socketWriter.close()
+ }
+ }
+ }
+
+ def createSocketStreamingThread(
+ spark: SparkSession,
+ port: Int,
+ carbonTable: CarbonTable,
+ tableIdentifier: TableIdentifier,
+ intervalSecond: Int = 2,
+ handoffSize: Long = CarbonCommonConstants.HANDOFF_SIZE_DEFAULT,
+ autoHandoff: Boolean = CarbonCommonConstants.ENABLE_AUTO_HANDOFF_DEFAULT.toBoolean
+ ): Thread = {
+ new Thread() {
+ override def run(): Unit = {
+ var qry: StreamingQuery = null
+ try {
+ import spark.implicits._
+ val readSocketDF = spark.readStream
+ .format("socket")
+ .option("host", "localhost")
+ .option("port", port)
+ .load().as[String]
+ .map(_.split(","))
+ .map { fields => {
+ val tmp = fields(10).split("\\$")
+ val file = FileElement(tmp(0).split(":"), tmp(1).toInt)
+ StreamLongStrData(fields(0).toInt, fields(1), fields(2), fields(3).toFloat,
+ BigDecimal.valueOf(fields(4).toDouble), fields(5).toDouble,
+ fields(6), fields(7), fields(8), fields(9), file)
+ } }
+
+ // Write data from socket stream to carbondata file
+ // repartition to simulate an empty partition when readSocketDF has only one row
+ qry = readSocketDF.repartition(2).writeStream
+ .format("carbondata")
+ .trigger(ProcessingTime(s"$intervalSecond seconds"))
+ .option("checkpointLocation", CarbonTablePath.getStreamingCheckpointDir(carbonTable.getTablePath))
+ .option("dbName", tableIdentifier.database.get)
+ .option("tableName", tableIdentifier.table)
+ .option(CarbonCommonConstants.HANDOFF_SIZE, handoffSize)
+ .option("timestampformat", CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+ .option(CarbonCommonConstants.ENABLE_AUTO_HANDOFF, autoHandoff)
+ .start()
+ qry.awaitTermination()
+ } catch {
+ case ex: Throwable =>
+ LOGGER.error(ex.getMessage)
+ throw new Exception(ex.getMessage, ex)
+ } finally {
+ if (null != qry) {
+ qry.stop()
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * start ingestion thread: write `rowNumsEachBatch` rows repeatly for `batchNums` times.
+ */
+ def executeStreamingIngest(
+ tableName: String,
+ batchNums: Int,
+ rowNumsEachBatch: Int,
+ intervalOfSource: Int,
+ intervalOfIngest: Int,
+ continueSeconds: Int,
+ handoffSize: Long = CarbonCommonConstants.HANDOFF_SIZE_DEFAULT,
+ autoHandoff: Boolean = CarbonCommonConstants.ENABLE_AUTO_HANDOFF_DEFAULT.toBoolean
+ ): Unit = {
+ val identifier = new TableIdentifier(tableName, Option("streaming_longstr"))
+ val carbonTable = CarbonEnv.getInstance(spark).carbonMetaStore.lookupRelation(identifier)(spark)
+ .asInstanceOf[CarbonRelation].metaData.carbonTable
+ var server: ServerSocket = null
+ try {
+ server = getServerSocket()
+ val thread1 = createWriteSocketThread(
+ serverSocket = server,
+ writeNums = batchNums,
+ rowNums = rowNumsEachBatch,
+ intervalSecond = intervalOfSource)
+ val thread2 = createSocketStreamingThread(
+ spark = spark,
+ port = server.getLocalPort,
+ carbonTable = carbonTable,
+ tableIdentifier = identifier,
+ intervalSecond = intervalOfIngest,
+ handoffSize = handoffSize,
+ autoHandoff = autoHandoff)
+ thread1.start()
+ thread2.start()
+ Thread.sleep(continueSeconds * 1000)
+ thread2.interrupt()
+ thread1.interrupt()
+ } finally {
+ if (null != server) {
+ server.close()
+ }
+ }
+ }
+
+ def generateCSVDataFile(
+ spark: SparkSession,
+ idStart: Int,
+ rowNums: Int,
+ csvDirPath: String,
+ saveMode: SaveMode = SaveMode.Overwrite): Unit = {
+ // Create csv data frame file
+ val csvDataDF = {
+ // generate data with dimension columns (name and city)
+ val csvRDD = spark.sparkContext.parallelize(idStart until idStart + rowNums)
+ .map { id =>
+ (id,
+ "name_" + id,
+ "city_" + id,
+ 10000.00 * id,
+ BigDecimal.valueOf(0.01),
+ 80.01,
+ "1990-01-01",
+ "2010-01-01 10:01:01",
+ "2010-01-01 10:01:01",
+ id.toString() + ("abc" * 12000),
+ "school_" + id + "\002school_" + id + id + "\001" + id)
+ }
+ spark.createDataFrame(csvRDD).toDF(
+ "id", "name", "city", "salary", "tax", "percent", "birthday", "register", "updated", "longstr", "file")
+ }
+
+ csvDataDF.write
+ .option("header", "false")
+ .mode(saveMode)
+ .csv(csvDirPath)
+ }
+
+ def createFileStreamingThread(
+ spark: SparkSession,
+ carbonTable: CarbonTable,
+ csvDataDir: String,
+ intervalSecond: Int,
+ tableIdentifier: TableIdentifier): Thread = {
+ new Thread() {
+ override def run(): Unit = {
+ var qry: StreamingQuery = null
+ try {
+ val readSocketDF = spark.readStream.text(csvDataDir)
+
+ // Write data from socket stream to carbondata file
+ qry = readSocketDF.writeStream
+ .format("carbondata")
+ .trigger(ProcessingTime(s"${ intervalSecond } seconds"))
+ .option("checkpointLocation", CarbonTablePath.getStreamingCheckpointDir(carbonTable.getTablePath))
+ .option("dbName", tableIdentifier.database.get)
+ .option("tableName", tableIdentifier.table)
+ .option("timestampformat", CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT)
+ .option(CarbonStreamParser.CARBON_STREAM_PARSER,
+ CarbonStreamParser.CARBON_STREAM_PARSER_CSV)
+ .start()
+
+ qry.awaitTermination()
+ } catch {
+ case _: InterruptedException =>
+ println("Done reading and writing streaming data")
+ } finally {
+ if (qry != null) {
+ qry.stop()
+ }
+ }
+ }
+ }
+ }
+
+ def createTable(tableName: String, streaming: Boolean, withBatchLoad: Boolean): Unit = {
+ sql(
+ s"""
+ | CREATE TABLE streaming_longstr.$tableName(
+ | id INT,
+ | name STRING,
+ | city STRING,
+ | salary FLOAT,
+ | tax DECIMAL(8,2),
+ | percent double,
+ | birthday DATE,
+ | register TIMESTAMP,
+ | updated TIMESTAMP,
+ | longstr STRING
+ | )
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES(${if (streaming) "'streaming'='true', " else "" }
+ | 'sort_columns'='name', 'dictionary_include'='city,register', 'LONG_STRING_COLUMNS'='longstr')
+ | """.stripMargin)
+
+ if (withBatchLoad) {
+ // batch loading 5 rows
+ executeBatchLoad(tableName)
+ }
+ }
+
+ def createTableWithComplexType(
+ tableName: String,
+ streaming: Boolean,
+ withBatchLoad: Boolean): Unit = {
+ sql(
+ s"""
+ | CREATE TABLE streaming_longstr.$tableName(
+ | id INT,
+ | name STRING,
+ | city STRING,
+ | salary FLOAT,
+ | tax DECIMAL(8,2),
+ | percent double,
+ | birthday DATE,
+ | register TIMESTAMP,
+ | updated TIMESTAMP,
+ | longstr STRING,
+ | file struct<school:array<string>, age:int>
+ | )
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES(${if (streaming) "'streaming'='true', " else "" }
+ | 'sort_columns'='name', 'dictionary_include'='id,name,salary,tax,percent,updated', 'LONG_STRING_COLUMNS'='longstr')
+ | """.stripMargin)
+
+ if (withBatchLoad) {
+ // batch loading 5 rows
+ executeBatchLoad(tableName)
+ }
+ }
+
+ def executeBatchLoad(tableName: String): Unit = {
+ sql(
+ s"LOAD DATA LOCAL INPATH '$dataFilePath' INTO TABLE streaming_longstr.$tableName OPTIONS" +
+ "('HEADER'='true','COMPLEX_DELIMITER_LEVEL_1'='$', 'COMPLEX_DELIMITER_LEVEL_2'=':')")
+ }
+
+ def wrap(array: Array[String]) = {
+ new mutable.WrappedArray.ofRef(array)
+ }
+
+ /**
+ * get a ServerSocket
+ * if the address was already used, it will retry to use new port number.
+ *
+ * @return ServerSocket
+ */
+ def getServerSocket(): ServerSocket = {
+ var port = 7071
+ var serverSocket: ServerSocket = null
+ var retry = false
+ do {
+ try {
+ retry = false
+ serverSocket = new ServerSocket(port)
+ } catch {
+ case ex: BindException =>
+ retry = true
+ port = port + 2
+ if (port >= 65535) {
+ throw ex
+ }
+ }
+ } while (retry)
+ serverSocket
+ }
+}
diff --git a/streaming/src/main/java/org/apache/carbondata/streaming/CarbonStreamRecordWriter.java b/streaming/src/main/java/org/apache/carbondata/streaming/CarbonStreamRecordWriter.java
index 5ef5ab9..1f642e5 100644
--- a/streaming/src/main/java/org/apache/carbondata/streaming/CarbonStreamRecordWriter.java
+++ b/streaming/src/main/java/org/apache/carbondata/streaming/CarbonStreamRecordWriter.java
@@ -91,6 +91,7 @@ public class CarbonStreamRecordWriter extends RecordWriter<Void, Object> {
private boolean[] isNoDictionaryDimensionColumn;
private int dimensionWithComplexCount;
private int measureCount;
+ private boolean[] dimensionsIsVarcharTypeMap;
private DataType[] measureDataTypes;
private StreamBlockletWriter output = null;
private String compressorName;
@@ -147,6 +148,10 @@ public class CarbonStreamRecordWriter extends RecordWriter<Void, Object> {
dimensionWithComplexCount = configuration.getDimensionCount();
measureCount = configuration.getMeasureCount();
dataFields = configuration.getDataFields();
+ dimensionsIsVarcharTypeMap = new boolean[dimensionWithComplexCount];
+ for (int i = 0; i < dimensionWithComplexCount; i++) {
+ dimensionsIsVarcharTypeMap[i] = dataFields[i].getColumn().getDataType() == DataTypes.VARCHAR;
+ }
measureDataTypes = new DataType[measureCount];
for (int i = 0; i < measureCount; i++) {
measureDataTypes[i] =
@@ -234,7 +239,11 @@ public class CarbonStreamRecordWriter extends RecordWriter<Void, Object> {
if (null != columnValue) {
if (isNoDictionaryDimensionColumn[dimCount]) {
byte[] col = (byte[]) columnValue;
- output.writeShort(col.length);
+ if (dimensionsIsVarcharTypeMap[dimCount]) {
+ output.writeInt(col.length);
+ } else {
+ output.writeShort(col.length);
+ }
output.writeBytes(col);
output.dimStatsCollectors[dimCount].update(col);
} else {
diff --git a/streaming/src/main/java/org/apache/carbondata/streaming/parser/CSVStreamParserImp.java b/streaming/src/main/java/org/apache/carbondata/streaming/parser/CSVStreamParserImp.java
index 00d06b6..bf2c460 100644
--- a/streaming/src/main/java/org/apache/carbondata/streaming/parser/CSVStreamParserImp.java
+++ b/streaming/src/main/java/org/apache/carbondata/streaming/parser/CSVStreamParserImp.java
@@ -21,6 +21,7 @@ import org.apache.carbondata.processing.loading.csvinput.CSVInputFormat;
import com.univocity.parsers.csv.CsvParser;
import com.univocity.parsers.csv.CsvParserSettings;
+
import org.apache.hadoop.conf.Configuration;
import org.apache.spark.sql.catalyst.InternalRow;
import org.apache.spark.sql.types.StructType;
@@ -32,7 +33,9 @@ public class CSVStreamParserImp implements CarbonStreamParser {
private CsvParser csvParser;
- @Override public void initialize(Configuration configuration, StructType structType) {
+ @Override public void initialize(
+ Configuration configuration,
+ StructType structType, boolean[] isVarcharTypeMapping) {
CsvParserSettings settings = CSVInputFormat.extractCsvParserSettings(configuration);
csvParser = new CsvParser(settings);
}
diff --git a/streaming/src/main/java/org/apache/carbondata/streaming/parser/CarbonStreamParser.java b/streaming/src/main/java/org/apache/carbondata/streaming/parser/CarbonStreamParser.java
index 94f0307..e68117c 100644
--- a/streaming/src/main/java/org/apache/carbondata/streaming/parser/CarbonStreamParser.java
+++ b/streaming/src/main/java/org/apache/carbondata/streaming/parser/CarbonStreamParser.java
@@ -36,7 +36,8 @@ public interface CarbonStreamParser {
String CARBON_STREAM_PARSER_DEFAULT = CARBON_STREAM_PARSER_ROW_PARSER;
- void initialize(Configuration configuration, StructType structType);
+ void initialize(Configuration configuration,
+ StructType structType, boolean[] isVarcharTypeMapping);
Object[] parserRow(InternalRow value);
diff --git a/streaming/src/main/scala/org/apache/carbondata/streaming/parser/RowStreamParserImp.scala b/streaming/src/main/scala/org/apache/carbondata/streaming/parser/RowStreamParserImp.scala
index cb12bb6..16e7258 100644
--- a/streaming/src/main/scala/org/apache/carbondata/streaming/parser/RowStreamParserImp.scala
+++ b/streaming/src/main/scala/org/apache/carbondata/streaming/parser/RowStreamParserImp.scala
@@ -36,6 +36,7 @@ import org.apache.carbondata.processing.loading.constants.DataLoadProcessorConst
class RowStreamParserImp extends CarbonStreamParser {
var configuration: Configuration = null
+ var isVarcharTypeMapping: Array[Boolean] = null
var structType: StructType = null
var encoder: ExpressionEncoder[Row] = null
@@ -44,10 +45,12 @@ class RowStreamParserImp extends CarbonStreamParser {
var complexDelimiters: util.ArrayList[String] = new util.ArrayList[String]()
var serializationNullFormat: String = null
- override def initialize(configuration: Configuration, structType: StructType): Unit = {
+ override def initialize(configuration: Configuration,
+ structType: StructType, isVarcharTypeMapping: Array[Boolean]): Unit = {
this.configuration = configuration
this.structType = structType
this.encoder = RowEncoder.apply(this.structType).resolveAndBind()
+ this.isVarcharTypeMapping = isVarcharTypeMapping
this.timeStampFormat = new SimpleDateFormat(
this.configuration.get(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT))
@@ -62,12 +65,12 @@ class RowStreamParserImp extends CarbonStreamParser {
}
override def parserRow(value: InternalRow): Array[Object] = {
- this.encoder.fromRow(value).toSeq.map { x => {
+ this.encoder.fromRow(value).toSeq.zipWithIndex.map { case (x, i) =>
FieldConverter.objectToString(
x, serializationNullFormat, complexDelimiters,
- timeStampFormat, dateFormat)
+ timeStampFormat, dateFormat,
+ isVarcharType = i < this.isVarcharTypeMapping.length && this.isVarcharTypeMapping(i))
} }.toArray
- }
override def close(): Unit = {
}
[carbondata] 15/33: [CARBONDATA-3507] Fix Create Table As Select
Failure in Spark-2.3
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit d509cd19e3a9249f18c6b8b0ab2bbe19df017e65
Author: manishnalla1994 <ma...@gmail.com>
AuthorDate: Thu Aug 29 12:00:11 2019 +0530
[CARBONDATA-3507] Fix Create Table As Select Failure in Spark-2.3
Problem: Create table as select fails with Spark-2.3.
Cause: When creating the table location path the function
removes the "hdfs://" part from the path and then stores it,
due to which in later stages the file is treated as a Local Carbon File.
Solution: Get the original table path without removing the prefix.
This closes #3368
---
.../main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
index b19b11c..684bcbb 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
@@ -567,9 +567,7 @@ class CarbonFileMetastore extends CarbonMetaStore {
}
val tableLocation = catalogTable.storage.locationUri match {
case tableLoc@Some(uri) =>
- if (tableLoc.get.isInstanceOf[URI]) {
- FileFactory.getUpdatedFilePath(tableLoc.get.asInstanceOf[URI].getPath)
- }
+ FileFactory.getUpdatedFilePath(tableLoc.get.toString)
case None =>
CarbonEnv.getTablePath(tableIdentifier.database, tableIdentifier.table)(sparkSession)
}
[carbondata] 12/33: [CARBONDATA-3513] fix 'taskNo' exceeding
Long.MAX_VALUE issue when execute major compaction
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 41ae280c16905687c7ea08b3cd05acef9e359c26
Author: changchun wang <ch...@tendcloud.com>
AuthorDate: Thu Sep 5 16:28:41 2019 +0800
[CARBONDATA-3513] fix 'taskNo' exceeding Long.MAX_VALUE issue when execute major compaction
Probelm:
Major compaction command runs error.
java.lang.NumberFormatException is thrown.java.lang.NumberFormatException: For input string: "32881200100001100000"
Through code analysis it was found that taskno is "long" type. taskno generate algorithm may generate a number bigger than "Long.MAX_VALUE". carbondata-3325 change taskno type to string. But in some places it still using long.
Solution:
Change taskno type to string.
This closes #3376
---
.../apache/carbondata/processing/merger/AbstractResultProcessor.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/processing/src/main/java/org/apache/carbondata/processing/merger/AbstractResultProcessor.java b/processing/src/main/java/org/apache/carbondata/processing/merger/AbstractResultProcessor.java
index f557e9b..951339a 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/merger/AbstractResultProcessor.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/merger/AbstractResultProcessor.java
@@ -61,7 +61,7 @@ public abstract class AbstractResultProcessor {
carbonDataFileAttributes = new CarbonDataFileAttributes(index, loadModel.getFactTimeStamp());
} else {
carbonDataFileAttributes =
- new CarbonDataFileAttributes(Long.parseLong(loadModel.getTaskNo()),
+ new CarbonDataFileAttributes(loadModel.getTaskNo(),
loadModel.getFactTimeStamp());
}
carbonFactDataHandlerModel.setCarbonDataFileAttributes(carbonDataFileAttributes);
[carbondata] 27/33: [CARBONDATA-3473] Fix data size calcution of
the last column in CarbonCli
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 57309d70d08675c31975d2a60692835e7a6c22cf
Author: Manhua <ke...@qq.com>
AuthorDate: Wed Jul 17 17:39:29 2019 +0800
[CARBONDATA-3473] Fix data size calcution of the last column in CarbonCli
When update last column chunk data size, current code use columnDataSize.add(fileSizeInBytes - footerSizeInBytes - previousChunkOffset) for every blocklet. This leads to wrong result for calculting the data size of the last column, especially when a carbon data file has multiple blocklet.
In this PR, we fix this problem and modify the calcultion by remarking the end offset of blocklet.
This closes #3330
---
.../java/org/apache/carbondata/tool/DataFile.java | 32 +++++++++++-----------
.../org/apache/carbondata/tool/CarbonCliTest.java | 6 ++--
2 files changed, 19 insertions(+), 19 deletions(-)
diff --git a/tools/cli/src/main/java/org/apache/carbondata/tool/DataFile.java b/tools/cli/src/main/java/org/apache/carbondata/tool/DataFile.java
index e553a78..4ed3945 100644
--- a/tools/cli/src/main/java/org/apache/carbondata/tool/DataFile.java
+++ b/tools/cli/src/main/java/org/apache/carbondata/tool/DataFile.java
@@ -121,16 +121,21 @@ class DataFile {
this.partNo = CarbonTablePath.DataFileUtil.getPartNo(fileName);
// calculate blocklet size and column size
- // first calculate the header size, it equals the offset of first
- // column chunk in first blocklet
- long headerSizeInBytes = footer.blocklet_info_list3.get(0).column_data_chunks_offsets.get(0);
- long previousOffset = headerSizeInBytes;
- for (BlockletInfo3 blockletInfo3 : footer.blocklet_info_list3) {
+ for (int j = 0; j < footer.getBlocklet_info_list3().size(); j++) {
+ // remark start and end offset of current blocklet for computing blocklet size
+ // and chunk data size of the last column
+ BlockletInfo3 blockletInfo3 = footer.blocklet_info_list3.get(j);
+ long blockletEndOffset;
+ if (j != footer.getBlocklet_info_list3().size() - 1) {
+ // use start offset of next blocklet as end offset of current blocklet
+ blockletEndOffset = footer.blocklet_info_list3.get(j + 1).column_data_chunks_offsets.get(j);
+ } else {
+ // use start offset of footer as end offset of current blocklet if it is the last blocklet
+ blockletEndOffset = fileSizeInBytes - footerSizeInBytes;
+ }
// calculate blocklet size in bytes
- long blockletOffset = blockletInfo3.column_data_chunks_offsets.get(0);
- blockletSizeInBytes.add(blockletOffset - previousOffset);
- previousOffset = blockletOffset;
-
+ this.blockletSizeInBytes.add(
+ blockletEndOffset - blockletInfo3.column_data_chunks_offsets.get(0));
// calculate column size in bytes for each column
LinkedList<Long> columnDataSize = new LinkedList<>();
LinkedList<Long> columnMetaSize = new LinkedList<>();
@@ -140,17 +145,12 @@ class DataFile {
columnMetaSize.add(blockletInfo3.column_data_chunks_length.get(i).longValue());
previousChunkOffset = blockletInfo3.column_data_chunks_offsets.get(i);
}
- // last column chunk data size
- columnDataSize.add(fileSizeInBytes - footerSizeInBytes - previousChunkOffset);
+ // update chunk data size of the last column
+ columnDataSize.add(blockletEndOffset - previousChunkOffset);
columnDataSize.removeFirst();
this.columnDataSizeInBytes.add(columnDataSize);
this.columnMetaSizeInBytes.add(columnMetaSize);
-
}
- // last blocklet size
- blockletSizeInBytes.add(
- fileSizeInBytes - footerSizeInBytes - headerSizeInBytes - previousOffset);
- this.blockletSizeInBytes.removeFirst();
assert (blockletSizeInBytes.size() == getNumBlocklets());
}
diff --git a/tools/cli/src/test/java/org/apache/carbondata/tool/CarbonCliTest.java b/tools/cli/src/test/java/org/apache/carbondata/tool/CarbonCliTest.java
index af8d51d..4d89777 100644
--- a/tools/cli/src/test/java/org/apache/carbondata/tool/CarbonCliTest.java
+++ b/tools/cli/src/test/java/org/apache/carbondata/tool/CarbonCliTest.java
@@ -234,11 +234,11 @@ public class CarbonCliTest {
expectedOutput = buildLines(
"BLK BLKLT Meta Size Data Size LocalDict DictEntries DictSize AvgPageSize Min% Max% Min Max " ,
- "0 0 3.36KB 5.14MB false 0 0.0B 93.76KB 0.0 100.0 0 2999990 " ,
+ "0 0 3.36KB 2.57MB false 0 0.0B 93.76KB 0.0 100.0 0 2999990 " ,
"0 1 3.36KB 2.57MB false 0 0.0B 93.76KB 0.0 100.0 1 2999992 " ,
- "1 0 3.36KB 5.14MB false 0 0.0B 93.76KB 0.0 100.0 3 2999994 " ,
+ "1 0 3.36KB 2.57MB false 0 0.0B 93.76KB 0.0 100.0 3 2999994 " ,
"1 1 3.36KB 2.57MB false 0 0.0B 93.76KB 0.0 100.0 5 2999996 " ,
- "2 0 3.36KB 4.06MB false 0 0.0B 93.76KB 0.0 100.0 7 2999998 " ,
+ "2 0 3.36KB 2.57MB false 0 0.0B 93.76KB 0.0 100.0 7 2999998 " ,
"2 1 2.04KB 1.49MB false 0 0.0B 89.62KB 0.0 100.0 9 2999999 ");
Assert.assertTrue(output.contains(expectedOutput));
Assert.assertTrue(output.contains("## version Details"));
[carbondata] 03/33: [CARBONDATA-3493] Initialize Profiler in
CarbonEnv
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 90b6c648d3b20e563e685b9aa8c8bafcefbcf3ad
Author: akashrn5 <ak...@gmail.com>
AuthorDate: Wed Jul 31 18:54:41 2019 +0530
[CARBONDATA-3493] Initialize Profiler in CarbonEnv
Problem: After enabling "enable.query.statistics", exception is
thrown while querying because profiler is not initialized before
setting up the rpc end point connection.
Solution: Initialized Profiler in CarbonEnv before setting up the
rpc end point connection.
This closes #3342
Co-authored-by: shivamasn <sh...@gmail.com>
---
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala | 2 ++
.../spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala | 2 --
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
index c13e7b9..1cbd156 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
@@ -28,6 +28,7 @@ import org.apache.spark.sql.execution.command.mv._
import org.apache.spark.sql.execution.command.preaaggregate._
import org.apache.spark.sql.execution.command.timeseries.TimeSeriesFunction
import org.apache.spark.sql.hive._
+import org.apache.spark.sql.profiler.Profiler
import org.apache.carbondata.common.logging.LogServiceFactory
import org.apache.carbondata.core.constants.CarbonCommonConstants
@@ -121,6 +122,7 @@ class CarbonEnv {
initialized = true
}
}
+ Profiler.initialize(sparkSession.sparkContext)
LOGGER.info("Initialize CarbonEnv completed...")
}
}
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala
index 7b1bf4c..deefcd1 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala
@@ -259,8 +259,6 @@ object CarbonSession {
}
options.foreach { case (k, v) => session.sessionState.conf.setConfString(k, v) }
SparkSession.setDefaultSession(session)
- // Setup monitor end point and register CarbonMonitorListener
- Profiler.initialize(sparkContext)
// Register a successfully instantiated context to the singleton. This should be at the
// end of the class definition so that the singleton is updated only if there is no
// exception in the construction of the instance.
[carbondata] 33/33: [CARBONDATA-3526]Fix cache issue during update
and query
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit be9580b5768389d6adf0392fc820fb7e7186bd4c
Author: akashrn5 <ak...@gmail.com>
AuthorDate: Thu Sep 12 14:30:48 2019 +0530
[CARBONDATA-3526]Fix cache issue during update and query
Problem:
When multiple updates happen on table, cache is loaded
during update operation, but since on second update the
horizontal compaction happens inside the segment, already
loaded into cache are invalid. So if we do clean files,
physical deletion of horizontal compacted takes place,
but still the cache contains old files. So when select
query is fired, query fails with file not found exception.
Solution:
once after horizontal compaction is finished, new compacted
files are generated, so the segments inside cache are now invalid,
so clear the cache of invalid segment after horizontal compaction.
During drop cache command, clear the cache of segmentMap also.
This closes #3385
---
.../sql/execution/command/cache/CarbonDropCacheCommand.scala | 8 +++-----
.../sql/execution/command/mutation/HorizontalCompaction.scala | 9 ++++++++-
2 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala
index 1554f6a..7b8e10f 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonDropCacheCommand.scala
@@ -25,7 +25,7 @@ import org.apache.spark.sql.execution.command.MetadataCommand
import org.apache.carbondata.common.logging.LogServiceFactory
import org.apache.carbondata.core.cache.CacheProvider
-import org.apache.carbondata.core.datamap.DataMapUtil
+import org.apache.carbondata.core.datamap.{DataMapStoreManager, DataMapUtil}
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
import org.apache.carbondata.core.util.CarbonProperties
import org.apache.carbondata.events.{DropTableCacheEvent, OperationContext, OperationListenerBus}
@@ -55,13 +55,11 @@ case class CarbonDropCacheCommand(tableIdentifier: TableIdentifier, internalCall
carbonTable.getTableName)) {
DataMapUtil.executeClearDataMapJob(carbonTable, DataMapUtil.DISTRIBUTED_JOB_NAME)
} else {
- val allIndexFiles = CacheUtil.getAllIndexFiles(carbonTable)(sparkSession)
// Extract dictionary keys for the table and create cache keys from those
val dictKeys: List[String] = CacheUtil.getAllDictCacheKeys(carbonTable)
-
// Remove elements from cache
- val keysToRemove = allIndexFiles ++ dictKeys
- cache.removeAll(keysToRemove.asJava)
+ cache.removeAll(dictKeys.asJava)
+ DataMapStoreManager.getInstance().clearDataMaps(carbonTable.getAbsoluteTableIdentifier)
}
}
LOGGER.info("Drop cache request served for table " + carbonTable.getTableUniqueName)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/HorizontalCompaction.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/HorizontalCompaction.scala
index fb20e4f..62a3486 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/HorizontalCompaction.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/HorizontalCompaction.scala
@@ -28,7 +28,7 @@ import org.apache.spark.sql.execution.command.management.CarbonAlterTableCompact
import org.apache.spark.sql.util.SparkSQLUtil
import org.apache.carbondata.common.logging.LogServiceFactory
-import org.apache.carbondata.core.datamap.Segment
+import org.apache.carbondata.core.datamap.{DataMapStoreManager, Segment}
import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
import org.apache.carbondata.core.statusmanager.SegmentUpdateStatusManager
@@ -106,6 +106,13 @@ object HorizontalCompaction {
segmentUpdateStatusManager,
deleteTimeStamp,
segLists)
+
+ // If there are already index and data files are present for old update operation, then the
+ // cache will be loaded for those files during current update, but once after horizontal
+ // compaction is finished, new compacted files are generated, so the segments inside cache are
+ // now invalid, so clear the cache of invalid segment after horizontal compaction.
+ DataMapStoreManager.getInstance()
+ .clearInvalidSegments(carbonTable, segLists.asScala.map(_.getSegmentNo).asJava)
}
/**
[carbondata] 24/33: [HOTFIX] Remove duplicate case for BYTE_ARRAY
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 336848365c06cc10d6bb39691b35f657be565c10
Author: Manhua <ke...@qq.com>
AuthorDate: Fri Sep 20 14:26:48 2019 +0800
[HOTFIX] Remove duplicate case for BYTE_ARRAY
This closes #3396
---
core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java | 4 ----
1 file changed, 4 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java b/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
index adb63cd..3e0edb1 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/DataTypeUtil.java
@@ -1018,8 +1018,6 @@ public final class DataTypeUtil {
return DataTypes.NULL;
} else if (DataTypes.BYTE_ARRAY.getName().equalsIgnoreCase(name)) {
return DataTypes.BYTE_ARRAY;
- } else if (DataTypes.BYTE_ARRAY.getName().equalsIgnoreCase(name)) {
- return DataTypes.BYTE_ARRAY;
} else if (DataTypes.BINARY.getName().equalsIgnoreCase(name)) {
return DataTypes.BINARY;
} else if (name.equalsIgnoreCase("decimal")) {
@@ -1070,8 +1068,6 @@ public final class DataTypeUtil {
return DataTypes.NULL;
} else if (DataTypes.BYTE_ARRAY.getName().equalsIgnoreCase(dataType.getName())) {
return DataTypes.BYTE_ARRAY;
- } else if (DataTypes.BYTE_ARRAY.getName().equalsIgnoreCase(dataType.getName())) {
- return DataTypes.BYTE_ARRAY;
} else if (DataTypes.BINARY.getName().equalsIgnoreCase(dataType.getName())) {
return DataTypes.BINARY;
} else if (dataType.getName().equalsIgnoreCase("decimal")) {
[carbondata] 30/33: [CARBONDATA-3523] Store data file size into
index file
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 49e9ea3c6a75532e4f51d924bce4334687597a9c
Author: QiangCai <qi...@qq.com>
AuthorDate: Tue Aug 13 10:25:31 2019 +0800
[CARBONDATA-3523] Store data file size into index file
In BlockIndex, the file_size is always zero. We can set the actual value during data loading and use it during the query to improve the query performance.
1. avoid invoking listFiles for each segment
2. avoid invoking getFileStatus for each data file
This closes #3356
---
.../core/datastore/block/TableBlockInfo.java | 13 +++++++++++++
.../carbondata/core/metadata/index/BlockIndexInfo.java | 18 ++++++++++++++++++
.../core/util/AbstractDataFileFooterConverter.java | 3 +++
.../carbondata/core/util/BlockletDataMapUtil.java | 17 ++++++++++++++---
.../carbondata/core/util/CarbonMetadataUtil.java | 1 +
.../store/writer/AbstractFactDataWriter.java | 5 +++--
.../store/writer/v3/CarbonFactDataWriterImplV3.java | 18 ++++++++++--------
7 files changed, 62 insertions(+), 13 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/datastore/block/TableBlockInfo.java b/core/src/main/java/org/apache/carbondata/core/datastore/block/TableBlockInfo.java
index 25d82f8..4dd1403 100644
--- a/core/src/main/java/org/apache/carbondata/core/datastore/block/TableBlockInfo.java
+++ b/core/src/main/java/org/apache/carbondata/core/datastore/block/TableBlockInfo.java
@@ -54,6 +54,11 @@ public class TableBlockInfo implements Distributable, Serializable {
private String filePath;
/**
+ * file size of the block
+ */
+ private long fileSize;
+
+ /**
* block offset in the file
*/
private long blockOffset;
@@ -439,6 +444,14 @@ public class TableBlockInfo implements Distributable, Serializable {
this.filePath = filePath;
}
+ public long getFileSize() {
+ return fileSize;
+ }
+
+ public void setFileSize(long fileSize) {
+ this.fileSize = fileSize;
+ }
+
public BlockletDetailInfo getDetailInfo() {
return detailInfo;
}
diff --git a/core/src/main/java/org/apache/carbondata/core/metadata/index/BlockIndexInfo.java b/core/src/main/java/org/apache/carbondata/core/metadata/index/BlockIndexInfo.java
index ae99ed8..f7f2d3c 100644
--- a/core/src/main/java/org/apache/carbondata/core/metadata/index/BlockIndexInfo.java
+++ b/core/src/main/java/org/apache/carbondata/core/metadata/index/BlockIndexInfo.java
@@ -51,6 +51,11 @@ public class BlockIndexInfo {
private BlockletInfo blockletInfo;
/**
+ * file size
+ */
+ private long fileSize;
+
+ /**
* Constructor
*
* @param numberOfRows number of rows
@@ -80,6 +85,12 @@ public class BlockIndexInfo {
this.blockletInfo = blockletInfo;
}
+ public BlockIndexInfo(long numberOfRows, String fileName, long offset,
+ BlockletIndex blockletIndex, BlockletInfo blockletInfo, long fileSize) {
+ this(numberOfRows, fileName, offset, blockletIndex, blockletInfo);
+ this.fileSize = fileSize;
+ }
+
/**
* @return the numberOfRows
*/
@@ -114,4 +125,11 @@ public class BlockIndexInfo {
public BlockletInfo getBlockletInfo() {
return blockletInfo;
}
+
+ /**
+ * @return file size
+ */
+ public long getFileSize() {
+ return fileSize;
+ }
}
diff --git a/core/src/main/java/org/apache/carbondata/core/util/AbstractDataFileFooterConverter.java b/core/src/main/java/org/apache/carbondata/core/util/AbstractDataFileFooterConverter.java
index 64d30c2..f16a3ae 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/AbstractDataFileFooterConverter.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/AbstractDataFileFooterConverter.java
@@ -244,6 +244,9 @@ public abstract class AbstractDataFileFooterConverter {
}
fileName = (CarbonCommonConstants.FILE_SEPARATOR + fileName).replaceAll("//", "/");
tableBlockInfo.setFilePath(parentPath + fileName);
+ if (readBlockIndexInfo.isSetFile_size()) {
+ tableBlockInfo.setFileSize(readBlockIndexInfo.getFile_size());
+ }
return tableBlockInfo;
}
diff --git a/core/src/main/java/org/apache/carbondata/core/util/BlockletDataMapUtil.java b/core/src/main/java/org/apache/carbondata/core/util/BlockletDataMapUtil.java
index 6cd60a2..5a988c4 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/BlockletDataMapUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/BlockletDataMapUtil.java
@@ -38,9 +38,11 @@ import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
import org.apache.carbondata.core.datamap.Segment;
import org.apache.carbondata.core.datastore.block.SegmentProperties;
+import org.apache.carbondata.core.datastore.block.TableBlockInfo;
import org.apache.carbondata.core.datastore.compression.CompressorFactory;
import org.apache.carbondata.core.datastore.filesystem.AbstractDFSCarbonFile;
import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
+import org.apache.carbondata.core.datastore.filesystem.S3CarbonFile;
import org.apache.carbondata.core.datastore.impl.FileFactory;
import org.apache.carbondata.core.indexstore.BlockMetaInfo;
import org.apache.carbondata.core.indexstore.TableBlockIndexUniqueIdentifier;
@@ -124,7 +126,8 @@ public class BlockletDataMapUtil {
}
String blockPath = footer.getBlockInfo().getTableBlockInfo().getFilePath();
if (null == blockMetaInfoMap.get(blockPath)) {
- BlockMetaInfo blockMetaInfo = createBlockMetaInfo(fileNameToMetaInfoMapping, blockPath);
+ BlockMetaInfo blockMetaInfo = createBlockMetaInfo(
+ fileNameToMetaInfoMapping, footer.getBlockInfo().getTableBlockInfo());
// if blockMetaInfo is null that means the file has been deleted from the file system.
// This can happen in case IUD scenarios where after deleting or updating the data the
// complete block is deleted but the entry still exists in index or merge index file
@@ -148,7 +151,7 @@ public class BlockletDataMapUtil {
String segmentFilePath, Configuration configuration) throws IOException {
Map<String, BlockMetaInfo> fileNameToMetaInfoMapping = new TreeMap();
CarbonFile carbonFile = FileFactory.getCarbonFile(segmentFilePath, configuration);
- if (carbonFile instanceof AbstractDFSCarbonFile) {
+ if (carbonFile instanceof AbstractDFSCarbonFile && !(carbonFile instanceof S3CarbonFile)) {
PathFilter pathFilter = new PathFilter() {
@Override public boolean accept(Path path) {
return CarbonTablePath.isCarbonDataFile(path.getName());
@@ -166,11 +169,19 @@ public class BlockletDataMapUtil {
}
private static BlockMetaInfo createBlockMetaInfo(
- Map<String, BlockMetaInfo> fileNameToMetaInfoMapping, String carbonDataFile)
+ Map<String, BlockMetaInfo> fileNameToMetaInfoMapping, TableBlockInfo blockInfo)
throws IOException {
+ String carbonDataFile = blockInfo.getFilePath();
FileFactory.FileType fileType = FileFactory.getFileType(carbonDataFile);
switch (fileType) {
+ case S3:
case LOCAL:
+ // consider backward compatibility
+ // when the file size in blockInfo is not zero, use this file size in blockInfo.
+ if (blockInfo.getFileSize() != 0) {
+ return new BlockMetaInfo(new String[] { "localhost" }, blockInfo.getFileSize());
+ }
+ // when the file size in blockInfo is zero, get the size of this file.
if (!FileFactory.isFileExist(carbonDataFile)) {
return null;
}
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java b/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java
index 7414ab7..5777f4d 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java
@@ -392,6 +392,7 @@ public class CarbonMetadataUtil {
if (blockIndexInfo.getBlockletInfo() != null) {
blockIndex.setBlocklet_info(getBlocletInfo3(blockIndexInfo.getBlockletInfo()));
}
+ blockIndex.setFile_size(blockIndexInfo.getFileSize());
thriftBlockIndexList.add(blockIndex);
}
return thriftBlockIndexList;
diff --git a/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java b/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
index eb1b15d..fbe5953 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
@@ -365,10 +365,11 @@ public abstract class AbstractFactDataWriter implements CarbonFactDataWriter {
*
* @param numberOfRows number of rows in file
* @param carbonDataFileName The name of carbonData file
- * @param currentPosition current offset
+ * @param footerOffset footer offset
+ * @param fileSize
*/
protected abstract void fillBlockIndexInfoDetails(long numberOfRows, String carbonDataFileName,
- long currentPosition);
+ long footerOffset, long fileSize);
public static List<org.apache.carbondata.format.ColumnSchema> getColumnSchemaListAndCardinality(
List<Integer> cardinality, int[] dictionaryColumnCardinality,
diff --git a/processing/src/main/java/org/apache/carbondata/processing/store/writer/v3/CarbonFactDataWriterImplV3.java b/processing/src/main/java/org/apache/carbondata/processing/store/writer/v3/CarbonFactDataWriterImplV3.java
index cac0e8b..c8673a0 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/store/writer/v3/CarbonFactDataWriterImplV3.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/store/writer/v3/CarbonFactDataWriterImplV3.java
@@ -101,7 +101,7 @@ public class CarbonFactDataWriterImplV3 extends AbstractFactDataWriter {
protected void writeFooterToFile() throws CarbonDataWriterException {
try {
// get the current file position
- long currentPosition = currentOffsetInFile;
+ long footerOffset = currentOffsetInFile;
// get thrift file footer instance
FileFooter3 convertFileMeta = CarbonMetadataUtil
.convertFileFooterVersion3(blockletMetadata, blockletIndex, localCardinality,
@@ -116,16 +116,17 @@ public class CarbonFactDataWriterImplV3 extends AbstractFactDataWriter {
convertFileMeta.putToExtra_info(CarbonCommonConstants.CARBON_WRITTEN_BY_FOOTER_INFO, appName);
convertFileMeta.putToExtra_info(CarbonCommonConstants.CARBON_WRITTEN_VERSION,
CarbonVersionConstants.CARBONDATA_VERSION);
- // fill the carbon index details
- fillBlockIndexInfoDetails(convertFileMeta.getNum_rows(), carbonDataFileName, currentPosition);
// write the footer
byte[] byteArray = CarbonUtil.getByteArray(convertFileMeta);
ByteBuffer buffer =
ByteBuffer.allocate(byteArray.length + CarbonCommonConstants.LONG_SIZE_IN_BYTE);
buffer.put(byteArray);
- buffer.putLong(currentPosition);
+ buffer.putLong(footerOffset);
buffer.flip();
currentOffsetInFile += fileChannel.write(buffer);
+ // fill the carbon index details
+ fillBlockIndexInfoDetails(
+ convertFileMeta.getNum_rows(), carbonDataFileName, footerOffset, currentOffsetInFile);
} catch (IOException e) {
LOGGER.error("Problem while writing the carbon file", e);
throw new CarbonDataWriterException("Problem while writing the carbon file: ", e);
@@ -346,11 +347,12 @@ public class CarbonFactDataWriterImplV3 extends AbstractFactDataWriter {
*
* @param numberOfRows number of rows in file
* @param carbonDataFileName The name of carbonData file
- * @param currentPosition current offset
+ * @param footerOffset footer offset
+ * @param fileSize file size
*/
@Override
protected void fillBlockIndexInfoDetails(long numberOfRows, String carbonDataFileName,
- long currentPosition) {
+ long footerOffset, long fileSize) {
int i = 0;
DataFileFooterConverterV3 converterV3 = new DataFileFooterConverterV3();
for (org.apache.carbondata.format.BlockletIndex index : blockletIndex) {
@@ -367,8 +369,8 @@ public class CarbonFactDataWriterImplV3 extends AbstractFactDataWriter {
new org.apache.carbondata.core.metadata.blocklet.index.BlockletIndex(bTreeIndex,
minMaxIndex);
BlockIndexInfo biInfo =
- new BlockIndexInfo(numberOfRows, carbonDataFileName, currentPosition, bIndex,
- blockletInfo);
+ new BlockIndexInfo(numberOfRows, carbonDataFileName, footerOffset, bIndex,
+ blockletInfo, fileSize);
blockIndexInfoList.add(biInfo);
i++;
}
[carbondata] 10/33: [CARBONDATA-3505] Drop database cascade fix
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 2328707b4477a11b7713aee8f123b780ad48cc25
Author: kunal642 <ku...@gmail.com>
AuthorDate: Tue Aug 27 14:49:58 2019 +0530
[CARBONDATA-3505] Drop database cascade fix
Problem: When 2 databases are created on same location and one of them is dropped
then the folder is also deleted from backend. If we try to drop the 2nd database
then it would try to lookup the other table, but the schema file would not exist
in the backend and the drop will fail.
Solution: Add a check to call CarbonDropDatabaseCommand only if the database
location exists in the backend.
This closes #3365
---
.../main/scala/org/apache/spark/sql/CarbonEnv.scala | 19 ++++++++++++++++++-
.../command/cache/CarbonShowCacheCommand.scala | 4 ++--
.../spark/sql/execution/strategy/DDLStrategy.scala | 4 +++-
.../apache/spark/sql/hive/CarbonFileMetastore.scala | 4 ++--
4 files changed, 25 insertions(+), 6 deletions(-)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
index 1cbd156..f2a52d2 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql
import java.util.concurrent.ConcurrentHashMap
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
+import org.apache.spark.sql.catalyst.analysis.{NoSuchDatabaseException, NoSuchTableException}
import org.apache.spark.sql.catalyst.catalog.SessionCatalog
import org.apache.spark.sql.events.{MergeBloomIndexEventListener, MergeIndexEventListener}
import org.apache.spark.sql.execution.command.cache._
@@ -267,6 +267,23 @@ object CarbonEnv {
}
/**
+ * Returns true with the database folder exists in file system. False in all other scenarios.
+ */
+ def databaseLocationExists(dbName: String,
+ sparkSession: SparkSession, ifExists: Boolean): Boolean = {
+ try {
+ FileFactory.getCarbonFile(getDatabaseLocation(dbName, sparkSession)).exists()
+ } catch {
+ case e: NoSuchDatabaseException =>
+ if (ifExists) {
+ false
+ } else {
+ throw e
+ }
+ }
+ }
+
+ /**
* The method returns the database location
* if carbon.storeLocation does point to spark.sql.warehouse.dir then returns
* the database locationUri as database location else follows the old behaviour
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
index 45e811a..4b7f680 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
@@ -443,9 +443,9 @@ case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
case (_, _, sum, provider) =>
provider.toLowerCase match {
case `bloomFilterIdentifier` =>
- allIndexSize += sum
- case _ =>
allDatamapSize += sum
+ case _ =>
+ allIndexSize += sum
}
}
(allIndexSize, allDatamapSize)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala
index 4791687..3ef8cfa 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala
@@ -37,6 +37,7 @@ import org.apache.spark.util.{CarbonReflectionUtils, DataMapUtil, FileUtils, Spa
import org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
import org.apache.carbondata.core.metadata.schema.table.CarbonTable
import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
import org.apache.carbondata.spark.util.Util
@@ -115,7 +116,8 @@ class DDLStrategy(sparkSession: SparkSession) extends SparkStrategy {
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
FileUtils.createDatabaseDirectory(dbName, dbLocation, sparkSession.sparkContext)
ExecutedCommandExec(createDb) :: Nil
- case drop@DropDatabaseCommand(dbName, ifExists, isCascade) =>
+ case drop@DropDatabaseCommand(dbName,
+ ifExists, isCascade) if CarbonEnv.databaseLocationExists(dbName, sparkSession, ifExists) =>
ExecutedCommandExec(CarbonDropDatabaseCommand(drop)) :: Nil
case alterTable@CarbonAlterTableCompactionCommand(altertablemodel, _, _) =>
val isCarbonTable = CarbonEnv.getInstance(sparkSession).carbonMetaStore
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
index 7ab2d47..b19b11c 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
@@ -288,13 +288,13 @@ class CarbonFileMetastore extends CarbonMetaStore {
Some(wrapperTableInfo)
} else {
val tableMetadataFile = CarbonTablePath.getSchemaFilePath(tablePath)
- schemaRefreshTime = FileFactory
- .getCarbonFile(CarbonTablePath.getSchemaFilePath(tablePath)).getLastModifiedTime
val fileType = FileFactory.getFileType(tableMetadataFile)
if (FileFactory.isFileExist(tableMetadataFile, fileType)) {
val tableInfo: TableInfo = CarbonUtil.readSchemaFile(tableMetadataFile)
val wrapperTableInfo =
schemaConverter.fromExternalToWrapperTableInfo(tableInfo, dbName, tableName, tablePath)
+ schemaRefreshTime = FileFactory
+ .getCarbonFile(tableMetadataFile).getLastModifiedTime
Some(wrapperTableInfo)
} else {
None
[carbondata] 17/33: [HOTFIX] Fix NPE on windows
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 631941b566ba015921ef2cdc89434f35d88aef5f
Author: Manhua <ke...@qq.com>
AuthorDate: Thu Sep 5 21:13:28 2019 +0800
[HOTFIX] Fix NPE on windows
Analyse
carbon index files are merged but SegmentFile did not update, so it fails to get any default datamap for pruning.
The reason for no updated is about path comparison like
/[your_path_here]/examples/spark2/target/store/default/source/\Fact\Part0\Segment_0
vs
D:/[your_path_here]/examples/spark2/target/store/default/source/Fact/Part0/Segment_0
Solution
use the AbsoluteTableIdentifier of table from CarbonMetadata
instead of the newly created object, keep the path style same
This closes #3377
---
.../main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
index 684bcbb..900b69c 100644
--- a/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
+++ b/integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
@@ -151,8 +151,8 @@ class CarbonFileMetastore extends CarbonMetaStore {
val tables = Option(CarbonMetadata.getInstance.getCarbonTable(database, tableName))
tables match {
case Some(t) =>
- if (isSchemaRefreshed(absIdentifier, sparkSession)) {
- readCarbonSchema(absIdentifier, parameters)
+ if (isSchemaRefreshed(t.getAbsoluteTableIdentifier, sparkSession)) {
+ readCarbonSchema(t.getAbsoluteTableIdentifier, parameters)
} else {
CarbonRelation(database, tableName, CarbonSparkUtil.createSparkMeta(t), t)
}
[carbondata] 26/33: [HOTFIX] Fix wrong min/max index of measure
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit de81b380b71b111061bf27d8837b50a995631268
Author: QiangCai <qi...@qq.com>
AuthorDate: Wed Sep 18 21:02:15 2019 +0800
[HOTFIX] Fix wrong min/max index of measure
This closes #3394
---
.../carbondata/core/util/CarbonMetadataUtil.java | 64 ++++----
.../org/apache/carbondata/sdk/file/MinMaxTest.java | 161 +++++++++++++++++++++
2 files changed, 188 insertions(+), 37 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java b/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java
index f35afc0..7414ab7 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java
@@ -477,50 +477,40 @@ public class CarbonMetadataUtil {
ByteBuffer firstBuffer = null;
ByteBuffer secondBuffer = null;
if (dataType == DataTypes.BOOLEAN || dataType == DataTypes.BYTE) {
- return first[0] - second[0];
+ if (first[0] > second[0]) {
+ return 1;
+ } else if (first[0] < second[0]) {
+ return -1;
+ }
+ return 0;
} else if (dataType == DataTypes.DOUBLE) {
- firstBuffer = ByteBuffer.allocate(8);
- firstBuffer.put(first);
- secondBuffer = ByteBuffer.allocate(8);
- secondBuffer.put(second);
- firstBuffer.flip();
- secondBuffer.flip();
- double compare = firstBuffer.getDouble() - secondBuffer.getDouble();
- if (compare > 0) {
- compare = 1;
- } else if (compare < 0) {
- compare = -1;
+ double firstValue = ((ByteBuffer) (ByteBuffer.allocate(8).put(first).flip())).getDouble();
+ double secondValue = ((ByteBuffer) (ByteBuffer.allocate(8).put(second).flip())).getDouble();
+ if (firstValue > secondValue) {
+ return 1;
+ } else if (firstValue < secondValue) {
+ return -1;
}
- return (int) compare;
+ return 0;
} else if (dataType == DataTypes.FLOAT) {
- firstBuffer = ByteBuffer.allocate(8);
- firstBuffer.put(first);
- secondBuffer = ByteBuffer.allocate(8);
- secondBuffer.put(second);
- firstBuffer.flip();
- secondBuffer.flip();
- double compare = firstBuffer.getFloat() - secondBuffer.getFloat();
- if (compare > 0) {
- compare = 1;
- } else if (compare < 0) {
- compare = -1;
+ float firstValue = ((ByteBuffer) (ByteBuffer.allocate(8).put(first).flip())).getFloat();
+ float secondValue = ((ByteBuffer) (ByteBuffer.allocate(8).put(second).flip())).getFloat();
+ if (firstValue > secondValue) {
+ return 1;
+ } else if (firstValue < secondValue) {
+ return -1;
}
- return (int) compare;
+ return 0;
} else if (dataType == DataTypes.LONG || dataType == DataTypes.INT
|| dataType == DataTypes.SHORT) {
- firstBuffer = ByteBuffer.allocate(8);
- firstBuffer.put(first);
- secondBuffer = ByteBuffer.allocate(8);
- secondBuffer.put(second);
- firstBuffer.flip();
- secondBuffer.flip();
- long compare = firstBuffer.getLong() - secondBuffer.getLong();
- if (compare > 0) {
- compare = 1;
- } else if (compare < 0) {
- compare = -1;
+ long firstValue = ((ByteBuffer) (ByteBuffer.allocate(8).put(first).flip())).getLong();
+ long secondValue = ((ByteBuffer) (ByteBuffer.allocate(8).put(second).flip())).getLong();
+ if (firstValue > secondValue) {
+ return 1;
+ } else if (firstValue < secondValue) {
+ return -1;
}
- return (int) compare;
+ return 0;
} else if (DataTypes.isDecimal(dataType)) {
return DataTypeUtil.byteToBigDecimal(first).compareTo(DataTypeUtil.byteToBigDecimal(second));
} else {
diff --git a/store/sdk/src/test/java/org/apache/carbondata/sdk/file/MinMaxTest.java b/store/sdk/src/test/java/org/apache/carbondata/sdk/file/MinMaxTest.java
new file mode 100644
index 0000000..c26fdd5
--- /dev/null
+++ b/store/sdk/src/test/java/org/apache/carbondata/sdk/file/MinMaxTest.java
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.file;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.reader.CarbonIndexFileReader;
+import org.apache.carbondata.format.BlockIndex;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * MinMaxTest
+ */
+public class MinMaxTest {
+ String path = "./testMinMax";
+ @Before
+ public void mkDir() throws IOException {
+ File file = new File(path);
+ FileUtils.deleteDirectory(file);
+ FileUtils.forceMkdir(file);
+ }
+
+ @After
+ public void deleteDir() throws IOException {
+ FileUtils.deleteDirectory(new File(path));
+ }
+
+ @Test
+ public void testMinMax() throws IOException {
+ Field[] fields = new Field[6];
+ fields[0] = new Field("byteField", DataTypes.BYTE);
+ fields[1] = new Field("shortField", DataTypes.SHORT);
+ fields[2] = new Field("infField", DataTypes.INT);
+ fields[3] = new Field("longField", DataTypes.LONG);
+ fields[4] = new Field("floatField", DataTypes.FLOAT);
+ fields[5] = new Field("doubleField", DataTypes.DOUBLE);
+ writeFilesAndVerify(new Schema(fields), path);
+ readIndexAndVerify(path);
+ }
+
+ public static void writeFilesAndVerify(Schema schema, String path) {
+ try {
+ CarbonWriter writer =
+ CarbonWriter
+ .builder()
+ .outputPath(path)
+ .withCsvInput(schema)
+ .writtenBy("TestUtil")
+ .build();
+ String[] row1 = new String[] {
+ String.valueOf(Byte.MIN_VALUE),
+ String.valueOf(Short.MIN_VALUE),
+ String.valueOf(Integer.MIN_VALUE),
+ String.valueOf(Long.MIN_VALUE),
+ String.valueOf(Float.MIN_VALUE),
+ String.valueOf(Double.MIN_VALUE)
+ };
+ for (int i = 0; i < 10; i++) {
+ writer.write(row1);
+ }
+ String[] row2 = new String[] {
+ String.valueOf(Byte.MAX_VALUE),
+ String.valueOf(Short.MAX_VALUE),
+ String.valueOf(Integer.MAX_VALUE),
+ String.valueOf(Long.MAX_VALUE),
+ String.valueOf(Float.MAX_VALUE),
+ String.valueOf(Double.MAX_VALUE)
+ };
+ for (int i = 0; i < 26664; i++) {
+ writer.write(row2);
+ }
+ String[] row3 = new String[] {
+ String.valueOf(Byte.MAX_VALUE - 10),
+ String.valueOf(Short.MAX_VALUE - 10),
+ String.valueOf(Integer.MAX_VALUE - 10),
+ String.valueOf(Long.MAX_VALUE - 10),
+ String.valueOf(Float.MAX_VALUE - 10),
+ String.valueOf(Double.MAX_VALUE - 10)
+ };
+ for (int i = 0; i < 6666; i++) {
+ writer.write(row3);
+ }
+ writer.close();
+ } catch (Exception e) {
+ e.printStackTrace();
+ Assert.assertTrue(false);
+ }
+ File[] dataFiles = new File(path).listFiles(new FileFilter() {
+ @Override public boolean accept(File pathname) {
+ return pathname.getName().endsWith(CarbonCommonConstants.FACT_FILE_EXT);
+ }
+ });
+ Assert.assertTrue(dataFiles != null);
+ Assert.assertTrue(dataFiles.length > 0);
+ }
+
+ public static void readIndexAndVerify(String path) {
+ File[] dataFiles = new File(path).listFiles(new FileFilter() {
+ @Override public boolean accept(File pathname) {
+ return pathname.getName().endsWith(CarbonCommonConstants.UPDATE_INDEX_FILE_EXT);
+ }
+ });
+ Assert.assertTrue(dataFiles != null);
+ Assert.assertTrue(dataFiles.length > 0);
+ CarbonIndexFileReader reader = new CarbonIndexFileReader();
+ try {
+ reader.openThriftReader(dataFiles[0].getCanonicalPath());
+ reader.readIndexHeader();
+ BlockIndex index = reader.readBlockIndexInfo();
+ List<ByteBuffer> maxValues = index.block_index.min_max_index.max_values;
+ Assert.assertEquals(Byte.MAX_VALUE, maxValues.get(0).get());
+ maxValues.get(1).position(6);
+ Assert.assertEquals(Short.MAX_VALUE, maxValues.get(1).getShort());
+ maxValues.get(2).position(4);
+ Assert.assertEquals(Integer.MAX_VALUE, maxValues.get(2).getInt());
+ Assert.assertEquals(Long.MAX_VALUE, maxValues.get(3).getLong());
+ Assert.assertTrue(Float.MAX_VALUE - maxValues.get(4).getFloat() < 1);
+ Assert.assertTrue(Double.MAX_VALUE - maxValues.get(5).getDouble() < 1);
+ List<ByteBuffer> minValues = index.block_index.min_max_index.min_values;
+ Assert.assertEquals(Byte.MIN_VALUE, minValues.get(0).get());
+ minValues.get(1).position(6);
+ Assert.assertEquals(Short.MIN_VALUE, minValues.get(1).getShort());
+ minValues.get(2).position(4);
+ Assert.assertEquals(Integer.MIN_VALUE, minValues.get(2).getInt());
+ Assert.assertEquals(Long.MIN_VALUE, minValues.get(3).getLong());
+ Assert.assertTrue(minValues.get(4).getFloat() - Float.MIN_VALUE < 1);
+ Assert.assertTrue(minValues.get(5).getDouble() - Double.MIN_VALUE < 1);
+ } catch (IOException e) {
+ e.printStackTrace();
+ Assert.assertTrue(false);
+ } finally {
+ reader.closeThriftReader();
+ }
+ }
+}
[carbondata] 06/33: [CARBONDATA-3509] Support disable query
prefetch by configuration
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 129b1634aa8eab29e6ec75653a9d9bd451d965b1
Author: ajantha-bhat <aj...@gmail.com>
AuthorDate: Fri Aug 30 11:08:09 2019 +0530
[CARBONDATA-3509] Support disable query prefetch by configuration
Support disable query prefetch by configuration:
Prefetch runs in asynch thread in query and it is always enable in query flow. If some user wants to disable it, they can use this property to disable and observe in logs.
This closes #3370
---
.../carbondata/core/constants/CarbonCommonConstants.java | 9 +++++++++
.../core/scan/executor/impl/AbstractQueryExecutor.java | 2 ++
.../apache/carbondata/core/scan/model/QueryModel.java | 4 +++-
.../apache/carbondata/core/util/CarbonProperties.java | 16 ++++++++++++++++
docs/configuration-parameters.md | 1 +
5 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
index 17b191d..67fa13f 100644
--- a/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
+++ b/core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
@@ -1483,6 +1483,15 @@ public final class CarbonCommonConstants {
public static final String CARBON_MAX_EXECUTOR_THREADS_FOR_BLOCK_PRUNING_DEFAULT = "4";
+ /*
+ * whether to enable prefetch for query
+ */
+ @CarbonProperty
+ public static final String CARBON_QUERY_PREFETCH_ENABLE =
+ "carbon.query.prefetch.enable";
+
+ public static final String CARBON_QUERY_PREFETCH_ENABLE_DEFAULT = "true";
+
//////////////////////////////////////////////////////////////////////////////////////////
// Datamap parameter start here
//////////////////////////////////////////////////////////////////////////////////////////
diff --git a/core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java b/core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java
index b3d4780..6760e77 100644
--- a/core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java
+++ b/core/src/main/java/org/apache/carbondata/core/scan/executor/impl/AbstractQueryExecutor.java
@@ -493,7 +493,9 @@ public abstract class AbstractQueryExecutor<E> implements QueryExecutor<E> {
segmentProperties.getDimensionOrdinalToChunkMapping().size());
if (queryModel.isReadPageByPage()) {
blockExecutionInfo.setPrefetchBlocklet(false);
+ LOGGER.info("Query prefetch is: false, read page by page");
} else {
+ LOGGER.info("Query prefetch is: " + queryModel.isPreFetchData());
blockExecutionInfo.setPrefetchBlocklet(queryModel.isPreFetchData());
}
// In case of fg datamap it should not go to direct fill.
diff --git a/core/src/main/java/org/apache/carbondata/core/scan/model/QueryModel.java b/core/src/main/java/org/apache/carbondata/core/scan/model/QueryModel.java
index 267527f..4d10492 100644
--- a/core/src/main/java/org/apache/carbondata/core/scan/model/QueryModel.java
+++ b/core/src/main/java/org/apache/carbondata/core/scan/model/QueryModel.java
@@ -34,6 +34,7 @@ import org.apache.carbondata.core.scan.expression.UnknownExpression;
import org.apache.carbondata.core.scan.expression.conditional.ConditionalExpression;
import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
import org.apache.carbondata.core.stats.QueryStatisticsRecorder;
+import org.apache.carbondata.core.util.CarbonProperties;
import org.apache.carbondata.core.util.CarbonUtil;
import org.apache.carbondata.core.util.DataTypeConverter;
@@ -110,7 +111,7 @@ public class QueryModel {
// whether to clear/free unsafe memory or not
private boolean freeUnsafeMemory = true;
- private boolean preFetchData = true;
+ private boolean preFetchData;
/**
* It fills the vector directly from decoded column page with out any staging and conversions.
@@ -125,6 +126,7 @@ public class QueryModel {
tableBlockInfos = new ArrayList<TableBlockInfo>();
this.table = carbonTable;
this.queryId = String.valueOf(System.nanoTime());
+ this.preFetchData = CarbonProperties.getQueryPrefetchEnable();
}
public static QueryModel newInstance(CarbonTable carbonTable) {
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java b/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
index c60dad8..adf4905 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java
@@ -1773,4 +1773,20 @@ public final class CarbonProperties {
}
}
}
+
+ /**
+ * Validate and get query prefetch enable
+ *
+ * @return boolean prefetch value
+ */
+ public static Boolean getQueryPrefetchEnable() {
+ String prefetchEnable = CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.CARBON_QUERY_PREFETCH_ENABLE);
+ if (prefetchEnable == null) {
+ return Boolean.parseBoolean(CarbonCommonConstants.CARBON_QUERY_PREFETCH_ENABLE_DEFAULT);
+ } else {
+ // return false only if false is set. any other case return true
+ return !prefetchEnable.equalsIgnoreCase("false");
+ }
+ }
}
diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md
index 808d507..da226ec 100644
--- a/docs/configuration-parameters.md
+++ b/docs/configuration-parameters.md
@@ -142,6 +142,7 @@ This section provides the details of all the configurations required for the Car
| carbon.max.driver.threads.for.block.pruning | 4 | Number of threads used for driver pruning when the carbon files are more than 100k Maximum memory. This configuration can used to set number of threads between 1 to 4. |
| carbon.heap.memory.pooling.threshold.bytes | 1048576 | CarbonData supports unsafe operations of Java to avoid GC overhead for certain operations. Using unsafe, memory can be allocated on Java Heap or off heap. This configuration controls the allocation mechanism on Java HEAP. If the heap memory allocations of the given size is greater or equal than this value,it should go through the pooling mechanism. But if set this size to -1, it should not go through the pooling mechanism. Default [...]
| carbon.push.rowfilters.for.vector | false | When enabled complete row filters will be handled by carbon in case of vector. If it is disabled then only page level pruning will be done by carbon and row level filtering will be done by spark for vector. And also there are scan optimizations in carbon to avoid multiple data copies when this parameter is set to false. There is no change in flow for non-vector based queries. |
+| carbon.query.prefetch.enable | true | By default this property is true, so prefetch is used in query to read next blocklet asynchronously in other thread while processing current blocklet in main thread. This can help to reduce CPU idle time. Setting this property false will disable this prefetch feature in query. |
## Data Mutation Configuration
| Parameter | Default Value | Description |
[carbondata] 22/33: [HOTFIX] fix incorrect word in index-server doc
Posted by ra...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a commit to branch branch-1.6
in repository https://gitbox.apache.org/repos/asf/carbondata.git
commit 8ffbc1d6afda2ad77efeb738168b972239c89731
Author: lamber-ken <22...@qq.com>
AuthorDate: Tue Sep 17 02:08:40 2019 +0800
[HOTFIX] fix incorrect word in index-server doc
fix incorrect word in index-server doc
This closes #3390
---
docs/index-server.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/index-server.md b/docs/index-server.md
index 9253f2a..0b888c4 100644
--- a/docs/index-server.md
+++ b/docs/index-server.md
@@ -191,7 +191,7 @@ that will authenticate the user to access the index server and no other service.
## Starting the Server
```
-./bin/spark-submit --master [yarn/local] --[o ptional parameters] --class org.apache.carbondata.indexserver.IndexServer [path to carbondata-spark2-<version>.jar]
+./bin/spark-submit --master [yarn/local] --[optional parameters] --class org.apache.carbondata.indexserver.IndexServer [path to carbondata-spark2-<version>.jar]
```
Or
```