You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@carbondata.apache.org by ra...@apache.org on 2019/01/21 11:56:23 UTC
[carbondata] branch branch-1.5 updated (951a781 -> e14033c)
This is an automated email from the ASF dual-hosted git repository.
ravipesala pushed a change to branch branch-1.5
in repository https://gitbox.apache.org/repos/asf/carbondata.git.
from 951a781 [CARBONDATA-3175]Fix Testcase failures in complex delimiters
new 27d4713 [CARBONDATA-3118] support parallel block pruning for non-default datamaps
new 85fc4db [CARBONDATA-3073] Support configure TableProperties,withLoadOption etc. interface in carbon writer of C++ SDK support withTableProperty, withLoadOption,taskNo, uniqueIdentifier, withThreadSafe,withBlockSize, withBlockletSize, localDictionaryThreshold, enableLocalDictionary, sortBy in C++ SDK
new 1a5aecf [CARBONDATA-3119] Fixed SDK Write for Complex Array Type when Array is Empty
new c501669 [HOTFIX] Fixed Query performance issue
new c6e2ad4 [CARBONDATA-3160] Compaction support with MAP data type
new 5ebb657 [CARBONDATA-3174]varchar column trailing space issue fixed
new 3ae9b53 [CARBONDATA-3162][CARBONDATA-3163][CARBONDATA-3164] 'no_sort' as default sort_scope
new 3d62b87 [CARBONDATA-2999] support read schema from S3
new 3abb174 [CARBONDATA-3182] Fixed SDV Testcase failures
new e3eaa75 [HOTFIX] replace apache common log with carbondata log4j
new 941d35f [SDV] Add datasource testcases for Spark File Format
new a423415 [CARBONDATA-3181][BloomDataMap] Fix access field error for BitSet in bloom filter
new 04e4d91 [CARBONDATA-3187] Supported Global Dictionary For Map
new a1be8f9 [CARBONDATA-3161]Pipe dilimiter is not working for streaming table
new 27d66da [CARBONDATA-3149]Support alter table column rename
new 574d469 [CARBONDATA-3179] Map Data Load Failure and Struct Projection Pushdown Issue
new dd667ba [CARBONDATA-3127]Fix the HiveExample & TestCarbonSerde exception
new afd8f4e [CARBONDATA-3186]Avoid creating empty carbondata file when all the records are bad record with action redirect.
new 58c69d7 [CARBONDATA-3192] Fix for compaction compatibilty issue
new 577528f [CARBONDATA-3184]Fix DataLoad Failure with 'using carbondata'
new a672e43 [CARBONDATA-3080] Supporting local dictionary enable by default for SDK
new 15717fa [CARBONDATA-3142]Add timestamp with thread name which created by CarbonThreadFactory
new b52387e [CARBONDATA-3176] Optimize quick-start-guide documentation
new 1c46415 [CARBONDATA-3188] Create carbon table as hive understandable metastore table needed by Presto and Hive
new 2019c3d [CARBONDATA-3165]Protection of Bloom Null Exception
new 44ec43a [HOTFIX] rename field thread_pool_size to match camel case
new 679ef3b [CARBONDATA-2218] AlluxioCarbonFile while trying to force rename causes a FileSytem error and is not a DistributionFileSystem.(Adding Alluxio Support)
new 7196118 [Carbondata-3173] Add the hive/presto documents index to the root of file ReadMe.md
new cd5c33c [CARBONDATA-3195]Added validation for Inverted Index columns and added a test case in case of varchar
new 0bcc84c [CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction for Complex types with Dictionary Include and also supported Compaction for restructured table
new 9ec889f [CARBONDATA-3194] Integrating Carbon with Presto using hive connector
new 6379f1c [CARBONDATA-3126] Correct some spell errors in CarbonData
new b649f3c [CARBONDATA-3202]update the schema to session catalog after add column, drop column and column rename
new 8530f45 [CARBONDATA-3208] Remove unused parameters, imports and optimize the spell errors
new 19733b4 [CARBONDATA-3209] Remove unused import
new 3ca5af8 [CARBONDATA-3213] Add license for java and md files
new 6b723b0 [CARBONDATA-3206] Fix some spell errors in CarbonData
new 35a82ef [CARBONDATA-3205]Fix Get Local Dictionary for empty Array of String
new 24bf76e [CARBONDATA-3218] Fix schema refresh and wrong query result issues in presto.
new e8c70af [CARBONDATA-3212] Fixed NegativeArraySizeException while querying in specific scenario
new 74cb116 [HOTFIX] Optimize the code style in csdk/sdk markdown doc
new 7bc5c13 [CARBONDATA-3216] Fix enableLocalDictionary with false issue in CSDK
new c1116c4 [CARBONDATA-3217] Optimize implicit filter expression performance by removing extra serialization
new 418084b [CARBONDATA-3149] Documentation for alter table column rename
new 59eca1e [CARBONDATA-3226] Remove duplicated and useless files
new 083228d [CARBONDATA-3223] Fixed Wrong Datasize and Indexsize calculation for old store using Show Segments
new 05ad0c9 [CARBONDATA-3189] Fix PreAggregate Datamap Issue
new cf62d0f [CARBONDATA-3222]Fix dataload failure after creation of preaggregate datamap on main table with long_string_columns
new 48b0584 [CARBONDATA-3224] Support SDK validate the improper value when using withLoadOptions
new cd25019 [CARBONDATA-3227] Fix some spell errors in the project
new 310c67d [CARBONDATA-3211] Optimize the documentation
new 8f81507 [CARBONDATA-3219] Support range partition the input data for local_sort/global sort data loading
new 13c18f0 [CARBONDATA-3221] Fix the error of SDK don't support read multiple file from S3
new 5ba9428 [CARBONDATA-3230] Add alter test case for datasource
new 6b0c067 [CARBONDATA-3201] Added load level SORT_SCOPE Added SORT_SCOPE in Load Options & in SET Command
new a40c6f1 [CARBONDATA-3210] Merge common method into CarbonSparkUtil and fix example error
new 8eebda6 [CARBONDATA-3235] Fix Rename-Fail & Datamap-creation-Fail
new 17210cd [CARBONDATA-3236] Fix for JVM Crash for insert into new table from old table
new 464556e [CARBONDATA-3200] No-Sort compaction
new b009373 [CARBONDATA-3237] Fix presto carbon issues in dictionary include scenario
new 20c675c [CARBONDATA-3220] Support presto to read stream segment data
new 13b94e0 [CARBONDATA-3226] Remove duplicated and useless files
new 7c3c442 [CARBONDATA-3241] Refactor the requested scan columns and the projection columns
new 8416178 [HOTFIX] Fix compile error after merging PR#3001
new 27e6c3e [HOTFIX] Exclude filter doesn't work in presto carbon in cluster
new 78c72ea [CARBONDATA-3239] Fix ArrayIndexOutOfBoundsException in DataSkewRangePartitioner
new 2cb130d [CARBONDATA-3233]Fix JVM crash issue in snappy compressor and update the pagesize correctly
new e559a2b [HOTFIX][DataLoad]fix task assignment issue using NODE_MIN_SIZE_FIRST block assignment strategy
new 6147833 [CARBONDATA-3215] Optimize the documentation
new d676619 [CARBONDATA-3251] Fix spark-2.1 UT errors
new c6fb607 [CARBONDATA-3238] Solve StackOverflowError using MV datamap
new f111907 [HOTFIX] Fixed NPE during query with Local Dictionary
new 7c13c52 [CARBONDATA-3246]Fix sdk reader issue if batch size is given as zero and vectorRead False
new e9242b7 [CARBONDATA-3253] Remove test case of bloom datamap using search mode
new 794369e Optimize presto-guide
new 142fb12 [CARBONDATA-3242] Move Range_Column into the table level properties
new 5517f5c [SDV] Fix SDV testcase failures
new e14033c [CARBONDATA-3261] support float and byte data type reading from presto carbon
The 78 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
README.md | 4 +
.../java/org/apache/carbondata/common/Strings.java | 6 +-
.../org/apache/carbondata/common/StringsSuite.java | 2 +-
.../cache/dictionary/AbstractDictionaryCache.java | 4 +-
.../cache/dictionary/ForwardDictionaryCache.java | 2 +-
.../cache/dictionary/ReverseDictionaryCache.java | 2 +-
.../core/constants/CarbonCommonConstants.java | 26 +-
.../core/constants/CarbonLoadOptionConstants.java | 7 +
.../core/constants/SortScopeOptions.java | 2 -
.../carbondata/core/datamap/TableDataMap.java | 62 +-
.../carbondata/core/datamap/dev/DataMap.java | 7 +
.../datamap/dev/cgdatamap/CoarseGrainDataMap.java | 5 +
.../datamap/dev/fgdatamap/FineGrainDataMap.java | 5 +
.../core/datastore/block/SegmentProperties.java | 20 +-
.../core/datastore/block/TableBlockInfo.java | 11 +
.../chunk/impl/DimensionRawColumnChunk.java | 6 +-
.../datastore/filesystem/AlluxioCarbonFile.java | 87 +-
.../core/datastore/filesystem/CarbonFile.java | 2 +-
.../core/datastore/filesystem/LocalCarbonFile.java | 2 +-
.../core/datastore/impl/FileFactory.java | 30 +-
.../carbondata/core/datastore/page/ColumnPage.java | 6 +-
.../core/datastore/page/LocalDictColumnPage.java | 3 +
.../datastore/page/UnsafeFixLengthColumnPage.java | 29 +-
.../core/datastore/row/WriteStepRowUtil.java | 4 +-
.../carbondata/core/features/TableOperation.java | 1 +
.../fileoperations/AtomicFileOperationFactory.java | 2 +-
.../indexstore/blockletindex/BlockDataMap.java | 17 +-
.../indexstore/blockletindex/BlockletDataMap.java | 1 +
.../blockletindex/BlockletDataMapFactory.java | 3 -
.../{S3FileLock.java => AlluxioFileLock.java} | 53 +-
.../carbondata/core/locks/CarbonLockFactory.java | 21 +-
.../core/metadata/blocklet/BlockletInfo.java | 19 +
.../core/metadata/blocklet/DataFileFooter.java | 13 +
.../ThriftWrapperSchemaConverterImpl.java | 1 +
.../core/metadata/schema/table/CarbonTable.java | 12 +-
.../impl/DictionaryBasedResultCollector.java | 8 +-
.../impl/DictionaryBasedVectorResultCollector.java | 11 +-
.../core/scan/complextypes/ArrayQueryType.java | 14 +-
.../core/scan/complextypes/ComplexQueryType.java | 6 +-
.../core/scan/complextypes/MapQueryType.java | 4 +-
.../core/scan/complextypes/PrimitiveQueryType.java | 16 +-
.../core/scan/complextypes/StructQueryType.java | 18 +-
.../scan/executor/impl/AbstractQueryExecutor.java | 5 +
.../expression/conditional/ImplicitExpression.java | 109 ++
.../core/scan/filter/ColumnFilterInfo.java | 43 +-
.../carbondata/core/scan/filter/FilterUtil.java | 76 +-
.../core/scan/filter/GenericQueryType.java | 4 +-
.../ImplicitIncludeFilterExecutorImpl.java | 23 +-
.../core/scan/filter/intf/ExpressionType.java | 3 +-
.../visitor/ImplicitColumnVisitor.java | 24 +-
.../core/scan/partition/PartitionUtil.java | 2 +-
.../core/scan/result/BlockletScannedResult.java | 6 +
.../AbstractDetailQueryResultIterator.java | 3 +
.../core/scan/wrappers/ByteArrayWrapper.java | 4 +
.../core/statusmanager/LoadMetadataDetails.java | 2 +-
.../core/statusmanager/SegmentStatusManager.java | 2 +-
.../core/util/AbstractDataFileFooterConverter.java | 7 +
.../carbondata/core/util/BlockletDataMapUtil.java | 7 +-
.../carbondata/core/util/CarbonProperties.java | 32 +-
.../carbondata/core/util/CarbonThreadFactory.java | 13 +-
.../apache/carbondata/core/util/CarbonUtil.java | 2 +-
.../core/util/DataFileFooterConverterV3.java | 5 +
.../apache/carbondata/core/util/DataTypeUtil.java | 8 +
.../core/util/ObjectSerializationUtil.java | 8 +-
.../apache/carbondata/core/util/SessionParams.java | 8 +-
.../carbondata/core/util/ThreadLocalTaskInfo.java | 7 +
.../carbondata/core/util/path/CarbonTablePath.java | 4 +-
.../core/CarbonPropertiesValidationTest.java | 2 +-
.../filesystem/AlluxioCarbonFileTest.java | 214 ++-
.../blockletindex/TestBlockletDataMap.java | 17 +
.../core/scan/complextypes/ArrayQueryTypeTest.java | 2 +-
.../scan/complextypes/StructQueryTypeTest.java | 2 +-
.../scan/filter/FilterExpressionProcessorTest.java | 17 +
.../carbondata/core/util/DataTypeUtilTest.java | 1 -
.../datamap/bloom/BloomCoarseGrainDataMap.java | 10 +
.../bloom/BloomCoarseGrainDataMapFactory.java | 7 +-
.../hadoop/util/bloom/CarbonBloomFilter.java | 20 +-
.../datamap/examples/MinMaxIndexDataMap.java | 4 +
.../datamap/lucene/LuceneDataMapWriter.java | 2 +-
.../lucene/LuceneFineGrainDataMapFactory.java | 6 +-
.../apache/carbondata/mv/datamap/MVHelper.scala | 12 +-
.../carbondata/mv/rewrite/DefaultMatchMaker.scala | 20 +-
.../carbondata/mv/rewrite/MVCreateTestCase.scala | 25 +
.../mv/plans/util/BirdcageOptimizer.scala | 4 +-
docs/configuration-parameters.md | 3 +-
docs/csdk-guide.md | 596 +++++----
docs/datamap-developer-guide.md | 25 +-
docs/datamap/bloomfilter-datamap-guide.md | 40 +-
docs/datamap/datamap-management.md | 14 +-
docs/datamap/timeseries-datamap-guide.md | 43 +-
docs/ddl-of-carbondata.md | 348 ++---
docs/documentation.md | 2 +-
docs/faq.md | 33 +-
docs/hive-guide.md | 8 +-
docs/introduction.md | 17 +
.../presto/README.md => docs/presto-guide.md | 153 ++-
docs/quick-start-guide.md | 228 ++--
docs/s3-guide.md | 9 +-
docs/sdk-guide.md | 949 +++++++-------
docs/streaming-guide.md | 14 +-
docs/usecases.md | 17 +
examples/flink/pom.xml | 2 +-
examples/spark2/pom.xml | 14 +-
.../examples/sdk/CarbonReaderExample.java | 239 ++--
.../carbondata/examples/sdk/SDKS3Example.java | 56 +-
.../carbondata/examples/sdk/SDKS3ReadExample.java | 15 +-
.../examples/sdk/SDKS3SchemaReadExample.java | 69 +
.../spark2/src/main/resources/sample.csv | 0
.../apache/carbondata/examples}/HiveExample.scala | 100 +-
.../org/apache/carbondata/examples/S3Example.scala | 34 +-
.../carbondata/examples/S3UsingSDkExample.scala | 51 +-
.../carbondata/examples/SparkSessionExample.scala | 6 +-
.../carbondata/examples/util/ExampleUtils.scala | 9 +-
.../apache/carbondata/examplesCI/RunExamples.scala | 8 +-
format/src/main/thrift/carbondata_index.thrift | 1 +
.../apache/carbondata/hadoop/CarbonInputSplit.java | 32 +-
.../hadoop/api/CarbonFileInputFormat.java | 2 +-
.../carbondata/hadoop/api/CarbonInputFormat.java | 78 +-
.../hadoop/api/CarbonTableInputFormat.java | 9 +-
.../hadoop/api/CarbonTableOutputFormat.java | 16 +-
.../hadoop/stream}/CarbonStreamInputFormat.java | 5 +-
.../hadoop/stream}/CarbonStreamUtils.java | 2 +-
.../hadoop/stream}/StreamBlockletReader.java | 2 +-
.../hadoop/stream/StreamRecordReader.java | 212 +--
.../carbondata/hadoop/testutil/StoreCreator.java | 2 +
.../hadoop/util/CarbonVectorizedRecordReader.java | 63 +-
.../carbondata/hive/CarbonHiveInputSplit.java | 35 +-
.../carbondata/hive/MapredCarbonInputFormat.java | 15 +-
.../hive/server/HiveEmbeddedServer2.java | 7 +-
.../apache/carbondata/hive/TestCarbonSerDe.java | 137 ++
.../apache/carbondata/hive/TestCarbonSerde.java | 133 --
.../hiveexampleCI/RunHiveExampleTest.scala | 43 -
.../presto/Presto_Cluster_Setup_For_Carbondata.md | 135 --
integration/presto/pom.xml | 23 +-
.../carbondata/presto/CarbonVectorBatch.java | 24 +-
.../carbondata/presto/CarbondataColumnHandle.java | 143 --
.../carbondata/presto/CarbondataConnector.java | 78 --
.../presto/CarbondataConnectorFactory.java | 157 ++-
.../carbondata/presto/CarbondataConnectorId.java | 52 -
.../carbondata/presto/CarbondataErrorCode.java | 38 -
.../presto/CarbondataHandleResolver.java | 43 -
.../carbondata/presto/CarbondataMetadata.java | 272 ----
.../apache/carbondata/presto/CarbondataModule.java | 153 ++-
.../carbondata/presto/CarbondataPageSource.java | 318 ++++-
.../presto/CarbondataPageSourceProvider.java | 209 +--
.../apache/carbondata/presto/CarbondataSplit.java | 102 --
.../carbondata/presto/CarbondataSplitManager.java | 151 ++-
.../carbondata/presto/CarbondataTableHandle.java | 71 -
.../presto/CarbondataTableLayoutHandle.java | 71 -
.../presto/CarbondataTransactionHandle.java | 65 -
.../presto/PrestoCarbonVectorizedRecordReader.java | 14 +-
.../apache/carbondata/presto/PrestoFilterUtil.java | 126 +-
.../presto/impl/CarbonLocalInputSplit.java | 44 +-
.../presto/impl/CarbonLocalMultiBlockSplit.java | 16 +-
.../presto/impl/CarbonTableCacheModel.java | 29 +-
.../carbondata/presto/impl/CarbonTableConfig.java | 35 -
.../carbondata/presto/impl/CarbonTableReader.java | 326 +----
.../presto/readers/BooleanStreamReader.java | 17 +
...leanStreamReader.java => ByteStreamReader.java} | 48 +-
.../presto/readers/DecimalSliceStreamReader.java | 33 +-
.../presto/readers/DoubleStreamReader.java | 24 +
...ortStreamReader.java => FloatStreamReader.java} | 46 +-
.../presto/readers/IntegerStreamReader.java | 19 +
.../presto/readers/LongStreamReader.java | 24 +
.../presto/readers/ObjectStreamReader.java | 6 +
.../presto/readers/ShortStreamReader.java | 24 +
.../presto/readers/SliceStreamReader.java | 80 +-
.../presto/readers/TimestampStreamReader.java | 24 +
.../presto/CarbonDictionaryDecodeReadSupport.scala | 22 +-
.../PrestoAllDataTypeLocalDictTest.scala | 16 +-
.../integrationtest/PrestoAllDataTypeTest.scala | 14 +-
.../PrestoTestNonTransactionalTableFiles.scala | 84 +-
.../carbondata/presto/server/PrestoServer.scala | 44 +-
.../presto/util/CarbonDataStoreCreator.scala | 3 +-
integration/spark-common-cluster-test/pom.xml | 6 +
.../CarbonV1toV3CompatabilityTestCase.scala | 17 +-
.../cluster/sdv/generated/ColumndictTestCase.scala | 4 +-
.../sdv/generated/DataLoadingTestCase.scala | 2 +-
.../cluster/sdv/generated/SinglepassTestCase.scala | 32 +-
...teTableUsingSparkCarbonFileFormatTestCase.scala | 336 +++++
.../datasource/SparkCarbonDataSourceTestCase.scala | 1380 ++++++++++++++++++++
.../carbondata/cluster/sdv/suite/SDVSuites.scala | 5 +-
.../apache/spark/sql/common/util/QueryTest.scala | 7 +-
.../src/test/resources/range_column/dataskew.csv | 20 +
.../src/test/resources/structofarray.csv | 10 +
.../lucene/LuceneFineGrainDataMapSuite.scala | 7 +-
...eEncodingSafeColumnPageForComplexDataType.scala | 6 -
...ncodingUnsafeColumnPageForComplexDataType.scala | 12 +-
.../complexType/TestCompactionComplexType.scala | 146 ++-
.../complexType/TestComplexDataType.scala | 37 +-
.../complexType/TestComplexTypeQuery.scala | 20 +-
.../testsuite/dataload/TestLoadDataGeneral.scala | 9 +-
.../TestLoadDataWithSortColumnBounds.scala | 12 +-
.../dataload/TestNoInvertedIndexLoadAndQuery.scala | 8 +-
.../preaggregate/TestPreAggCreateCommand.scala | 54 +-
.../badrecordloger/BadRecordLoggerTest.scala | 57 +-
.../compaction/TestHybridCompaction.scala | 262 ++++
.../TestLoadTableConcurrentScenario.scala | 0
.../TestCreateDDLForComplexMapType.scala | 150 ++-
.../TestCreateHiveTableWithCarbonDS.scala | 86 ++
.../createTable/TestCreateTableWithSortScope.scala | 29 +-
.../TestNonTransactionalCarbonTable.scala | 55 +-
.../CompactionSupportGlobalSortFunctionTest.scala | 1 +
.../CompactionSupportGlobalSortParameterTest.scala | 1 +
.../testsuite/dataload/TestBatchSortDataLoad.scala | 9 +-
.../dataload/TestGlobalSortDataLoad.scala | 18 +-
.../testsuite/dataload/TestLoadDataFrame.scala | 3 +-
.../TestLoadDataWithHiveSyntaxDefaultFormat.scala | 6 +-
...adDataWithMalformedCarbonCommandException.scala | 8 +-
.../dataload/TestRangeColumnDataLoad.scala | 266 ++++
.../testsuite/datamap/CGDataMapTestCase.scala | 2 +
.../testsuite/datamap/FGDataMapTestCase.scala | 2 +
.../testsuite/datamap/TestDataMapCommand.scala | 2 +-
.../testsuite/datamap/TestDataMapStatus.scala | 12 +-
.../ValueCompressionDataTypeTestCase.scala | 22 +-
.../filterexpr/TestImplicitFilterExpression.scala | 117 ++
.../iud/TestInsertAndOtherCommandConcurrent.scala | 4 +-
.../testsuite/iud/UpdateCarbonTableTestCase.scala | 3 +-
.../longstring/VarcharDataTypesBasicTestCase.scala | 46 +-
.../testsuite/sortcolumns/TestSortColumns.scala | 3 +
.../sortcolumns/TestSortColumnsWithUnsafe.scala | 4 +
.../StandardPartitionTableQueryTestCase.scala | 10 +-
.../org/apache/carbondata/api/CarbonStore.scala | 12 +-
.../carbondata/events/AlterTableEvents.scala | 5 +-
.../carbondata/spark/CarbonSparkFactory.scala | 2 +-
.../apache/carbondata/spark/PartitionFactory.scala | 2 +-
.../carbondata/spark/load/CsvRDDHelper.scala | 3 +
.../spark/load/DataLoadProcessBuilderOnSpark.scala | 209 ++-
.../spark/load/DataLoadProcessorStepOnSpark.scala | 183 ++-
.../carbondata/spark/load/GlobalSortHelper.scala | 2 -
.../spark/rdd/AlterTableLoadPartitionRDD.scala | 6 +-
.../spark/rdd/CarbonDropPartitionRDD.scala | 2 +-
.../spark/rdd/CarbonGlobalDictionaryRDD.scala | 18 +-
.../carbondata/spark/rdd/CarbonMergeFilesRDD.scala | 0
.../carbondata/spark/rdd/CarbonMergerRDD.scala | 53 +-
.../carbondata/spark/rdd/CarbonScanRDD.scala | 33 +-
.../spark/rdd/InsertTaskCompletionListener.scala | 7 +-
.../spark/rdd/NewCarbonDataLoadRDD.scala | 30 +-
.../spark/rdd/QueryTaskCompletionListener.scala | 4 +-
.../carbondata/spark/rdd/StreamHandoffRDD.scala | 6 +-
.../spark/tasks/SortIndexWriterTask.scala | 1 -
.../carbondata/spark/util/CarbonScalaUtil.scala | 55 +-
.../apache/carbondata/spark/util/CommonUtil.scala | 3 +-
.../spark/util/GlobalDictionaryUtil.scala | 13 +-
.../streaming/CarbonStreamingQueryListener.scala | 1 -
.../carbondata/streaming/StreamSinkFactory.scala | 2 +-
.../apache/spark/DataSkewRangePartitioner.scala | 368 ++++++
.../spark/sql/catalyst/CarbonDDLSqlParser.scala | 133 +-
.../command/carbonTableSchemaCommon.scala | 22 +-
.../streaming/CarbonAppendableStreamSink.scala | 2 +
.../apache/spark/sql/test/TestQueryExecutor.scala | 2 +-
.../org/apache/spark/sql/test/util/QueryTest.scala | 4 +-
.../apache/spark/util/CarbonReflectionUtils.scala | 32 +
.../datasources/CarbonSparkDataSourceUtil.scala | 2 +-
.../datasources/SparkCarbonFileFormat.scala | 13 +-
.../CarbonTaskCompletionListener.scala | 8 +-
.../org/apache/spark/sql/CarbonVectorProxy.java | 1 +
.../src/test/resources/test_json.json | 1 +
.../datasource/SparkCarbonDataSourceTest.scala | 353 ++++-
...TestCreateTableUsingSparkCarbonFileFormat.scala | 24 +-
.../spark/sql/carbondata/datasource/TestUtil.scala | 2 +-
integration/spark2/pom.xml | 1 -
.../sql/hive/CarbonInMemorySessionState.scala | 15 +-
.../apache/spark/sql/hive/CarbonSessionState.scala | 55 +-
.../apache/spark/sql/hive/CarbonSessionUtil.scala | 55 +-
.../spark/sql/hive/SqlAstBuilderHelper.scala | 24 +-
.../spark/rdd/CarbonDataRDDFactory.scala | 33 +-
.../spark/thriftserver/CarbonThriftServer.scala | 37 +-
.../carbondata/spark/util/CarbonSparkUtil.scala | 40 +-
.../stream/CarbonStreamRecordReader.java | 684 +---------
.../org/apache/spark/sql/CarbonCountStar.scala | 2 +-
.../spark/sql/CarbonDatasourceHadoopRelation.scala | 37 +-
.../scala/org/apache/spark/sql/CarbonEnv.scala | 3 +-
.../scala/org/apache/spark/sql/CarbonSession.scala | 3 +-
.../scala/org/apache/spark/sql/CarbonSource.scala | 3 +-
.../sql/execution/command/carbonTableSchema.scala | 0
.../command/datamap/CarbonDropDataMapCommand.scala | 2 +-
.../CarbonAlterTableCompactionCommand.scala | 9 +-
.../CarbonAlterTableFinishStreaming.scala | 2 +-
.../command/management/CarbonLoadDataCommand.scala | 100 +-
.../management/CarbonShowLoadsCommand.scala | 2 +-
.../command/mutation/DeleteExecution.scala | 2 +-
.../spark/sql/execution/command/package.scala | 7 +-
.../CarbonAlterTableDropHivePartitionCommand.scala | 3 +-
.../preaaggregate/PreAggregateListeners.scala | 20 +-
.../preaaggregate/PreAggregateTableHelper.scala | 27 +-
.../command/preaaggregate/PreAggregateUtil.scala | 10 +-
.../schema/CarbonAlterTableAddColumnCommand.scala | 12 +-
...nAlterTableColRenameDataTypeChangeCommand.scala | 336 +++++
.../CarbonAlterTableDataTypeChangeCommand.scala | 180 ---
.../schema/CarbonAlterTableDropColumnCommand.scala | 9 +-
.../schema/CarbonAlterTableRenameCommand.scala | 10 +-
.../command/table/CarbonCreateTableCommand.scala | 9 +-
.../datasources/SparkCarbonTableFormat.scala | 4 -
.../strategy/CarbonLateDecodeStrategy.scala | 35 +-
.../spark/sql/execution/strategy/DDLStrategy.scala | 14 +-
.../strategy/StreamingTableStrategy.scala | 12 +-
.../apache/spark/sql/hive/CarbonMetaStore.scala | 2 +-
.../spark/sql/hive/CarbonPreAggregateRules.scala | 38 +-
.../spark/sql/hive/CarbonSessionCatalog.scala | 26 +-
.../execution/command/CarbonHiveCommands.scala | 19 +-
.../apache/spark/sql/optimizer/CarbonFilters.scala | 15 +-
.../spark/sql/optimizer/CarbonLateDecodeRule.scala | 27 +-
.../spark/sql/parser/CarbonSpark2SqlParser.scala | 28 +-
.../spark/sql/parser/CarbonSparkSqlParser.scala | 1 -
.../spark/sql/test/Spark2TestQueryExecutor.scala | 4 +-
.../org/apache/spark/util/AlterTableUtil.scala | 92 +-
.../apache/spark/sql/CarbonToSparkAdapater.scala | 11 +-
.../apache/spark/sql/hive/CarbonSessionState.scala | 31 +-
...rkAdapater.scala => CarbonToSparkAdapter.scala} | 12 +-
.../apache/spark/sql/CarbonToSparkAdapater.scala | 11 +-
.../stream/CarbonStreamRecordReaderTest.java | 2 +-
.../bloom/BloomCoarseGrainDataMapSuite.scala | 53 +-
.../bloom/BloomCoarseGrainDataMapTestUtil.scala | 2 +-
.../InsertIntoCarbonTableSpark2TestCase.scala | 3 -
.../spark/util/AllDictionaryTestCase.scala | 20 +-
.../spark/util/DictionaryLRUCacheTestCase.scala | 2 -
.../util/ExternalColumnDictionaryTestCase.scala | 4 +-
.../carbondata/BadRecordPathLoadOptionTest.scala | 11 +-
.../carbondata/TestStreamingTableOpName.scala | 4 +
.../carbondata/commands/SetCommandTestCase.scala | 28 +
.../datatype/NumericDimensionBadRecordTest.scala | 2 -
.../iud/DeleteCarbonTableSubqueryTestCase.scala | 3 -
.../restructure/AlterTableRevertTestCase.scala | 1 -
.../restructure/AlterTableValidationTestCase.scala | 3 +-
.../vectorreader/AddColumnTestCases.scala | 4 +-
.../AlterTableColumnRenameTestCase.scala | 356 +++++
.../vectorreader/ChangeDataTypeTestCases.scala | 2 +-
.../spark/sql/common/util/Spark2QueryTest.scala | 3 +-
.../org/apache/spark/util/CarbonCommandSuite.scala | 1 -
pom.xml | 4 +-
.../processing/datatypes/ArrayDataType.java | 20 +-
.../processing/datatypes/GenericDataType.java | 2 +-
.../processing/datatypes/PrimitiveDataType.java | 22 +-
.../processing/datatypes/StructDataType.java | 22 +-
.../loading/CarbonDataLoadConfiguration.java | 32 +-
.../converter/impl/MeasureFieldConverterImpl.java | 8 +-
.../impl/NonDictionaryFieldConverterImpl.java | 8 +-
.../loading/csvinput/CSVInputFormat.java | 10 +
.../processing/loading/events/LoadEvents.java | 13 +-
.../iterator/CarbonOutputIteratorWrapper.java | 7 +-
.../processing/loading/model/CarbonLoadModel.java | 41 +
.../loading/model/CarbonLoadModelBuilder.java | 48 +-
.../processing/loading/model/LoadOption.java | 5 +-
.../loading/parser/impl/ArrayParserImpl.java | 4 +
.../loading/parser/impl/RangeColumnParserImpl.java | 56 +
.../CarbonRowDataWriterProcessorStepImpl.java | 8 +-
.../steps/DataWriterBatchProcessorStepImpl.java | 6 +-
.../loading/steps/DataWriterProcessorStepImpl.java | 6 +-
.../processing/merger/AbstractResultProcessor.java | 6 +-
.../merger/CarbonCompactionExecutor.java | 35 +-
.../processing/merger/CarbonCompactionUtil.java | 88 +-
.../processing/merger/CarbonDataMergerUtil.java | 2 +-
.../merger/CompactionResultSortProcessor.java | 37 +-
.../merger/RowResultMergerProcessor.java | 11 +-
.../sort/sortdata/InMemorySortTempChunkHolder.java | 147 +++
.../sortdata/SingleThreadFinalSortFilesMerger.java | 52 +-
.../processing/sort/sortdata/SortParameters.java | 4 +-
.../sort/sortdata/SortTempFileChunkHolder.java | 18 +-
.../processing/sort/sortdata/TableFieldStat.java | 34 +-
.../store/CarbonFactDataHandlerColumnar.java | 8 +-
.../store/CarbonFactDataHandlerModel.java | 151 +--
.../processing/store/CarbonFactHandlerFactory.java | 1 -
.../carbondata/processing/store/TablePage.java | 6 +-
.../store/writer/AbstractFactDataWriter.java | 48 +-
.../writer/v3/CarbonFactDataWriterImplV3.java | 5 +-
.../processing/util/CarbonDataProcessorUtil.java | 40 +-
.../processing/util/CarbonLoaderUtil.java | 18 +-
store/CSDK/CMakeLists.txt | 8 +-
store/CSDK/src/CarbonReader.h | 4 +-
store/CSDK/src/CarbonSchemaReader.cpp | 22 +-
store/CSDK/src/CarbonSchemaReader.h | 26 +
store/CSDK/src/CarbonWriter.cpp | 208 +++
store/CSDK/src/CarbonWriter.h | 105 ++
store/CSDK/src/Configuration.cpp | 101 ++
store/CSDK/src/Configuration.h | 85 ++
store/CSDK/src/Schema.h | 5 -
store/CSDK/test/main.cpp | 565 +++++---
.../carbondata/sdk/file/CarbonSchemaReader.java | 84 +-
.../carbondata/sdk/file/CarbonWriterBuilder.java | 113 +-
.../java/org/apache/carbondata/sdk/file/Field.java | 15 +-
.../carbondata/sdk/file/CSVCarbonWriterTest.java | 38 +
.../carbondata/sdk/file/CarbonReaderTest.java | 351 +++--
.../sdk/file/ConcurrentSdkReaderTest.java | 39 +
.../streaming/parser/FieldConverter.scala | 44 +-
.../streaming/parser/RowStreamParserImp.scala | 15 +-
.../org/apache/carbondata/tool/CarbonCliTest.java | 69 +-
387 files changed, 12728 insertions(+), 6192 deletions(-)
copy core/src/main/java/org/apache/carbondata/core/locks/{S3FileLock.java => AlluxioFileLock.java} (54%)
create mode 100644 core/src/main/java/org/apache/carbondata/core/scan/expression/conditional/ImplicitExpression.java
rename integration/presto/README.md => docs/presto-guide.md (57%)
create mode 100644 examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3SchemaReadExample.java
rename integration/hive/src/main/resources/data.csv => examples/spark2/src/main/resources/sample.csv (100%)
rename {integration/hive/src/main/scala/org/apache/carbondata/hiveexample => examples/spark2/src/main/scala/org/apache/carbondata/examples}/HiveExample.scala (71%)
rename {streaming/src/main/java/org/apache/carbondata/streaming => hadoop/src/main/java/org/apache/carbondata/hadoop/stream}/CarbonStreamInputFormat.java (97%)
rename {streaming/src/main/java/org/apache/carbondata/streaming => hadoop/src/main/java/org/apache/carbondata/hadoop/stream}/CarbonStreamUtils.java (97%)
rename {streaming/src/main/java/org/apache/carbondata/streaming => hadoop/src/main/java/org/apache/carbondata/hadoop/stream}/StreamBlockletReader.java (99%)
copy integration/spark2/src/main/scala/org/apache/carbondata/stream/CarbonStreamRecordReader.java => hadoop/src/main/java/org/apache/carbondata/hadoop/stream/StreamRecordReader.java (78%)
create mode 100644 integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java
delete mode 100644 integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerde.java
delete mode 100644 integration/hive/src/test/scala/org/apache/carbondata/hiveexampleCI/RunHiveExampleTest.scala
delete mode 100644 integration/presto/Presto_Cluster_Setup_For_Carbondata.md
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataColumnHandle.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataConnector.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataConnectorId.java
delete mode 100644 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataErrorCode.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataHandleResolver.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataMetadata.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataSplit.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataTableHandle.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataTableLayoutHandle.java
delete mode 100755 integration/presto/src/main/java/org/apache/carbondata/presto/CarbondataTransactionHandle.java
copy integration/presto/src/main/java/org/apache/carbondata/presto/readers/{BooleanStreamReader.java => ByteStreamReader.java} (70%)
copy integration/presto/src/main/java/org/apache/carbondata/presto/readers/{ShortStreamReader.java => FloatStreamReader.java} (64%)
create mode 100644 integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/datasource/CreateTableUsingSparkCarbonFileFormatTestCase.scala
create mode 100644 integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/datasource/SparkCarbonDataSourceTestCase.scala
create mode 100644 integration/spark-common-test/src/test/resources/range_column/dataskew.csv
create mode 100644 integration/spark-common-test/src/test/resources/structofarray.csv
create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/compaction/TestHybridCompaction.scala
delete mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/concurrent/TestLoadTableConcurrentScenario.scala
create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateHiveTableWithCarbonDS.scala
create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestRangeColumnDataLoad.scala
create mode 100644 integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/filterexpr/TestImplicitFilterExpression.scala
delete mode 100644 integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergeFilesRDD.scala
create mode 100644 integration/spark-common/src/main/scala/org/apache/spark/DataSkewRangePartitioner.scala
create mode 100644 integration/spark-datasource/src/test/resources/test_json.json
delete mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala
create mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
delete mode 100644 integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableDataTypeChangeCommand.scala
rename integration/spark2/src/main/spark2.2/org/apache/spark/sql/{CarbonToSparkAdapater.scala => CarbonToSparkAdapter.scala} (89%)
create mode 100644 integration/spark2/src/test/scala/org/apache/spark/carbondata/restructure/vectorreader/AlterTableColumnRenameTestCase.scala
create mode 100644 processing/src/main/java/org/apache/carbondata/processing/loading/parser/impl/RangeColumnParserImpl.java
create mode 100644 processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/InMemorySortTempChunkHolder.java
create mode 100644 store/CSDK/src/Configuration.cpp
create mode 100644 store/CSDK/src/Configuration.h