You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2019/12/27 14:04:07 UTC
[GitHub] [carbondata] ajantha-bhat opened a new pull request #3538: [WIP]
Separate Insert and load to later optimize insert.
ajantha-bhat opened a new pull request #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538
[WIP] Separate Insert and load to later optimize insert.
1. separated load and insert command; load command is used only for load DDL, rest of the flow uses insert command
2. remove update and dataframe argument from load command
3. currently code is duplicated for separation. later once insert flow is optimized, common code will be extracted
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585021232
@jackylk , @kunal642 : handled the comments please check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-579763503
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1795/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581336230
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1832/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376196174
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala
##########
@@ -316,16 +315,15 @@ private[sql] case class CarbonProjectForUpdateCommand(
val header = getHeader(carbonRelation, plan)
- CarbonLoadDataCommand(
+ CarbonInsertIntoWithDf(
Some(carbonRelation.identifier.getCarbonTableIdentifier.getDatabaseName),
carbonRelation.identifier.getCarbonTableIdentifier.getTableName,
- null,
- Seq(),
Map(("fileheader" -> header)),
false,
null,
- Some(dataFrame),
- Some(updateTableModel)).run(sparkSession)
+ dataFrame,
Review comment:
please add parameter names
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376357574
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java
##########
@@ -32,7 +33,7 @@
* Generic DataType interface which will be used while data loading for complex types like Array &
* Struct
*/
-public interface GenericDataType<T> {
+public interface GenericDataType<T> extends Serializable {
Review comment:
ok done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569676801
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1379/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378800963
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
Review comment:
yes. done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584011058
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/207/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569854185
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1388/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-572439137
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1556/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376996416
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584507515
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/230/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376348551
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378801215
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -953,28 +983,46 @@ object CarbonDataRDDFactory {
/**
* Execute load process to load from input dataframe
+ *
+ * @param sqlContext sql context
+ * @param dataFrame optional dataframe for insert
+ * @param scanResultRDD optional internal row rdd for direct insert
+ * @param carbonLoadModel load model
+ * @return Return an array that contains all of the elements in NewDataFrameLoaderRDD.
*/
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
carbonLoadModel: CarbonLoadModel
): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
try {
- val rdd = dataFrame.get.rdd
-
+ val rdd = if (dataFrame.isDefined) {
+ dataFrame.get.rdd
+ } else {
+ scanResultRDD.get
+ }
val nodeNumOfData = rdd.partitions.flatMap[String, Array[String]] { p =>
DataLoadPartitionCoalescer.getPreferredLocs(rdd, p).map(_.host)
}.distinct.length
val nodes = DistributionUtil.ensureExecutorsByNumberAndGetNodeList(
nodeNumOfData,
sqlContext.sparkContext)
- val newRdd = new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, rdd, nodes.toArray
- .distinct)
-
+ val newRdd =
+ if (dataFrame.isDefined) {
+ new DataLoadCoalescedRDD[Row](
+ sqlContext.sparkSession, dataFrame.get.rdd, nodes.toArray.distinct)
+ } else {
+ new DataLoadCoalescedRDD[InternalRow](
+ sqlContext.sparkSession,
+ scanResultRDD.get,
+ nodes.toArray.distinct)
+ }
new NewDataFrameLoaderRDD(
sqlContext.sparkSession,
new DataLoadResultImpl(),
carbonLoadModel,
+ dataFrame.isDefined,
Review comment:
yes. done like that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586253092
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/289/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378728896
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle alter table in future, this also must use new flow.
+ if (CarbonProperties.isBadRecordHandlingEnabledForInsert ||
+ isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp = databaseNameOp,
+ tableName = tableName,
+ options = options,
+ isOverwriteTable = isOverwriteTable,
+ dimFilesPath = dimFilesPath,
+ dataFrame = dataFrame,
+ inputSqlString = inputSqlString,
+ updateModel = None,
+ tableInfoOp = tableInfoOp,
+ internalOptions = internalOptions,
+ partition = partition).process(sparkSession)
+ }
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ CarbonProperties.getInstance().addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession)
+ val optionsFinal: util.Map[String, String] = CommonLoadUtils.getFinalLoadOptions(table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf = hadoopConf,
+ factPath = factPath,
+ optionsFinal = optionsFinal,
+ parentTablePath = parentTablePath,
+ table = table,
+ isDataFrame = true,
+ internalOptions = internalOptions,
+ partition = partition,
+ options = options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = getConvertedStaticPartitionMap(partitionColumnSchema)
+ val (reArrangedIndex, selectedColumnSchema) = getReArrangedIndexAndSelectedSchema(
+ tableInfoOp.get,
+ partitionColumnSchema)
+ val newLogicalPlan = getReArrangedLogicalPlan(
+ reArrangedIndex,
+ selectedColumnSchema,
+ convertedStaticPartition)
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedLogicalPlan(
+ reArrangedIndex: Seq[Int],
+ selectedColumnSchema: Seq[ColumnSchema],
+ convertedStaticPartition: mutable.Map[String, AnyRef]): LogicalPlan = {
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
Review comment:
move it to the case code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376343994
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -957,24 +987,36 @@ object CarbonDataRDDFactory {
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
carbonLoadModel: CarbonLoadModel
): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
try {
- val rdd = dataFrame.get.rdd
-
+ val rdd = if (dataFrame.isDefined) {
+ dataFrame.get.rdd
+ } else {
+ scanResultRDD.get
+ }
val nodeNumOfData = rdd.partitions.flatMap[String, Array[String]] { p =>
DataLoadPartitionCoalescer.getPreferredLocs(rdd, p).map(_.host)
}.distinct.length
val nodes = DistributionUtil.ensureExecutorsByNumberAndGetNodeList(
nodeNumOfData,
sqlContext.sparkContext)
- val newRdd = new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, rdd, nodes.toArray
- .distinct)
-
+ val newRdd =
+ if (dataFrame.isDefined) {
+ new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, dataFrame.get.rdd, nodes.toArray
+ .distinct)
+ } else {
+ new DataLoadCoalescedRDD[InternalRow](sqlContext.sparkSession,
+ scanResultRDD.get,
+ nodes.toArray
+ .distinct)
Review comment:
yes, this flow is for local sort and no sort. Have plan to send no sort to global sort plan.
local sort, one task one node. Hence this logic
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585270911
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1969/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378720643
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
+ val data = new Array[AnyRef](fields.size)
+ val badRecordLogHolder = new BadRecordLogHolder();
+ var i = 0
+ val fieldTypesLen = fields.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fields(i).dataType match {
+ case StringType =>
+ data(i) = DataTypeUtil.getBytesDataDataTypeForNoDictionaryColumn(row.getString(i),
+ DataTypes.STRING)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, arrayType), arrayType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
+ case structType : StructType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, structType), structType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[StructDataType]
+ .writeByteArray(result.asInstanceOf[StructObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
+ case mapType : MapType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, mapType), mapType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
Review comment:
reuse the method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585015939
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1947/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194673
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
Review comment:
move to previous line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-574483942
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1641/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194827
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
Review comment:
move `partition(col.getColumnName.toLowerCase)` to next line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376356414
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala
##########
@@ -509,30 +514,17 @@ private class CarbonOutputWriter(path: String,
// TODO Implement writesupport interface to support writing Row directly to recordwriter
def writeCarbon(row: InternalRow): Unit = {
- val data = new Array[AnyRef](fieldTypes.length + partitionData.length)
- var i = 0
- val fieldTypesLen = fieldTypes.length
- while (i < fieldTypesLen) {
- if (!row.isNullAt(i)) {
- fieldTypes(i) match {
- case StringType =>
- data(i) = row.getString(i)
- case d: DecimalType =>
- data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
- case other =>
- data(i) = row.get(i, other)
- }
- }
- i += 1
- }
+ val totalLength = fieldTypes.length + partitionData.length
+ val data: Array[AnyRef] = CommonUtil.getObjectArrayFromInternalRowAndConvertComplexType(row,
+ fieldTypes,
+ totalLength)
if (partitionData.length > 0) {
- System.arraycopy(partitionData, 0, data, fieldTypesLen, partitionData.length)
+ System.arraycopy(partitionData, 0, data, fieldTypes.length, partitionData.length)
Review comment:
yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376349852
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581264430
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/9/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585241664
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/266/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376339989
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -179,6 +182,93 @@ object DataLoadProcessBuilderOnSpark {
updateLoadStatus(model, partialSuccessAccum)
}
+ def insertDataUsingGlobalSortWithInternalRow(sparkSession: SparkSession,
+ scanResultRDD : RDD[InternalRow],
+ model: CarbonLoadModel,
+ hadoopConf: Configuration): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
+ val originRDD = scanResultRDD
+
+ val sc = sparkSession.sparkContext
+ val modelBroadcast = sc.broadcast(model)
+ val partialSuccessAccum = sc.longAccumulator("Partial Success Accumulator")
+ val sortStepRowCounter = sc.longAccumulator("Sort Processor Accumulator")
+ val writeStepRowCounter = sc.longAccumulator("Write Processor Accumulator")
+
+ hadoopConf
+ .set(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME, sparkSession.sparkContext.appName)
+
+ val conf = SparkSQLUtil.broadCastHadoopConf(sc, hadoopConf)
+ val configuration = DataLoadProcessBuilder.createConfiguration(model)
+
+ // 1. Convert internalRow to object array
+ val fields = Util
+ .convertToSparkSchemaFromColumnSchema(model.getCarbonDataLoadSchema.getCarbonTable, true)
+ .fields
+ .toSeq
+
+ val dataTypes = fields.map(field => field.dataType)
+ val map: mutable.Map[String, GenericDataType[_]] = mutable.Map[String, GenericDataType[_]]()
+ CommonUtil.convertComplexDataType(map, configuration)
+ val rdd = originRDD.map { internalRow =>
+ CommonUtil.getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(internalRow,
+ fields,
+ map)
+ }
+
+ // 2. sort
+ var numPartitions = CarbonDataProcessorUtil.getGlobalSortPartitions(
+ configuration.getDataLoadProperty(CarbonCommonConstants.LOAD_GLOBAL_SORT_PARTITIONS))
+ if (numPartitions <= 0) {
+ numPartitions = originRDD.partitions.length
+ }
+ // Because if the number of partitions greater than 1, there will be action operator
+ // (sample) in
+ // sortBy operator. So here we cache the rdd to avoid do input and convert again.
+ if (numPartitions > 1) {
+ rdd.persist(StorageLevel.fromString(
+ CarbonProperties.getInstance().getGlobalSortRddStorageLevel()))
+ }
+ val sortColumnsLength = model.getCarbonDataLoadSchema.getCarbonTable.getSortColumns.size()
+ val sortColumnDataTypes = dataTypes.take(sortColumnsLength)
+ val rowComparator = GlobalSortHelper.generateRowComparator(sortColumnDataTypes)
+ val sortRDD = rdd.sortBy(x => getKey(x, sortColumnsLength, sortColumnDataTypes),
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585092206
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1952/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376996464
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal,
+ parentTablePath,
+ table,
+ isDataFrame = true,
+ internalOptions,
+ finalPartition,
+ options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584654655
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1942/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581172106
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1827/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586117728
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/281/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378751755
##########
File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestLoadDataGeneral.scala
##########
@@ -150,10 +150,14 @@ class TestLoadDataGeneral extends QueryTest with BeforeAndAfterEach {
sql("drop table if exists load32000chardata")
sql("drop table if exists load32000chardata_dup")
sql("CREATE TABLE load32000chardata(dim1 String, dim2 String, mes1 int) STORED AS carbondata")
- sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1 int) STORED AS carbondata")
+ sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1 int) STORED AS " +
+ "carbondata tblproperties('local_dictionary_enable'='false')")
sql(s"LOAD DATA LOCAL INPATH '$testdata' into table load32000chardata OPTIONS('FILEHEADER'='dim1,dim2,mes1')")
+ // Previously converter step was checking more than 32k length and throwing exception.
+ // Now, due to local dictionary is true. Insert will not fail.
+ // when local dictionary is false, insert will fail at write step
intercept[Exception] {
- sql("insert into load32000chardata_dup select dim1,concat(load32000chardata.dim2,'aaaa'),mes1 from load32000chardata").show()
+ sql("insert into load32000chardata_dup select dim1,concat(load32000chardata.dim2,load32000chardata.dim2),mes1 from load32000chardata").show()
Review comment:
In new flow no converter step doesn't check string is more than 32k or not now. Write step didn't fail because after compression it is less than 32k. so I add 32k + 32k. so after compression it will still fail.
I can set bad record handle and make it same as old test case also
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r374615892
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal,
+ parentTablePath,
+ table,
+ isDataFrame = true,
+ internalOptions,
+ finalPartition,
+ options)
Review comment:
Can we have a single map instead of 'options' and 'internalOptions'?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378299353
##########
File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestLoadDataGeneral.scala
##########
@@ -150,10 +150,14 @@ class TestLoadDataGeneral extends QueryTest with BeforeAndAfterEach {
sql("drop table if exists load32000chardata")
sql("drop table if exists load32000chardata_dup")
sql("CREATE TABLE load32000chardata(dim1 String, dim2 String, mes1 int) STORED AS carbondata")
- sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1 int) STORED AS carbondata")
+ sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1 int) STORED AS " +
+ "carbondata tblproperties('local_dictionary_enable'='false')")
sql(s"LOAD DATA LOCAL INPATH '$testdata' into table load32000chardata OPTIONS('FILEHEADER'='dim1,dim2,mes1')")
+ // Previously converter step was checking more than 32k length and throwing exception.
+ // Now, due to local dictionary is true. Insert will not fail.
+ // when local dictionary is false, insert will fail at write step
intercept[Exception] {
- sql("insert into load32000chardata_dup select dim1,concat(load32000chardata.dim2,'aaaa'),mes1 from load32000chardata").show()
+ sql("insert into load32000chardata_dup select dim1,concat(load32000chardata.dim2,load32000chardata.dim2),mes1 from load32000chardata").show()
Review comment:
why change it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378754084
##########
File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestLoadDataGeneral.scala
##########
@@ -150,10 +150,14 @@ class TestLoadDataGeneral extends QueryTest with BeforeAndAfterEach {
sql("drop table if exists load32000chardata")
sql("drop table if exists load32000chardata_dup")
sql("CREATE TABLE load32000chardata(dim1 String, dim2 String, mes1 int) STORED AS carbondata")
- sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1 int) STORED AS carbondata")
+ sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1 int) STORED AS " +
+ "carbondata tblproperties('local_dictionary_enable'='false')")
sql(s"LOAD DATA LOCAL INPATH '$testdata' into table load32000chardata OPTIONS('FILEHEADER'='dim1,dim2,mes1')")
+ // Previously converter step was checking more than 32k length and throwing exception.
+ // Now, due to local dictionary is true. Insert will not fail.
+ // when local dictionary is false, insert will fail at write step
intercept[Exception] {
- sql("insert into load32000chardata_dup select dim1,concat(load32000chardata.dim2,'aaaa'),mes1 from load32000chardata").show()
+ sql("insert into load32000chardata_dup select dim1,concat(load32000chardata.dim2,load32000chardata.dim2),mes1 from load32000chardata").show()
Review comment:
I set bad record handle and make it same as old test case. Else nobody else will understand :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584628870
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/240/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570164984
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1399/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376196366
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonCreateTableAsSelectCommand.scala
##########
@@ -76,11 +76,16 @@ case class CarbonCreateTableAsSelectCommand(
.createCarbonDataSourceHadoopRelation(sparkSession,
TableIdentifier(tableName, Option(dbName)))
// execute command to load data into carbon table
- loadCommand = CarbonInsertIntoCommand(
- carbonDataSourceHadoopRelation,
- query,
- overwrite = false,
- partition = Map.empty)
+ loadCommand = CarbonInsertIntoCommand(Some(carbonDataSourceHadoopRelation
Review comment:
change
```
Some(carbonDataSourceHadoopRelation
.carbonRelation
.databaseName),
```
to same line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376195210
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
+ .get,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(col.getDataType),
+ timeStampFormat,
+ dateFormat)
+ }
+ }
+ }
+ val partitionColumnNames = if (partitionColumnSchema != null) {
+ partitionColumnSchema.map(x => x.getColumnName).toSet
+ } else {
+ null
+ }
+ // get invisible column indexes, alter table scenarios can have it before or after new column
+ // dummy measure will have ordinal -1 and it is invisible, ignore that column.
+ // alter table old columns are just invisible columns with proper ordinal
+ val invisibleIndex = columnSchema.filter(col => col.isInvisible && col.getSchemaOrdinal != -1)
+ .map(col => col.getSchemaOrdinal)
+ columnSchema.filterNot(col => col.isInvisible).foreach {
+ col =>
+ var skipPartitionColumn = false
+ if (col.getColumnName.contains(".")) {
+ // If the schema ordinal is -1,
+ // no need to consider it during shifting columns to derive new shifted ordinal
+ if (col.getSchemaOrdinal != -1) {
+ complexChildCount = complexChildCount + 1
+ }
+ } else {
+ // get number of invisible index count before this column
+ val invisibleIndexCount = invisibleIndex.count(index => index < col.getSchemaOrdinal)
+ if (col.getDataType.isComplexType) {
+ // Calculate re-arrange index by ignoring the complex child count.
+ // As projection will have only parent columns
+ reArrangedIndex = reArrangedIndex :+
+ (col.getSchemaOrdinal - complexChildCount - invisibleIndexCount)
+ } else {
+ if (partitionColumnNames != null && partitionColumnNames.contains(col.getColumnName)) {
+ partitionIndex = partitionIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ skipPartitionColumn = true
+ } else {
+ reArrangedIndex = reArrangedIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ }
+ }
+ if (!skipPartitionColumn) {
+ selectedColumnSchema = selectedColumnSchema :+ col
+ }
+ }
+ }
+ if (partitionColumnSchema != null) {
+ // keep partition columns in the end
+ selectedColumnSchema = selectedColumnSchema ++ partitionColumnSchema
+ }
+ if (partitionIndex.nonEmpty) {
+ // keep partition columns in the end and in the original create order
+ reArrangedIndex = reArrangedIndex ++ partitionIndex.sortBy(x => x)
+ }
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
+ if (partition.nonEmpty) {
+ // partition keyword is present in insert and
+ // select query partition projections may not be same as create order.
+ // So, bring to create table order
+ val dynamicPartition = partition.filterNot(entry => entry._2.isDefined)
+ var index = 0
+ val map = mutable.Map[String, Int]()
+ for (part <- dynamicPartition) {
+ map(part._1) = index
+ index = index + 1
+ }
+ var tempList = oldProjectionList.take(oldProjectionList.size - dynamicPartition.size)
+ val partitionList = oldProjectionList.takeRight(dynamicPartition.size)
+ val partitionSchema = table.getPartitionInfo.getColumnSchemaList.asScala
+ for (partitionCol <- partitionSchema) {
+ if (map.get(partitionCol.getColumnName).isDefined) {
+ tempList = tempList :+ partitionList(map(partitionCol.getColumnName))
+ }
+ }
+ oldProjectionList = tempList
+ }
+ if (reArrangedIndex.size != oldProjectionList.size) {
+ // for non-partition table columns must match
+ if (partition.isEmpty) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ if (reArrangedIndex.size - oldProjectionList.size != convertedStaticPartition.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ // TODO: For partition case, remaining projections need to validate ?
+ }
+ }
+ }
+ var newProjectionList: Seq[NamedExpression] = Seq.empty
+ var i = 0
+ while (i < reArrangedIndex.size) {
+ // column schema is already has sortColumns-dimensions-measures. Collect the ordinal &
+ // re-arrange the projection in the same order
+ if (partition.nonEmpty &&
+ convertedStaticPartition.contains(selectedColumnSchema(i).getColumnName
+ .toLowerCase())) {
+ // If column schema present in partitionSchema means it is a static partition,
+ // then add a value literal expression in the project.
+ val value = convertedStaticPartition(selectedColumnSchema(i).getColumnName
+ .toLowerCase())
+ newProjectionList = newProjectionList :+
+ Alias(new Literal(value,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(
+ selectedColumnSchema(i).getDataType)), value.toString)(
+ NamedExpression.newExprId,
+ None,
+ None).asInstanceOf[NamedExpression]
+ } else {
+ // If column schema NOT present in partition column,
+ // get projection column mapping its ordinal.
+ if (partition.contains(selectedColumnSchema(i).getColumnName.toLowerCase())) {
+ // static partition + dynamic partition case,
+ // here dynamic partition ordinal will be more than projection size
+ newProjectionList = newProjectionList :+
+ oldProjectionList(
+ reArrangedIndex(i) - convertedStaticPartition.size)
+ } else {
+ newProjectionList = newProjectionList :+
+ oldProjectionList(reArrangedIndex(i))
+ }
+ }
+ i = i + 1
+ }
+ processedProject = true
+ Project(newProjectionList, p.child)
+ } else {
+ p
+ }
+ }
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedSchemaLogicalRelation(reArrangedIndex: Seq[Int],
+ logicalRelation: LogicalRelation): LogicalRelation = {
+ if (reArrangedIndex.size != logicalRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ logicalRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](logicalRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](logicalRelation.schema.size)
+ val fields = logicalRelation.schema.fields
+ val output = logicalRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = logicalRelation.catalogTable
+ .get
+ .copy(schema = new StructType(reArrangedFields))
+ logicalRelation.copy(logicalRelation.relation,
+ reArrangedAttributes,
+ Some(catalogTable))
+ }
+
+ def getReArrangedSchemaHiveRelation(reArrangedIndex: Seq[Int],
Review comment:
add comment to explain the logic
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-580739283
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1819/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585058162
@QiangCai , @ravipesala , @kumarvishal09 : please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569387388
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1330/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570573011
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1451/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585146200
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/260/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584103304
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/211/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376355932
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/table/CarbonCreateTableAsSelectCommand.scala
##########
@@ -76,11 +76,16 @@ case class CarbonCreateTableAsSelectCommand(
.createCarbonDataSourceHadoopRelation(sparkSession,
TableIdentifier(tableName, Option(dbName)))
// execute command to load data into carbon table
- loadCommand = CarbonInsertIntoCommand(
- carbonDataSourceHadoopRelation,
- query,
- overwrite = false,
- partition = Map.empty)
+ loadCommand = CarbonInsertIntoCommand(Some(carbonDataSourceHadoopRelation
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585000077
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/245/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378690024
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
##########
@@ -140,6 +141,89 @@ object CarbonScalaUtil {
}
}
+ /**
+ * Converts incoming value to String after converting data as per the data type.
+ *
+ * @param value Input value to convert
+ * @param dataType Datatype to convert and then convert to String
+ * @param timeStampFormat Timestamp format to convert in case of timestamp datatypes
+ * @param dateFormat DataFormat to convert in case of DateType datatype
+ * @return converted String
+ */
+ def convertStaticPartitionToValues(
+ value: String,
+ dataType: DataType,
+ timeStampFormat: SimpleDateFormat,
+ dateFormat: SimpleDateFormat): AnyRef = {
+ val defaultValue = value != null && value.equalsIgnoreCase(hiveDefaultPartition)
+ try {
+ dataType match {
+ case TimestampType if timeStampFormat != null =>
+ val formattedString =
+ if (defaultValue) {
+ timeStampFormat.format(new Date())
+ } else {
+ timeStampFormat.format(DateTimeUtils.stringToTime(value))
+ }
+ val convertedValue =
+ DataTypeUtil
+ .getDataBasedOnDataType(formattedString,
+ CarbonSparkDataSourceUtil.convertSparkToCarbonDataType(TimestampType))
+ convertedValue
+ case DateType if dateFormat != null =>
+ val formattedString =
+ if (defaultValue) {
+ dateFormat.format(new Date())
+ } else {
+ dateFormat.format(DateTimeUtils.stringToTime(value))
+ }
+ val convertedValue =
+ DataTypeUtil
+ .getDataBasedOnDataType(formattedString,
+ CarbonSparkDataSourceUtil.convertSparkToCarbonDataType(TimestampType))
+ val date = generateDictionaryKey(convertedValue.asInstanceOf[Long])
+ date.asInstanceOf[AnyRef]
+ case BinaryType =>
+ // TODO: decode required ? currently it is working
+ ByteUtil.toBytes(value)
+ case _ =>
+ val convertedValue =
+ DataTypeUtil
+ .getDataBasedOnDataType(value,
+ CarbonSparkDataSourceUtil.convertSparkToCarbonDataType(dataType))
+ if (convertedValue == null) {
+ if (defaultValue) {
+ dataType match {
+ case BooleanType =>
+ return false.asInstanceOf[AnyRef]
+ case _ =>
+ return 0.asInstanceOf[AnyRef]
+ }
+ }
+ throw new MalformedCarbonCommandException(
+ s"Value $value with datatype $dataType on static partition is not correct")
+ }
+ convertedValue
+ }
+ } catch {
+ case e: Exception =>
+ throw new MalformedCarbonCommandException(
+ s"Value $value with datatype $dataType on static partition is not correct")
+ }
+ }
+
+ def generateDictionaryKey(timeValue: Long): Int = {
+ if (timeValue < DateDirectDictionaryGenerator.MIN_VALUE ||
+ timeValue > DateDirectDictionaryGenerator.MAX_VALUE) {
+ if (LOGGER.isDebugEnabled) {
+ LOGGER.debug("Value for date type column is not in valid range. Value considered as null.")
+ }
+ return CarbonCommonConstants.DIRECT_DICT_VALUE_NULL
+ }
+ Math.floor(timeValue.toDouble / DateDirectDictionaryGenerator.MILLIS_PER_DAY).toInt +
+ DateDirectDictionaryGenerator.cutOffDate
Review comment:
please fix the code style
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r374623053
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal,
+ parentTablePath,
+ table,
+ isDataFrame = true,
+ internalOptions,
+ finalPartition,
+ options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
Review comment:
Remove this log
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376191797
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -179,6 +182,93 @@ object DataLoadProcessBuilderOnSpark {
updateLoadStatus(model, partialSuccessAccum)
}
+ def insertDataUsingGlobalSortWithInternalRow(sparkSession: SparkSession,
Review comment:
move `sparkSession: SparkSession,` to next line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584085042
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1912/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-574080444
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1633/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-580603426
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1815/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376195242
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
+ .get,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(col.getDataType),
+ timeStampFormat,
+ dateFormat)
+ }
+ }
+ }
+ val partitionColumnNames = if (partitionColumnSchema != null) {
+ partitionColumnSchema.map(x => x.getColumnName).toSet
+ } else {
+ null
+ }
+ // get invisible column indexes, alter table scenarios can have it before or after new column
+ // dummy measure will have ordinal -1 and it is invisible, ignore that column.
+ // alter table old columns are just invisible columns with proper ordinal
+ val invisibleIndex = columnSchema.filter(col => col.isInvisible && col.getSchemaOrdinal != -1)
+ .map(col => col.getSchemaOrdinal)
+ columnSchema.filterNot(col => col.isInvisible).foreach {
+ col =>
+ var skipPartitionColumn = false
+ if (col.getColumnName.contains(".")) {
+ // If the schema ordinal is -1,
+ // no need to consider it during shifting columns to derive new shifted ordinal
+ if (col.getSchemaOrdinal != -1) {
+ complexChildCount = complexChildCount + 1
+ }
+ } else {
+ // get number of invisible index count before this column
+ val invisibleIndexCount = invisibleIndex.count(index => index < col.getSchemaOrdinal)
+ if (col.getDataType.isComplexType) {
+ // Calculate re-arrange index by ignoring the complex child count.
+ // As projection will have only parent columns
+ reArrangedIndex = reArrangedIndex :+
+ (col.getSchemaOrdinal - complexChildCount - invisibleIndexCount)
+ } else {
+ if (partitionColumnNames != null && partitionColumnNames.contains(col.getColumnName)) {
+ partitionIndex = partitionIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ skipPartitionColumn = true
+ } else {
+ reArrangedIndex = reArrangedIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ }
+ }
+ if (!skipPartitionColumn) {
+ selectedColumnSchema = selectedColumnSchema :+ col
+ }
+ }
+ }
+ if (partitionColumnSchema != null) {
+ // keep partition columns in the end
+ selectedColumnSchema = selectedColumnSchema ++ partitionColumnSchema
+ }
+ if (partitionIndex.nonEmpty) {
+ // keep partition columns in the end and in the original create order
+ reArrangedIndex = reArrangedIndex ++ partitionIndex.sortBy(x => x)
+ }
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
+ if (partition.nonEmpty) {
+ // partition keyword is present in insert and
+ // select query partition projections may not be same as create order.
+ // So, bring to create table order
+ val dynamicPartition = partition.filterNot(entry => entry._2.isDefined)
+ var index = 0
+ val map = mutable.Map[String, Int]()
+ for (part <- dynamicPartition) {
+ map(part._1) = index
+ index = index + 1
+ }
+ var tempList = oldProjectionList.take(oldProjectionList.size - dynamicPartition.size)
+ val partitionList = oldProjectionList.takeRight(dynamicPartition.size)
+ val partitionSchema = table.getPartitionInfo.getColumnSchemaList.asScala
+ for (partitionCol <- partitionSchema) {
+ if (map.get(partitionCol.getColumnName).isDefined) {
+ tempList = tempList :+ partitionList(map(partitionCol.getColumnName))
+ }
+ }
+ oldProjectionList = tempList
+ }
+ if (reArrangedIndex.size != oldProjectionList.size) {
+ // for non-partition table columns must match
+ if (partition.isEmpty) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ if (reArrangedIndex.size - oldProjectionList.size != convertedStaticPartition.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ // TODO: For partition case, remaining projections need to validate ?
+ }
+ }
+ }
+ var newProjectionList: Seq[NamedExpression] = Seq.empty
+ var i = 0
+ while (i < reArrangedIndex.size) {
+ // column schema is already has sortColumns-dimensions-measures. Collect the ordinal &
+ // re-arrange the projection in the same order
+ if (partition.nonEmpty &&
+ convertedStaticPartition.contains(selectedColumnSchema(i).getColumnName
+ .toLowerCase())) {
+ // If column schema present in partitionSchema means it is a static partition,
+ // then add a value literal expression in the project.
+ val value = convertedStaticPartition(selectedColumnSchema(i).getColumnName
+ .toLowerCase())
+ newProjectionList = newProjectionList :+
+ Alias(new Literal(value,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(
+ selectedColumnSchema(i).getDataType)), value.toString)(
+ NamedExpression.newExprId,
+ None,
+ None).asInstanceOf[NamedExpression]
+ } else {
+ // If column schema NOT present in partition column,
+ // get projection column mapping its ordinal.
+ if (partition.contains(selectedColumnSchema(i).getColumnName.toLowerCase())) {
+ // static partition + dynamic partition case,
+ // here dynamic partition ordinal will be more than projection size
+ newProjectionList = newProjectionList :+
+ oldProjectionList(
+ reArrangedIndex(i) - convertedStaticPartition.size)
+ } else {
+ newProjectionList = newProjectionList :+
+ oldProjectionList(reArrangedIndex(i))
+ }
+ }
+ i = i + 1
+ }
+ processedProject = true
+ Project(newProjectionList, p.child)
+ } else {
+ p
+ }
+ }
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedSchemaLogicalRelation(reArrangedIndex: Seq[Int],
Review comment:
add comment to explain the logic
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581464371
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/135/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-574158103
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1636/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] asfgit closed pull request #3538: [CARBONDATA-3637]
Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569695697
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1371/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376346935
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-571958870
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1533/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378801595
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CarbonScalaUtil.scala
##########
@@ -140,6 +141,89 @@ object CarbonScalaUtil {
}
}
+ /**
+ * Converts incoming value to String after converting data as per the data type.
+ *
+ * @param value Input value to convert
+ * @param dataType Datatype to convert and then convert to String
+ * @param timeStampFormat Timestamp format to convert in case of timestamp datatypes
+ * @param dateFormat DataFormat to convert in case of DateType datatype
+ * @return converted String
+ */
+ def convertStaticPartitionToValues(
+ value: String,
+ dataType: DataType,
+ timeStampFormat: SimpleDateFormat,
+ dateFormat: SimpleDateFormat): AnyRef = {
+ val defaultValue = value != null && value.equalsIgnoreCase(hiveDefaultPartition)
+ try {
+ dataType match {
+ case TimestampType if timeStampFormat != null =>
+ val formattedString =
+ if (defaultValue) {
+ timeStampFormat.format(new Date())
+ } else {
+ timeStampFormat.format(DateTimeUtils.stringToTime(value))
+ }
+ val convertedValue =
+ DataTypeUtil
+ .getDataBasedOnDataType(formattedString,
+ CarbonSparkDataSourceUtil.convertSparkToCarbonDataType(TimestampType))
+ convertedValue
+ case DateType if dateFormat != null =>
+ val formattedString =
+ if (defaultValue) {
+ dateFormat.format(new Date())
+ } else {
+ dateFormat.format(DateTimeUtils.stringToTime(value))
+ }
+ val convertedValue =
+ DataTypeUtil
+ .getDataBasedOnDataType(formattedString,
+ CarbonSparkDataSourceUtil.convertSparkToCarbonDataType(TimestampType))
+ val date = generateDictionaryKey(convertedValue.asInstanceOf[Long])
+ date.asInstanceOf[AnyRef]
+ case BinaryType =>
+ // TODO: decode required ? currently it is working
+ ByteUtil.toBytes(value)
+ case _ =>
+ val convertedValue =
+ DataTypeUtil
+ .getDataBasedOnDataType(value,
+ CarbonSparkDataSourceUtil.convertSparkToCarbonDataType(dataType))
+ if (convertedValue == null) {
+ if (defaultValue) {
+ dataType match {
+ case BooleanType =>
+ return false.asInstanceOf[AnyRef]
+ case _ =>
+ return 0.asInstanceOf[AnyRef]
+ }
+ }
+ throw new MalformedCarbonCommandException(
+ s"Value $value with datatype $dataType on static partition is not correct")
+ }
+ convertedValue
+ }
+ } catch {
+ case e: Exception =>
+ throw new MalformedCarbonCommandException(
+ s"Value $value with datatype $dataType on static partition is not correct")
+ }
+ }
+
+ def generateDictionaryKey(timeValue: Long): Int = {
+ if (timeValue < DateDirectDictionaryGenerator.MIN_VALUE ||
+ timeValue > DateDirectDictionaryGenerator.MAX_VALUE) {
+ if (LOGGER.isDebugEnabled) {
+ LOGGER.debug("Value for date type column is not in valid range. Value considered as null.")
+ }
+ return CarbonCommonConstants.DIRECT_DICT_VALUE_NULL
+ }
+ Math.floor(timeValue.toDouble / DateDirectDictionaryGenerator.MILLIS_PER_DAY).toInt +
+ DateDirectDictionaryGenerator.cutOffDate
Review comment:
no change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378771252
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
+ val data = new Array[AnyRef](fields.size)
+ val badRecordLogHolder = new BadRecordLogHolder();
+ var i = 0
+ val fieldTypesLen = fields.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fields(i).dataType match {
+ case StringType =>
+ data(i) = DataTypeUtil.getBytesDataDataTypeForNoDictionaryColumn(row.getString(i),
+ DataTypes.STRING)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, arrayType), arrayType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
+ case structType : StructType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, structType), structType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[StructDataType]
+ .writeByteArray(result.asInstanceOf[StructObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
+ case mapType : MapType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, mapType), mapType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
Review comment:
same as above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376191565
##########
File path: integration/spark-common/src/main/java/org/apache/carbondata/spark/util/Util.java
##########
@@ -76,9 +76,21 @@ public static boolean isBlockWithoutBlockletInfoExists(List<CarbonInputSplit> sp
} else if (org.apache.carbondata.core.metadata.datatype.DataTypes.isDecimal(carbonDataType)) {
return DataTypes.createDecimalType();
} else if (carbonDataType == org.apache.carbondata.core.metadata.datatype.DataTypes.TIMESTAMP) {
+ if (forGlobalSort) {
Review comment:
Timestamp is not direct dictionary, is this needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585136781
@QiangCai : OK rebased
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194051
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -957,24 +987,36 @@ object CarbonDataRDDFactory {
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
carbonLoadModel: CarbonLoadModel
): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
try {
- val rdd = dataFrame.get.rdd
-
+ val rdd = if (dataFrame.isDefined) {
+ dataFrame.get.rdd
+ } else {
+ scanResultRDD.get
+ }
val nodeNumOfData = rdd.partitions.flatMap[String, Array[String]] { p =>
DataLoadPartitionCoalescer.getPreferredLocs(rdd, p).map(_.host)
}.distinct.length
val nodes = DistributionUtil.ensureExecutorsByNumberAndGetNodeList(
nodeNumOfData,
sqlContext.sparkContext)
- val newRdd = new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, rdd, nodes.toArray
- .distinct)
-
+ val newRdd =
+ if (dataFrame.isDefined) {
+ new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, dataFrame.get.rdd, nodes.toArray
+ .distinct)
+ } else {
+ new DataLoadCoalescedRDD[InternalRow](sqlContext.sparkSession,
Review comment:
move`sqlContext.sparkSession,` to next line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194251
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -957,24 +987,36 @@ object CarbonDataRDDFactory {
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
carbonLoadModel: CarbonLoadModel
): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
try {
- val rdd = dataFrame.get.rdd
-
+ val rdd = if (dataFrame.isDefined) {
+ dataFrame.get.rdd
+ } else {
+ scanResultRDD.get
+ }
val nodeNumOfData = rdd.partitions.flatMap[String, Array[String]] { p =>
DataLoadPartitionCoalescer.getPreferredLocs(rdd, p).map(_.host)
}.distinct.length
val nodes = DistributionUtil.ensureExecutorsByNumberAndGetNodeList(
nodeNumOfData,
sqlContext.sparkContext)
- val newRdd = new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, rdd, nodes.toArray
- .distinct)
-
+ val newRdd =
+ if (dataFrame.isDefined) {
+ new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, dataFrame.get.rdd, nodes.toArray
+ .distinct)
+ } else {
+ new DataLoadCoalescedRDD[InternalRow](sqlContext.sparkSession,
+ scanResultRDD.get,
+ nodes.toArray
+ .distinct)
Review comment:
Is this for local sort? why distinct of the nodes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-579834563
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1797/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r379262676
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
Review comment:
No. cannot send to 'other' branch.
**other branch will give utf8 string, we need java string object here.**
added comment now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570167520
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1408/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-580828952
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1821/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r374615892
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal,
+ parentTablePath,
+ table,
+ isDataFrame = true,
+ internalOptions,
+ finalPartition,
+ options)
Review comment:
Can we have a single map instead of 'options' and 'internalOptions'?
Can be fixed in this PR as all Load classes are being refactored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376197271
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/DataLoadProcessBuilder.java
##########
@@ -110,10 +113,10 @@ private AbstractDataLoadProcessorStep buildInternalForNoSort(CarbonIterator[] in
*/
private AbstractDataLoadProcessorStep buildInternalWithNoConverter(
CarbonIterator[] inputIterators, CarbonDataLoadConfiguration configuration,
- SortScopeOptions.SortScope sortScope) {
+ SortScopeOptions.SortScope sortScope, boolean withoutReArrange) {
Review comment:
can we remove all re-arrange in all flow?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569617458
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1373/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378286214
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
Review comment:
not required
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378801536
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
Review comment:
it was from base code.
yes. done like that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585078502
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/249/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-576496694
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1710/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376337691
##########
File path: integration/spark-common/src/main/java/org/apache/carbondata/spark/util/Util.java
##########
@@ -76,9 +76,21 @@ public static boolean isBlockWithoutBlockletInfoExists(List<CarbonInputSplit> sp
} else if (org.apache.carbondata.core.metadata.datatype.DataTypes.isDecimal(carbonDataType)) {
return DataTypes.createDecimalType();
} else if (carbonDataType == org.apache.carbondata.core.metadata.datatype.DataTypes.TIMESTAMP) {
+ if (forGlobalSort) {
Review comment:
It is required, In MeasureFieldConverterImpl, we convert it to long value in getNoDictionaryValueBasedOnDataType method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [WIP] Separate Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#discussion_r369382408
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -172,6 +342,7 @@ object DataLoadProcessBuilderOnSpark {
updateLoadStatus(model, partialSuccessAccum)
}
+
Review comment:
This is WIP PR, I will update you once ready.
I am making 3 flows. one of load, one of insert with converter, insert without converter. I have duplicated code for this. Once all the test case is fixed. I will extract common code from all 3 flows.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376342021
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -957,24 +987,36 @@ object CarbonDataRDDFactory {
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
carbonLoadModel: CarbonLoadModel
): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
try {
- val rdd = dataFrame.get.rdd
-
+ val rdd = if (dataFrame.isDefined) {
+ dataFrame.get.rdd
+ } else {
+ scanResultRDD.get
+ }
val nodeNumOfData = rdd.partitions.flatMap[String, Array[String]] { p =>
DataLoadPartitionCoalescer.getPreferredLocs(rdd, p).map(_.host)
}.distinct.length
val nodes = DistributionUtil.ensureExecutorsByNumberAndGetNodeList(
nodeNumOfData,
sqlContext.sparkContext)
- val newRdd = new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, rdd, nodes.toArray
- .distinct)
-
+ val newRdd =
+ if (dataFrame.isDefined) {
+ new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, dataFrame.get.rdd, nodes.toArray
+ .distinct)
+ } else {
+ new DataLoadCoalescedRDD[InternalRow](sqlContext.sparkSession,
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584529631
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1932/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581517381
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/138/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378801391
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -304,11 +305,11 @@ object CarbonDataRDDFactory {
def loadCarbonData(
sqlContext: SQLContext,
carbonLoadModel: CarbonLoadModel,
- columnar: Boolean,
partitionStatus: SegmentStatus = SegmentStatus.SUCCESS,
overwriteTable: Boolean,
hadoopConf: Configuration,
dataFrame: Option[DataFrame] = None,
+ scanResultRdd : Option[RDD[InternalRow]],
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376340053
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal,
+ parentTablePath,
+ table,
+ isDataFrame = true,
+ internalOptions,
+ finalPartition,
+ options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378722658
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -358,19 +359,41 @@ object CarbonDataRDDFactory {
}
}
} else {
- status = if (dataFrame.isEmpty && isSortTable &&
- carbonLoadModel.getRangePartitionColumn != null &&
- (sortScope.equals(SortScopeOptions.SortScope.GLOBAL_SORT) ||
- sortScope.equals(SortScopeOptions.SortScope.LOCAL_SORT))) {
- DataLoadProcessBuilderOnSpark
- .loadDataUsingRangeSort(sqlContext.sparkSession, carbonLoadModel, hadoopConf)
- } else if (isSortTable && sortScope.equals(SortScopeOptions.SortScope.GLOBAL_SORT)) {
- DataLoadProcessBuilderOnSpark.loadDataUsingGlobalSort(sqlContext.sparkSession,
- dataFrame, carbonLoadModel, hadoopConf)
- } else if (dataFrame.isDefined) {
- loadDataFrame(sqlContext, dataFrame, carbonLoadModel)
+ status = if (scanResultRdd.isDefined) {
Review comment:
now the if code block becomes bigger, better to cut some branches in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573320963
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1601/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569617569
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1362/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-579178742
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1780/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376361675
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java
##########
@@ -32,7 +33,7 @@
* Generic DataType interface which will be used while data loading for complex types like Array &
* Struct
*/
-public interface GenericDataType<T> {
+public interface GenericDataType<T> extends Serializable {
Review comment:
Primitive type logger was not user, removed the field itself
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569383717
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1307/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376195414
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
+ .get,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(col.getDataType),
+ timeStampFormat,
+ dateFormat)
+ }
+ }
+ }
+ val partitionColumnNames = if (partitionColumnSchema != null) {
+ partitionColumnSchema.map(x => x.getColumnName).toSet
+ } else {
+ null
+ }
+ // get invisible column indexes, alter table scenarios can have it before or after new column
+ // dummy measure will have ordinal -1 and it is invisible, ignore that column.
+ // alter table old columns are just invisible columns with proper ordinal
+ val invisibleIndex = columnSchema.filter(col => col.isInvisible && col.getSchemaOrdinal != -1)
+ .map(col => col.getSchemaOrdinal)
+ columnSchema.filterNot(col => col.isInvisible).foreach {
+ col =>
+ var skipPartitionColumn = false
+ if (col.getColumnName.contains(".")) {
+ // If the schema ordinal is -1,
+ // no need to consider it during shifting columns to derive new shifted ordinal
+ if (col.getSchemaOrdinal != -1) {
+ complexChildCount = complexChildCount + 1
+ }
+ } else {
+ // get number of invisible index count before this column
+ val invisibleIndexCount = invisibleIndex.count(index => index < col.getSchemaOrdinal)
+ if (col.getDataType.isComplexType) {
+ // Calculate re-arrange index by ignoring the complex child count.
+ // As projection will have only parent columns
+ reArrangedIndex = reArrangedIndex :+
+ (col.getSchemaOrdinal - complexChildCount - invisibleIndexCount)
+ } else {
+ if (partitionColumnNames != null && partitionColumnNames.contains(col.getColumnName)) {
+ partitionIndex = partitionIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ skipPartitionColumn = true
+ } else {
+ reArrangedIndex = reArrangedIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ }
+ }
+ if (!skipPartitionColumn) {
+ selectedColumnSchema = selectedColumnSchema :+ col
+ }
+ }
+ }
+ if (partitionColumnSchema != null) {
+ // keep partition columns in the end
+ selectedColumnSchema = selectedColumnSchema ++ partitionColumnSchema
+ }
+ if (partitionIndex.nonEmpty) {
+ // keep partition columns in the end and in the original create order
+ reArrangedIndex = reArrangedIndex ++ partitionIndex.sortBy(x => x)
+ }
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
+ if (partition.nonEmpty) {
+ // partition keyword is present in insert and
+ // select query partition projections may not be same as create order.
+ // So, bring to create table order
+ val dynamicPartition = partition.filterNot(entry => entry._2.isDefined)
+ var index = 0
+ val map = mutable.Map[String, Int]()
+ for (part <- dynamicPartition) {
+ map(part._1) = index
+ index = index + 1
+ }
+ var tempList = oldProjectionList.take(oldProjectionList.size - dynamicPartition.size)
+ val partitionList = oldProjectionList.takeRight(dynamicPartition.size)
+ val partitionSchema = table.getPartitionInfo.getColumnSchemaList.asScala
+ for (partitionCol <- partitionSchema) {
+ if (map.get(partitionCol.getColumnName).isDefined) {
+ tempList = tempList :+ partitionList(map(partitionCol.getColumnName))
+ }
+ }
+ oldProjectionList = tempList
+ }
+ if (reArrangedIndex.size != oldProjectionList.size) {
+ // for non-partition table columns must match
+ if (partition.isEmpty) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ if (reArrangedIndex.size - oldProjectionList.size != convertedStaticPartition.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ // TODO: For partition case, remaining projections need to validate ?
+ }
+ }
+ }
+ var newProjectionList: Seq[NamedExpression] = Seq.empty
+ var i = 0
+ while (i < reArrangedIndex.size) {
+ // column schema is already has sortColumns-dimensions-measures. Collect the ordinal &
+ // re-arrange the projection in the same order
+ if (partition.nonEmpty &&
+ convertedStaticPartition.contains(selectedColumnSchema(i).getColumnName
+ .toLowerCase())) {
+ // If column schema present in partitionSchema means it is a static partition,
+ // then add a value literal expression in the project.
+ val value = convertedStaticPartition(selectedColumnSchema(i).getColumnName
+ .toLowerCase())
+ newProjectionList = newProjectionList :+
+ Alias(new Literal(value,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(
+ selectedColumnSchema(i).getDataType)), value.toString)(
+ NamedExpression.newExprId,
+ None,
+ None).asInstanceOf[NamedExpression]
+ } else {
+ // If column schema NOT present in partition column,
+ // get projection column mapping its ordinal.
+ if (partition.contains(selectedColumnSchema(i).getColumnName.toLowerCase())) {
+ // static partition + dynamic partition case,
+ // here dynamic partition ordinal will be more than projection size
+ newProjectionList = newProjectionList :+
+ oldProjectionList(
+ reArrangedIndex(i) - convertedStaticPartition.size)
+ } else {
+ newProjectionList = newProjectionList :+
+ oldProjectionList(reArrangedIndex(i))
+ }
+ }
+ i = i + 1
+ }
+ processedProject = true
+ Project(newProjectionList, p.child)
+ } else {
+ p
+ }
+ }
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedSchemaLogicalRelation(reArrangedIndex: Seq[Int],
+ logicalRelation: LogicalRelation): LogicalRelation = {
+ if (reArrangedIndex.size != logicalRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ logicalRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](logicalRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](logicalRelation.schema.size)
+ val fields = logicalRelation.schema.fields
+ val output = logicalRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = logicalRelation.catalogTable
+ .get
+ .copy(schema = new StructType(reArrangedFields))
+ logicalRelation.copy(logicalRelation.relation,
+ reArrangedAttributes,
+ Some(catalogTable))
+ }
+
+ def getReArrangedSchemaHiveRelation(reArrangedIndex: Seq[Int],
+ hiveTableRelation: HiveTableRelation): HiveTableRelation = {
+ if (reArrangedIndex.size != hiveTableRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ hiveTableRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](hiveTableRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](hiveTableRelation.schema.size)
+ val fields = hiveTableRelation.schema.fields
+ val output = hiveTableRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = hiveTableRelation.tableMeta.copy(schema = new StructType(reArrangedFields))
+ hiveTableRelation.copy(catalogTable,
+ reArrangedAttributes)
+ }
+
+ def insertData(loadParams: CarbonLoadParams): (Seq[Row], LoadMetadataDetails) = {
+ var rows = Seq.empty[Row]
+ val table = loadParams.carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
+ var loadResult : LoadMetadataDetails = null
+ if (table.isHivePartitionTable) {
+ rows = CommonLoadUtils.loadDataWithPartition(loadParams)
} else {
- Seq.empty
+ loadResult = CarbonDataRDDFactory.loadCarbonData(loadParams.sparkSession.sqlContext,
+ loadParams.carbonLoadModel,
+ loadParams.partitionStatus,
+ isOverwriteTable,
+ loadParams.hadoopConf,
+ None,
+ loadParams.scanResultRDD,
+ None,
+ operationContext)
}
+ (rows, loadResult)
}
override protected def opName: String = {
- if (overwrite) {
+ if (isOverwriteTable) {
"INSERT OVERWRITE"
} else {
"INSERT INTO"
}
}
+
+ private def isAlteredSchema(tableSchema: TableSchema): Boolean = {
+ if (tableInfoOp.get.getFactTable.getSchemaEvolution != null) {
+ for (entry: SchemaEvolutionEntry <- tableInfoOp
Review comment:
do not use `for` statement, use tableInfoOp....exist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-583398606
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/179/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-576102784
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1694/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573572634
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1623/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570167100
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1420/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573273445
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1593/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570565536
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194470
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
Review comment:
since this function has many parameter, add parameter names like `databaseNameOp = databaseNameOp`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376347781
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r377000854
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on issue #3538: [CARBONDATA-3637]
Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585119385
please rebase it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569658559
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1358/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569641629
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1374/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376351144
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
+ .get,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(col.getDataType),
+ timeStampFormat,
+ dateFormat)
+ }
+ }
+ }
+ val partitionColumnNames = if (partitionColumnSchema != null) {
+ partitionColumnSchema.map(x => x.getColumnName).toSet
+ } else {
+ null
+ }
+ // get invisible column indexes, alter table scenarios can have it before or after new column
+ // dummy measure will have ordinal -1 and it is invisible, ignore that column.
+ // alter table old columns are just invisible columns with proper ordinal
+ val invisibleIndex = columnSchema.filter(col => col.isInvisible && col.getSchemaOrdinal != -1)
+ .map(col => col.getSchemaOrdinal)
+ columnSchema.filterNot(col => col.isInvisible).foreach {
+ col =>
+ var skipPartitionColumn = false
+ if (col.getColumnName.contains(".")) {
+ // If the schema ordinal is -1,
+ // no need to consider it during shifting columns to derive new shifted ordinal
+ if (col.getSchemaOrdinal != -1) {
+ complexChildCount = complexChildCount + 1
+ }
+ } else {
+ // get number of invisible index count before this column
+ val invisibleIndexCount = invisibleIndex.count(index => index < col.getSchemaOrdinal)
+ if (col.getDataType.isComplexType) {
+ // Calculate re-arrange index by ignoring the complex child count.
+ // As projection will have only parent columns
+ reArrangedIndex = reArrangedIndex :+
+ (col.getSchemaOrdinal - complexChildCount - invisibleIndexCount)
+ } else {
+ if (partitionColumnNames != null && partitionColumnNames.contains(col.getColumnName)) {
+ partitionIndex = partitionIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ skipPartitionColumn = true
+ } else {
+ reArrangedIndex = reArrangedIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ }
+ }
+ if (!skipPartitionColumn) {
+ selectedColumnSchema = selectedColumnSchema :+ col
+ }
+ }
+ }
+ if (partitionColumnSchema != null) {
+ // keep partition columns in the end
+ selectedColumnSchema = selectedColumnSchema ++ partitionColumnSchema
+ }
+ if (partitionIndex.nonEmpty) {
+ // keep partition columns in the end and in the original create order
+ reArrangedIndex = reArrangedIndex ++ partitionIndex.sortBy(x => x)
+ }
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
+ if (partition.nonEmpty) {
+ // partition keyword is present in insert and
+ // select query partition projections may not be same as create order.
+ // So, bring to create table order
+ val dynamicPartition = partition.filterNot(entry => entry._2.isDefined)
+ var index = 0
+ val map = mutable.Map[String, Int]()
+ for (part <- dynamicPartition) {
+ map(part._1) = index
+ index = index + 1
+ }
+ var tempList = oldProjectionList.take(oldProjectionList.size - dynamicPartition.size)
+ val partitionList = oldProjectionList.takeRight(dynamicPartition.size)
+ val partitionSchema = table.getPartitionInfo.getColumnSchemaList.asScala
+ for (partitionCol <- partitionSchema) {
+ if (map.get(partitionCol.getColumnName).isDefined) {
+ tempList = tempList :+ partitionList(map(partitionCol.getColumnName))
+ }
+ }
+ oldProjectionList = tempList
+ }
+ if (reArrangedIndex.size != oldProjectionList.size) {
+ // for non-partition table columns must match
+ if (partition.isEmpty) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ if (reArrangedIndex.size - oldProjectionList.size != convertedStaticPartition.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ // TODO: For partition case, remaining projections need to validate ?
+ }
+ }
+ }
+ var newProjectionList: Seq[NamedExpression] = Seq.empty
+ var i = 0
+ while (i < reArrangedIndex.size) {
+ // column schema is already has sortColumns-dimensions-measures. Collect the ordinal &
+ // re-arrange the projection in the same order
+ if (partition.nonEmpty &&
+ convertedStaticPartition.contains(selectedColumnSchema(i).getColumnName
+ .toLowerCase())) {
+ // If column schema present in partitionSchema means it is a static partition,
+ // then add a value literal expression in the project.
+ val value = convertedStaticPartition(selectedColumnSchema(i).getColumnName
+ .toLowerCase())
+ newProjectionList = newProjectionList :+
+ Alias(new Literal(value,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(
+ selectedColumnSchema(i).getDataType)), value.toString)(
+ NamedExpression.newExprId,
+ None,
+ None).asInstanceOf[NamedExpression]
+ } else {
+ // If column schema NOT present in partition column,
+ // get projection column mapping its ordinal.
+ if (partition.contains(selectedColumnSchema(i).getColumnName.toLowerCase())) {
+ // static partition + dynamic partition case,
+ // here dynamic partition ordinal will be more than projection size
+ newProjectionList = newProjectionList :+
+ oldProjectionList(
+ reArrangedIndex(i) - convertedStaticPartition.size)
+ } else {
+ newProjectionList = newProjectionList :+
+ oldProjectionList(reArrangedIndex(i))
+ }
+ }
+ i = i + 1
+ }
+ processedProject = true
+ Project(newProjectionList, p.child)
+ } else {
+ p
+ }
+ }
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedSchemaLogicalRelation(reArrangedIndex: Seq[Int],
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569400483
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1320/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581543315
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1842/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569304318
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1326/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-574957397
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1649/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581495747
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1839/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586219611
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-575127856
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1662/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376357834
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/DataLoadProcessBuilder.java
##########
@@ -110,10 +113,10 @@ private AbstractDataLoadProcessorStep buildInternalForNoSort(CarbonIterator[] in
*/
private AbstractDataLoadProcessorStep buildInternalWithNoConverter(
CarbonIterator[] inputIterators, CarbonDataLoadConfiguration configuration,
- SortScopeOptions.SortScope sortScope) {
+ SortScopeOptions.SortScope sortScope, boolean withoutReArrange) {
Review comment:
That is more change, may be refactor in another PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581283852
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1830/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194756
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
Review comment:
seems a invalid comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376193657
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
Review comment:
return type should be Array[Byte]?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573288006
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1595/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on issue #3538: [CARBONDATA-3637]
Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586535639
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376341490
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -957,24 +987,36 @@ object CarbonDataRDDFactory {
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569283314
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1311/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378771218
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
+ val data = new Array[AnyRef](fields.size)
+ val badRecordLogHolder = new BadRecordLogHolder();
+ var i = 0
+ val fieldTypesLen = fields.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fields(i).dataType match {
+ case StringType =>
+ data(i) = DataTypeUtil.getBytesDataDataTypeForNoDictionaryColumn(row.getString(i),
+ DataTypes.STRING)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, arrayType), arrayType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
+ case structType : StructType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, structType), structType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[StructDataType]
+ .writeByteArray(result.asInstanceOf[StructObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
Review comment:
same as above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573643041
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1624/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194652
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
Review comment:
format the parameter list
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378690929
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
Review comment:
why StringType not go to this 'other' branch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-577632512
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1759/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584991407
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kumarvishal09 commented on a change in pull request
#3538: [WIP] Separate Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
kumarvishal09 commented on a change in pull request #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#discussion_r369374807
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -172,6 +342,7 @@ object DataLoadProcessBuilderOnSpark {
updateLoadStatus(model, partialSuccessAccum)
}
+
Review comment:
revert this change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-579049924
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1774/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376196114
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadParams.scala
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.command.UpdateTableModel
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.statusmanager.SegmentStatus
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+
+/*
+* intermediate object to pass between load functions
+*/
+case class CarbonLoadParams(sparkSession: SparkSession,
Review comment:
Will this be refactored later? We already have `CarbonLoadModel` to pass the parameters
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378720533
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
+ val data = new Array[AnyRef](fields.size)
+ val badRecordLogHolder = new BadRecordLogHolder();
+ var i = 0
+ val fieldTypesLen = fields.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fields(i).dataType match {
+ case StringType =>
+ data(i) = DataTypeUtil.getBytesDataDataTypeForNoDictionaryColumn(row.getString(i),
+ DataTypes.STRING)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, arrayType), arrayType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
+ case structType : StructType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, structType), structType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[StructDataType]
+ .writeByteArray(result.asInstanceOf[StructObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
Review comment:
reuse the method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376192708
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -179,6 +182,93 @@ object DataLoadProcessBuilderOnSpark {
updateLoadStatus(model, partialSuccessAccum)
}
+ def insertDataUsingGlobalSortWithInternalRow(sparkSession: SparkSession,
+ scanResultRDD : RDD[InternalRow],
+ model: CarbonLoadModel,
+ hadoopConf: Configuration): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
+ val originRDD = scanResultRDD
+
+ val sc = sparkSession.sparkContext
+ val modelBroadcast = sc.broadcast(model)
+ val partialSuccessAccum = sc.longAccumulator("Partial Success Accumulator")
+ val sortStepRowCounter = sc.longAccumulator("Sort Processor Accumulator")
+ val writeStepRowCounter = sc.longAccumulator("Write Processor Accumulator")
+
+ hadoopConf
+ .set(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME, sparkSession.sparkContext.appName)
+
+ val conf = SparkSQLUtil.broadCastHadoopConf(sc, hadoopConf)
+ val configuration = DataLoadProcessBuilder.createConfiguration(model)
+
+ // 1. Convert internalRow to object array
+ val fields = Util
+ .convertToSparkSchemaFromColumnSchema(model.getCarbonDataLoadSchema.getCarbonTable, true)
+ .fields
+ .toSeq
+
+ val dataTypes = fields.map(field => field.dataType)
+ val map: mutable.Map[String, GenericDataType[_]] = mutable.Map[String, GenericDataType[_]]()
+ CommonUtil.convertComplexDataType(map, configuration)
+ val rdd = originRDD.map { internalRow =>
+ CommonUtil.getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(internalRow,
+ fields,
+ map)
+ }
+
+ // 2. sort
+ var numPartitions = CarbonDataProcessorUtil.getGlobalSortPartitions(
+ configuration.getDataLoadProperty(CarbonCommonConstants.LOAD_GLOBAL_SORT_PARTITIONS))
+ if (numPartitions <= 0) {
+ numPartitions = originRDD.partitions.length
+ }
+ // Because if the number of partitions greater than 1, there will be action operator
+ // (sample) in
+ // sortBy operator. So here we cache the rdd to avoid do input and convert again.
+ if (numPartitions > 1) {
+ rdd.persist(StorageLevel.fromString(
+ CarbonProperties.getInstance().getGlobalSortRddStorageLevel()))
+ }
+ val sortColumnsLength = model.getCarbonDataLoadSchema.getCarbonTable.getSortColumns.size()
+ val sortColumnDataTypes = dataTypes.take(sortColumnsLength)
+ val rowComparator = GlobalSortHelper.generateRowComparator(sortColumnDataTypes)
+ val sortRDD = rdd.sortBy(x => getKey(x, sortColumnsLength, sortColumnDataTypes),
Review comment:
move `getKey(x, sortColumnsLength, sortColumnDataTypes)` to next line
use pattern like ```
rdd.sortBy { row =>
...
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573322241
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1602/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584037967
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1909/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376351172
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
+ .get,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(col.getDataType),
+ timeStampFormat,
+ dateFormat)
+ }
+ }
+ }
+ val partitionColumnNames = if (partitionColumnSchema != null) {
+ partitionColumnSchema.map(x => x.getColumnName).toSet
+ } else {
+ null
+ }
+ // get invisible column indexes, alter table scenarios can have it before or after new column
+ // dummy measure will have ordinal -1 and it is invisible, ignore that column.
+ // alter table old columns are just invisible columns with proper ordinal
+ val invisibleIndex = columnSchema.filter(col => col.isInvisible && col.getSchemaOrdinal != -1)
+ .map(col => col.getSchemaOrdinal)
+ columnSchema.filterNot(col => col.isInvisible).foreach {
+ col =>
+ var skipPartitionColumn = false
+ if (col.getColumnName.contains(".")) {
+ // If the schema ordinal is -1,
+ // no need to consider it during shifting columns to derive new shifted ordinal
+ if (col.getSchemaOrdinal != -1) {
+ complexChildCount = complexChildCount + 1
+ }
+ } else {
+ // get number of invisible index count before this column
+ val invisibleIndexCount = invisibleIndex.count(index => index < col.getSchemaOrdinal)
+ if (col.getDataType.isComplexType) {
+ // Calculate re-arrange index by ignoring the complex child count.
+ // As projection will have only parent columns
+ reArrangedIndex = reArrangedIndex :+
+ (col.getSchemaOrdinal - complexChildCount - invisibleIndexCount)
+ } else {
+ if (partitionColumnNames != null && partitionColumnNames.contains(col.getColumnName)) {
+ partitionIndex = partitionIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ skipPartitionColumn = true
+ } else {
+ reArrangedIndex = reArrangedIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ }
+ }
+ if (!skipPartitionColumn) {
+ selectedColumnSchema = selectedColumnSchema :+ col
+ }
+ }
+ }
+ if (partitionColumnSchema != null) {
+ // keep partition columns in the end
+ selectedColumnSchema = selectedColumnSchema ++ partitionColumnSchema
+ }
+ if (partitionIndex.nonEmpty) {
+ // keep partition columns in the end and in the original create order
+ reArrangedIndex = reArrangedIndex ++ partitionIndex.sortBy(x => x)
+ }
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
+ if (partition.nonEmpty) {
+ // partition keyword is present in insert and
+ // select query partition projections may not be same as create order.
+ // So, bring to create table order
+ val dynamicPartition = partition.filterNot(entry => entry._2.isDefined)
+ var index = 0
+ val map = mutable.Map[String, Int]()
+ for (part <- dynamicPartition) {
+ map(part._1) = index
+ index = index + 1
+ }
+ var tempList = oldProjectionList.take(oldProjectionList.size - dynamicPartition.size)
+ val partitionList = oldProjectionList.takeRight(dynamicPartition.size)
+ val partitionSchema = table.getPartitionInfo.getColumnSchemaList.asScala
+ for (partitionCol <- partitionSchema) {
+ if (map.get(partitionCol.getColumnName).isDefined) {
+ tempList = tempList :+ partitionList(map(partitionCol.getColumnName))
+ }
+ }
+ oldProjectionList = tempList
+ }
+ if (reArrangedIndex.size != oldProjectionList.size) {
+ // for non-partition table columns must match
+ if (partition.isEmpty) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ if (reArrangedIndex.size - oldProjectionList.size != convertedStaticPartition.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ // TODO: For partition case, remaining projections need to validate ?
+ }
+ }
+ }
+ var newProjectionList: Seq[NamedExpression] = Seq.empty
+ var i = 0
+ while (i < reArrangedIndex.size) {
+ // column schema is already has sortColumns-dimensions-measures. Collect the ordinal &
+ // re-arrange the projection in the same order
+ if (partition.nonEmpty &&
+ convertedStaticPartition.contains(selectedColumnSchema(i).getColumnName
+ .toLowerCase())) {
+ // If column schema present in partitionSchema means it is a static partition,
+ // then add a value literal expression in the project.
+ val value = convertedStaticPartition(selectedColumnSchema(i).getColumnName
+ .toLowerCase())
+ newProjectionList = newProjectionList :+
+ Alias(new Literal(value,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(
+ selectedColumnSchema(i).getDataType)), value.toString)(
+ NamedExpression.newExprId,
+ None,
+ None).asInstanceOf[NamedExpression]
+ } else {
+ // If column schema NOT present in partition column,
+ // get projection column mapping its ordinal.
+ if (partition.contains(selectedColumnSchema(i).getColumnName.toLowerCase())) {
+ // static partition + dynamic partition case,
+ // here dynamic partition ordinal will be more than projection size
+ newProjectionList = newProjectionList :+
+ oldProjectionList(
+ reArrangedIndex(i) - convertedStaticPartition.size)
+ } else {
+ newProjectionList = newProjectionList :+
+ oldProjectionList(reArrangedIndex(i))
+ }
+ }
+ i = i + 1
+ }
+ processedProject = true
+ Project(newProjectionList, p.child)
+ } else {
+ p
+ }
+ }
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedSchemaLogicalRelation(reArrangedIndex: Seq[Int],
+ logicalRelation: LogicalRelation): LogicalRelation = {
+ if (reArrangedIndex.size != logicalRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ logicalRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](logicalRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](logicalRelation.schema.size)
+ val fields = logicalRelation.schema.fields
+ val output = logicalRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = logicalRelation.catalogTable
+ .get
+ .copy(schema = new StructType(reArrangedFields))
+ logicalRelation.copy(logicalRelation.relation,
+ reArrangedAttributes,
+ Some(catalogTable))
+ }
+
+ def getReArrangedSchemaHiveRelation(reArrangedIndex: Seq[Int],
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569649247
Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1363/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569626224
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1353/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573989921
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1626/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378801434
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
##########
@@ -263,17 +265,35 @@ class NewDataFrameLoaderRDD[K, V](
carbonLoadModel.setPreFetch(false)
val recordReaders = mutable.Buffer[CarbonIterator[Array[AnyRef]]]()
- val partitionIterator = firstParent[DataLoadPartitionWrap[Row]].iterator(theSplit, context)
val serializer = SparkEnv.get.closureSerializer.newInstance()
var serializeBytes: Array[Byte] = null
- while(partitionIterator.hasNext) {
- val value = partitionIterator.next()
- if (serializeBytes == null) {
- serializeBytes = serializer.serialize[RDD[Row]](value.rdd).array()
+ if (isDataFrame) {
+ val partitionIterator = firstParent[DataLoadPartitionWrap[Row]].iterator(theSplit,
+ context)
+ while (partitionIterator.hasNext) {
+ val value = partitionIterator.next()
+ if (serializeBytes == null) {
+ serializeBytes = serializer.serialize[RDD[Row]](value.rdd).array()
+ }
+ recordReaders += new LazyRddIterator(serializer, serializeBytes, value.partition,
+ carbonLoadModel, context)
}
- recordReaders += new LazyRddIterator(serializer, serializeBytes, value.partition,
+ } else {
+ // For internal row, no need of converter and re-arrange step,
+ model.setLoadWithoutConverterWithoutReArrangeStep(true)
Review comment:
yes. done like that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376195809
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadParams.scala
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.command.UpdateTableModel
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.statusmanager.SegmentStatus
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+
+/*
+* intermediate object to pass between load functions
+*/
+case class CarbonLoadParams(sparkSession: SparkSession,
Review comment:
move `sparkSession: SparkSession,` to next line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585739133
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1977/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378800852
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle alter table in future, this also must use new flow.
+ if (CarbonProperties.isBadRecordHandlingEnabledForInsert ||
+ isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp = databaseNameOp,
+ tableName = tableName,
+ options = options,
+ isOverwriteTable = isOverwriteTable,
+ dimFilesPath = dimFilesPath,
+ dataFrame = dataFrame,
+ inputSqlString = inputSqlString,
+ updateModel = None,
+ tableInfoOp = tableInfoOp,
+ internalOptions = internalOptions,
+ partition = partition).process(sparkSession)
+ }
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ CarbonProperties.getInstance().addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession)
+ val optionsFinal: util.Map[String, String] = CommonLoadUtils.getFinalLoadOptions(table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf = hadoopConf,
+ factPath = factPath,
+ optionsFinal = optionsFinal,
+ parentTablePath = parentTablePath,
+ table = table,
+ isDataFrame = true,
+ internalOptions = internalOptions,
+ partition = partition,
+ options = options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = getConvertedStaticPartitionMap(partitionColumnSchema)
+ val (reArrangedIndex, selectedColumnSchema) = getReArrangedIndexAndSelectedSchema(
+ tableInfoOp.get,
+ partitionColumnSchema)
+ val newLogicalPlan = getReArrangedLogicalPlan(
+ reArrangedIndex,
+ selectedColumnSchema,
+ convertedStaticPartition)
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedLogicalPlan(
+ reArrangedIndex: Seq[Int],
+ selectedColumnSchema: Seq[ColumnSchema],
+ convertedStaticPartition: mutable.Map[String, AnyRef]): LogicalPlan = {
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378286697
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, DataFrame, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.metadata.schema.table.TableInfo
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
Review comment:
not required
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378725483
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
Review comment:
if tableInfoOp is must be defined, no need to use an option variable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376356520
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala
##########
@@ -509,30 +514,17 @@ private class CarbonOutputWriter(path: String,
// TODO Implement writesupport interface to support writing Row directly to recordwriter
def writeCarbon(row: InternalRow): Unit = {
- val data = new Array[AnyRef](fieldTypes.length + partitionData.length)
- var i = 0
- val fieldTypesLen = fieldTypes.length
- while (i < fieldTypesLen) {
- if (!row.isNullAt(i)) {
- fieldTypes(i) match {
- case StringType =>
- data(i) = row.getString(i)
- case d: DecimalType =>
- data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
- case other =>
- data(i) = row.get(i, other)
- }
- }
- i += 1
- }
+ val totalLength = fieldTypes.length + partitionData.length
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376354706
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadParams.scala
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.mutable
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{DataFrame, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.command.UpdateTableModel
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.statusmanager.SegmentStatus
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+
+/*
+* intermediate object to pass between load functions
+*/
+case class CarbonLoadParams(sparkSession: SparkSession,
Review comment:
load model will become bulky and it is used in other places also, this is only for this place
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-576697216
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1718/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376349173
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
Review comment:
it is valid comment, I will make it more understandable.
`setting a converter will remove the thread local entries of previous load configurations.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376197122
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java
##########
@@ -32,7 +33,7 @@
* Generic DataType interface which will be used while data loading for complex types like Array &
* Struct
*/
-public interface GenericDataType<T> {
+public interface GenericDataType<T> extends Serializable {
Review comment:
There are some logger in subclass, make them transient
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378722658
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -358,19 +359,41 @@ object CarbonDataRDDFactory {
}
}
} else {
- status = if (dataFrame.isEmpty && isSortTable &&
- carbonLoadModel.getRangePartitionColumn != null &&
- (sortScope.equals(SortScopeOptions.SortScope.GLOBAL_SORT) ||
- sortScope.equals(SortScopeOptions.SortScope.LOCAL_SORT))) {
- DataLoadProcessBuilderOnSpark
- .loadDataUsingRangeSort(sqlContext.sparkSession, carbonLoadModel, hadoopConf)
- } else if (isSortTable && sortScope.equals(SortScopeOptions.SortScope.GLOBAL_SORT)) {
- DataLoadProcessBuilderOnSpark.loadDataUsingGlobalSort(sqlContext.sparkSession,
- dataFrame, carbonLoadModel, hadoopConf)
- } else if (dataFrame.isDefined) {
- loadDataFrame(sqlContext, dataFrame, carbonLoadModel)
+ status = if (scanResultRdd.isDefined) {
Review comment:
now the if code block becomes bigger, better to cut some branches.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581309191
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/128/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573118720
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1592/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376471953
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
Review comment:
ok, refactored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585718233
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/274/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376196806
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala
##########
@@ -509,30 +514,17 @@ private class CarbonOutputWriter(path: String,
// TODO Implement writesupport interface to support writing Row directly to recordwriter
def writeCarbon(row: InternalRow): Unit = {
- val data = new Array[AnyRef](fieldTypes.length + partitionData.length)
- var i = 0
- val fieldTypesLen = fieldTypes.length
- while (i < fieldTypesLen) {
- if (!row.isNullAt(i)) {
- fieldTypes(i) match {
- case StringType =>
- data(i) = row.getString(i)
- case d: DecimalType =>
- data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
- case other =>
- data(i) = row.get(i, other)
- }
- }
- i += 1
- }
+ val totalLength = fieldTypes.length + partitionData.length
+ val data: Array[AnyRef] = CommonUtil.getObjectArrayFromInternalRowAndConvertComplexType(row,
+ fieldTypes,
+ totalLength)
if (partitionData.length > 0) {
- System.arraycopy(partitionData, 0, data, fieldTypesLen, partitionData.length)
+ System.arraycopy(partitionData, 0, data, fieldTypes.length, partitionData.length)
Review comment:
Is `fieldTypes.length` equal to number of non-partition columns?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569678506
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569397052
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1310/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586139194
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1984/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376194067
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -957,24 +987,36 @@ object CarbonDataRDDFactory {
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
carbonLoadModel: CarbonLoadModel
): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
try {
- val rdd = dataFrame.get.rdd
-
+ val rdd = if (dataFrame.isDefined) {
+ dataFrame.get.rdd
+ } else {
+ scanResultRDD.get
+ }
val nodeNumOfData = rdd.partitions.flatMap[String, Array[String]] { p =>
DataLoadPartitionCoalescer.getPreferredLocs(rdd, p).map(_.host)
}.distinct.length
val nodes = DistributionUtil.ensureExecutorsByNumberAndGetNodeList(
nodeNumOfData,
sqlContext.sparkContext)
- val newRdd = new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, rdd, nodes.toArray
- .distinct)
-
+ val newRdd =
+ if (dataFrame.isDefined) {
+ new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, dataFrame.get.rdd, nodes.toArray
+ .distinct)
+ } else {
+ new DataLoadCoalescedRDD[InternalRow](sqlContext.sparkSession,
+ scanResultRDD.get,
+ nodes.toArray
+ .distinct)
Review comment:
move to previous line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378801973
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, DataFrame, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.metadata.schema.table.TableInfo
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584131585
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1913/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378719858
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
+ val data = new Array[AnyRef](fields.size)
+ val badRecordLogHolder = new BadRecordLogHolder();
+ var i = 0
+ val fieldTypesLen = fields.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fields(i).dataType match {
+ case StringType =>
+ data(i) = DataTypeUtil.getBytesDataDataTypeForNoDictionaryColumn(row.getString(i),
+ DataTypes.STRING)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, arrayType), arrayType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
Review comment:
please extract the method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378724111
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -953,28 +983,46 @@ object CarbonDataRDDFactory {
/**
* Execute load process to load from input dataframe
+ *
+ * @param sqlContext sql context
+ * @param dataFrame optional dataframe for insert
+ * @param scanResultRDD optional internal row rdd for direct insert
+ * @param carbonLoadModel load model
+ * @return Return an array that contains all of the elements in NewDataFrameLoaderRDD.
*/
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
carbonLoadModel: CarbonLoadModel
): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
try {
- val rdd = dataFrame.get.rdd
-
+ val rdd = if (dataFrame.isDefined) {
+ dataFrame.get.rdd
+ } else {
+ scanResultRDD.get
+ }
val nodeNumOfData = rdd.partitions.flatMap[String, Array[String]] { p =>
DataLoadPartitionCoalescer.getPreferredLocs(rdd, p).map(_.host)
}.distinct.length
val nodes = DistributionUtil.ensureExecutorsByNumberAndGetNodeList(
nodeNumOfData,
sqlContext.sparkContext)
- val newRdd = new DataLoadCoalescedRDD[Row](sqlContext.sparkSession, rdd, nodes.toArray
- .distinct)
-
+ val newRdd =
+ if (dataFrame.isDefined) {
+ new DataLoadCoalescedRDD[Row](
+ sqlContext.sparkSession, dataFrame.get.rdd, nodes.toArray.distinct)
+ } else {
+ new DataLoadCoalescedRDD[InternalRow](
+ sqlContext.sparkSession,
+ scanResultRDD.get,
+ nodes.toArray.distinct)
+ }
new NewDataFrameLoaderRDD(
sqlContext.sparkSession,
new DataLoadResultImpl(),
carbonLoadModel,
+ dataFrame.isDefined,
Review comment:
no need this parameter if loadDataFrame method
better to set carbonLoadModel.setLoadWithoutConverterWithoutReArrangeStep
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569400849
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1333/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376193992
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -957,24 +987,36 @@ object CarbonDataRDDFactory {
private def loadDataFrame(
sqlContext: SQLContext,
dataFrame: Option[DataFrame],
+ scanResultRDD: Option[RDD[InternalRow]],
Review comment:
since you modify this function, can you add parameter description in function header
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378801875
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378771800
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -358,19 +359,41 @@ object CarbonDataRDDFactory {
}
}
} else {
- status = if (dataFrame.isEmpty && isSortTable &&
- carbonLoadModel.getRangePartitionColumn != null &&
- (sortScope.equals(SortScopeOptions.SortScope.GLOBAL_SORT) ||
- sortScope.equals(SortScopeOptions.SortScope.LOCAL_SORT))) {
- DataLoadProcessBuilderOnSpark
- .loadDataUsingRangeSort(sqlContext.sparkSession, carbonLoadModel, hadoopConf)
- } else if (isSortTable && sortScope.equals(SortScopeOptions.SortScope.GLOBAL_SORT)) {
- DataLoadProcessBuilderOnSpark.loadDataUsingGlobalSort(sqlContext.sparkSession,
- dataFrame, carbonLoadModel, hadoopConf)
- } else if (dataFrame.isDefined) {
- loadDataFrame(sqlContext, dataFrame, carbonLoadModel)
+ status = if (scanResultRdd.isDefined) {
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573300811
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1597/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585231947
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586073909
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/280/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376339644
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
Review comment:
No, not always. Primitive measure type will be object. This converting internal row to object array. Only in the write step. It will be stored as byte array
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570503917
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1440/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581278725
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/126/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376195096
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
Review comment:
this function is too big, can you separate the logic into some smaller but manageable functions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376192708
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -179,6 +182,93 @@ object DataLoadProcessBuilderOnSpark {
updateLoadStatus(model, partialSuccessAccum)
}
+ def insertDataUsingGlobalSortWithInternalRow(sparkSession: SparkSession,
+ scanResultRDD : RDD[InternalRow],
+ model: CarbonLoadModel,
+ hadoopConf: Configuration): Array[(String, (LoadMetadataDetails, ExecutionErrors))] = {
+ val originRDD = scanResultRDD
+
+ val sc = sparkSession.sparkContext
+ val modelBroadcast = sc.broadcast(model)
+ val partialSuccessAccum = sc.longAccumulator("Partial Success Accumulator")
+ val sortStepRowCounter = sc.longAccumulator("Sort Processor Accumulator")
+ val writeStepRowCounter = sc.longAccumulator("Write Processor Accumulator")
+
+ hadoopConf
+ .set(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME, sparkSession.sparkContext.appName)
+
+ val conf = SparkSQLUtil.broadCastHadoopConf(sc, hadoopConf)
+ val configuration = DataLoadProcessBuilder.createConfiguration(model)
+
+ // 1. Convert internalRow to object array
+ val fields = Util
+ .convertToSparkSchemaFromColumnSchema(model.getCarbonDataLoadSchema.getCarbonTable, true)
+ .fields
+ .toSeq
+
+ val dataTypes = fields.map(field => field.dataType)
+ val map: mutable.Map[String, GenericDataType[_]] = mutable.Map[String, GenericDataType[_]]()
+ CommonUtil.convertComplexDataType(map, configuration)
+ val rdd = originRDD.map { internalRow =>
+ CommonUtil.getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(internalRow,
+ fields,
+ map)
+ }
+
+ // 2. sort
+ var numPartitions = CarbonDataProcessorUtil.getGlobalSortPartitions(
+ configuration.getDataLoadProperty(CarbonCommonConstants.LOAD_GLOBAL_SORT_PARTITIONS))
+ if (numPartitions <= 0) {
+ numPartitions = originRDD.partitions.length
+ }
+ // Because if the number of partitions greater than 1, there will be action operator
+ // (sample) in
+ // sortBy operator. So here we cache the rdd to avoid do input and convert again.
+ if (numPartitions > 1) {
+ rdd.persist(StorageLevel.fromString(
+ CarbonProperties.getInstance().getGlobalSortRddStorageLevel()))
+ }
+ val sortColumnsLength = model.getCarbonDataLoadSchema.getCarbonTable.getSortColumns.size()
+ val sortColumnDataTypes = dataTypes.take(sortColumnsLength)
+ val rowComparator = GlobalSortHelper.generateRowComparator(sortColumnDataTypes)
+ val sortRDD = rdd.sortBy(x => getKey(x, sortColumnsLength, sortColumnDataTypes),
Review comment:
move `getKey(x, sortColumnsLength, sortColumnDataTypes)` to next line
use pattern like
```
rdd.sortBy { row =>
...
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-584060297
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/210/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586218935
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1988/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378692400
##########
File path: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala
##########
@@ -304,11 +305,11 @@ object CarbonDataRDDFactory {
def loadCarbonData(
sqlContext: SQLContext,
carbonLoadModel: CarbonLoadModel,
- columnar: Boolean,
partitionStatus: SegmentStatus = SegmentStatus.SUCCESS,
overwriteTable: Boolean,
hadoopConf: Configuration,
dataFrame: Option[DataFrame] = None,
+ scanResultRdd : Option[RDD[InternalRow]],
Review comment:
provide a default value
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573371968
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1607/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-577503959
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1746/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-571644330
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1510/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378728741
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle alter table in future, this also must use new flow.
+ if (CarbonProperties.isBadRecordHandlingEnabledForInsert ||
+ isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp = databaseNameOp,
+ tableName = tableName,
+ options = options,
+ isOverwriteTable = isOverwriteTable,
+ dimFilesPath = dimFilesPath,
+ dataFrame = dataFrame,
+ inputSqlString = inputSqlString,
+ updateModel = None,
+ tableInfoOp = tableInfoOp,
+ internalOptions = internalOptions,
+ partition = partition).process(sparkSession)
+ }
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ CarbonProperties.getInstance().addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession)
+ val optionsFinal: util.Map[String, String] = CommonLoadUtils.getFinalLoadOptions(table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf = hadoopConf,
+ factPath = factPath,
+ optionsFinal = optionsFinal,
+ parentTablePath = parentTablePath,
+ table = table,
+ isDataFrame = true,
+ internalOptions = internalOptions,
+ partition = partition,
+ options = options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = getConvertedStaticPartitionMap(partitionColumnSchema)
+ val (reArrangedIndex, selectedColumnSchema) = getReArrangedIndexAndSelectedSchema(
+ tableInfoOp.get,
+ partitionColumnSchema)
+ val newLogicalPlan = getReArrangedLogicalPlan(
+ reArrangedIndex,
+ selectedColumnSchema,
+ convertedStaticPartition)
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedLogicalPlan(
+ reArrangedIndex: Seq[Int],
+ selectedColumnSchema: Seq[ColumnSchema],
+ convertedStaticPartition: mutable.Map[String, AnyRef]): LogicalPlan = {
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
Review comment:
case p: Project if ! processedProject =>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569387151
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1317/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569283318
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1324/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376361675
##########
File path: processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java
##########
@@ -32,7 +33,7 @@
* Generic DataType interface which will be used while data loading for complex types like Array &
* Struct
*/
-public interface GenericDataType<T> {
+public interface GenericDataType<T> extends Serializable {
Review comment:
Primitive type logger was not used, removed the field itself
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581713906
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/139/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569854272
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1378/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-578829774
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1773/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-574600563
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1645/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376195414
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
+ .get,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(col.getDataType),
+ timeStampFormat,
+ dateFormat)
+ }
+ }
+ }
+ val partitionColumnNames = if (partitionColumnSchema != null) {
+ partitionColumnSchema.map(x => x.getColumnName).toSet
+ } else {
+ null
+ }
+ // get invisible column indexes, alter table scenarios can have it before or after new column
+ // dummy measure will have ordinal -1 and it is invisible, ignore that column.
+ // alter table old columns are just invisible columns with proper ordinal
+ val invisibleIndex = columnSchema.filter(col => col.isInvisible && col.getSchemaOrdinal != -1)
+ .map(col => col.getSchemaOrdinal)
+ columnSchema.filterNot(col => col.isInvisible).foreach {
+ col =>
+ var skipPartitionColumn = false
+ if (col.getColumnName.contains(".")) {
+ // If the schema ordinal is -1,
+ // no need to consider it during shifting columns to derive new shifted ordinal
+ if (col.getSchemaOrdinal != -1) {
+ complexChildCount = complexChildCount + 1
+ }
+ } else {
+ // get number of invisible index count before this column
+ val invisibleIndexCount = invisibleIndex.count(index => index < col.getSchemaOrdinal)
+ if (col.getDataType.isComplexType) {
+ // Calculate re-arrange index by ignoring the complex child count.
+ // As projection will have only parent columns
+ reArrangedIndex = reArrangedIndex :+
+ (col.getSchemaOrdinal - complexChildCount - invisibleIndexCount)
+ } else {
+ if (partitionColumnNames != null && partitionColumnNames.contains(col.getColumnName)) {
+ partitionIndex = partitionIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ skipPartitionColumn = true
+ } else {
+ reArrangedIndex = reArrangedIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ }
+ }
+ if (!skipPartitionColumn) {
+ selectedColumnSchema = selectedColumnSchema :+ col
+ }
+ }
+ }
+ if (partitionColumnSchema != null) {
+ // keep partition columns in the end
+ selectedColumnSchema = selectedColumnSchema ++ partitionColumnSchema
+ }
+ if (partitionIndex.nonEmpty) {
+ // keep partition columns in the end and in the original create order
+ reArrangedIndex = reArrangedIndex ++ partitionIndex.sortBy(x => x)
+ }
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
+ if (partition.nonEmpty) {
+ // partition keyword is present in insert and
+ // select query partition projections may not be same as create order.
+ // So, bring to create table order
+ val dynamicPartition = partition.filterNot(entry => entry._2.isDefined)
+ var index = 0
+ val map = mutable.Map[String, Int]()
+ for (part <- dynamicPartition) {
+ map(part._1) = index
+ index = index + 1
+ }
+ var tempList = oldProjectionList.take(oldProjectionList.size - dynamicPartition.size)
+ val partitionList = oldProjectionList.takeRight(dynamicPartition.size)
+ val partitionSchema = table.getPartitionInfo.getColumnSchemaList.asScala
+ for (partitionCol <- partitionSchema) {
+ if (map.get(partitionCol.getColumnName).isDefined) {
+ tempList = tempList :+ partitionList(map(partitionCol.getColumnName))
+ }
+ }
+ oldProjectionList = tempList
+ }
+ if (reArrangedIndex.size != oldProjectionList.size) {
+ // for non-partition table columns must match
+ if (partition.isEmpty) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ if (reArrangedIndex.size - oldProjectionList.size != convertedStaticPartition.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ // TODO: For partition case, remaining projections need to validate ?
+ }
+ }
+ }
+ var newProjectionList: Seq[NamedExpression] = Seq.empty
+ var i = 0
+ while (i < reArrangedIndex.size) {
+ // column schema is already has sortColumns-dimensions-measures. Collect the ordinal &
+ // re-arrange the projection in the same order
+ if (partition.nonEmpty &&
+ convertedStaticPartition.contains(selectedColumnSchema(i).getColumnName
+ .toLowerCase())) {
+ // If column schema present in partitionSchema means it is a static partition,
+ // then add a value literal expression in the project.
+ val value = convertedStaticPartition(selectedColumnSchema(i).getColumnName
+ .toLowerCase())
+ newProjectionList = newProjectionList :+
+ Alias(new Literal(value,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(
+ selectedColumnSchema(i).getDataType)), value.toString)(
+ NamedExpression.newExprId,
+ None,
+ None).asInstanceOf[NamedExpression]
+ } else {
+ // If column schema NOT present in partition column,
+ // get projection column mapping its ordinal.
+ if (partition.contains(selectedColumnSchema(i).getColumnName.toLowerCase())) {
+ // static partition + dynamic partition case,
+ // here dynamic partition ordinal will be more than projection size
+ newProjectionList = newProjectionList :+
+ oldProjectionList(
+ reArrangedIndex(i) - convertedStaticPartition.size)
+ } else {
+ newProjectionList = newProjectionList :+
+ oldProjectionList(reArrangedIndex(i))
+ }
+ }
+ i = i + 1
+ }
+ processedProject = true
+ Project(newProjectionList, p.child)
+ } else {
+ p
+ }
+ }
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedSchemaLogicalRelation(reArrangedIndex: Seq[Int],
+ logicalRelation: LogicalRelation): LogicalRelation = {
+ if (reArrangedIndex.size != logicalRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ logicalRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](logicalRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](logicalRelation.schema.size)
+ val fields = logicalRelation.schema.fields
+ val output = logicalRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = logicalRelation.catalogTable
+ .get
+ .copy(schema = new StructType(reArrangedFields))
+ logicalRelation.copy(logicalRelation.relation,
+ reArrangedAttributes,
+ Some(catalogTable))
+ }
+
+ def getReArrangedSchemaHiveRelation(reArrangedIndex: Seq[Int],
+ hiveTableRelation: HiveTableRelation): HiveTableRelation = {
+ if (reArrangedIndex.size != hiveTableRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ hiveTableRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](hiveTableRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](hiveTableRelation.schema.size)
+ val fields = hiveTableRelation.schema.fields
+ val output = hiveTableRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = hiveTableRelation.tableMeta.copy(schema = new StructType(reArrangedFields))
+ hiveTableRelation.copy(catalogTable,
+ reArrangedAttributes)
+ }
+
+ def insertData(loadParams: CarbonLoadParams): (Seq[Row], LoadMetadataDetails) = {
+ var rows = Seq.empty[Row]
+ val table = loadParams.carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
+ var loadResult : LoadMetadataDetails = null
+ if (table.isHivePartitionTable) {
+ rows = CommonLoadUtils.loadDataWithPartition(loadParams)
} else {
- Seq.empty
+ loadResult = CarbonDataRDDFactory.loadCarbonData(loadParams.sparkSession.sqlContext,
+ loadParams.carbonLoadModel,
+ loadParams.partitionStatus,
+ isOverwriteTable,
+ loadParams.hadoopConf,
+ None,
+ loadParams.scanResultRDD,
+ None,
+ operationContext)
}
+ (rows, loadResult)
}
override protected def opName: String = {
- if (overwrite) {
+ if (isOverwriteTable) {
"INSERT OVERWRITE"
} else {
"INSERT INTO"
}
}
+
+ private def isAlteredSchema(tableSchema: TableSchema): Boolean = {
+ if (tableInfoOp.get.getFactTable.getSchemaEvolution != null) {
+ for (entry: SchemaEvolutionEntry <- tableInfoOp
Review comment:
do not use `for` statement, use tableInfoOp....foreach
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570499508
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1420/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r374614202
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
Review comment:
Please remove unnecessary class level variables like these
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-585167813
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1963/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569851775
Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1368/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-579064820
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1775/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r374614556
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal,
+ parentTablePath,
+ table,
+ isDataFrame = true,
+ internalOptions,
+ finalPartition,
+ options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
Review comment:
Make dateFormat and timeStampFormat as local variables
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378034178
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal,
+ parentTablePath,
+ table,
+ isDataFrame = true,
+ internalOptions,
+ finalPartition,
+ options)
Review comment:
I tried it, some test cases fails. While sending compaction and other flows to new insert into flow. I can handle this base code refactoring.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569682742
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1361/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581508197
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/137/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-574112736
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1635/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-572479378
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1561/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569282513
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1301/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586280885
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1993/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376334978
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepWithNoConverterImpl.java
##########
@@ -334,6 +356,42 @@ private CarbonRowBatch getBatch() {
return newData;
}
+ private Object[] convertToNoDictionaryToBytesWithoutReArrange(Object[] data,
Review comment:
I checked , This is different. In original method when ordered data is present it is based on no dictionary mapping, I don't want to use no dictionary mapping.
complex type logic is same, may be I will extract a common method for that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376196613
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala
##########
@@ -509,30 +514,17 @@ private class CarbonOutputWriter(path: String,
// TODO Implement writesupport interface to support writing Row directly to recordwriter
def writeCarbon(row: InternalRow): Unit = {
- val data = new Array[AnyRef](fieldTypes.length + partitionData.length)
- var i = 0
- val fieldTypesLen = fieldTypes.length
- while (i < fieldTypesLen) {
- if (!row.isNullAt(i)) {
- fieldTypes(i) match {
- case StringType =>
- data(i) = row.getString(i)
- case d: DecimalType =>
- data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
- case other =>
- data(i) = row.get(i, other)
- }
- }
- i += 1
- }
+ val totalLength = fieldTypes.length + partitionData.length
Review comment:
Is this number of columns? suggest to change to `numColumns`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378771038
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala
##########
@@ -834,4 +843,179 @@ object CommonUtil {
displaySize
}
+ def getObjectArrayFromInternalRowAndConvertComplexType(row: InternalRow,
+ fieldTypes: Seq[DataType],
+ outputArrayLength: Int): Array[AnyRef] = {
+ val data = new Array[AnyRef](outputArrayLength)
+ var i = 0
+ val fieldTypesLen = fieldTypes.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fieldTypes(i) match {
+ case StringType =>
+ data(i) = row.getString(i)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getArray(i), arrayType)
+ case structType : StructType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getStruct(i,
+ structType.fields.length), structType)
+ case mapType : MapType =>
+ data(i) = convertSparkComplexTypeToCarbonObject(row.getMap(i), mapType)
+ case other =>
+ data(i) = row.get(i, other)
+ }
+ }
+ i += 1
+ }
+ data
+ }
+
+ /**
+ * After converting complex objects to carbon objects, need to convert to byte array
+ *
+ * @param row
+ * @param fields
+ * @param dataFieldsWithComplexDataType
+ * @return
+ */
+ def getObjectArrayFromInternalRowAndConvertComplexTypeForGlobalSort(
+ row: InternalRow,
+ fields: Seq[StructField],
+ dataFieldsWithComplexDataType: Map[String, GenericDataType[_]]): Array[AnyRef] = {
+ val data = new Array[AnyRef](fields.size)
+ val badRecordLogHolder = new BadRecordLogHolder();
+ var i = 0
+ val fieldTypesLen = fields.length
+ while (i < fieldTypesLen) {
+ if (!row.isNullAt(i)) {
+ fields(i).dataType match {
+ case StringType =>
+ data(i) = DataTypeUtil.getBytesDataDataTypeForNoDictionaryColumn(row.getString(i),
+ DataTypes.STRING)
+ case d: DecimalType =>
+ data(i) = row.getDecimal(i, d.precision, d.scale).toJavaBigDecimal
+ case arrayType : ArrayType =>
+ val result = convertSparkComplexTypeToCarbonObject(row.get(i, arrayType), arrayType)
+ // convert carbon complex object to byte array
+ val byteArray: ByteArrayOutputStream = new ByteArrayOutputStream()
+ val dataOutputStream: DataOutputStream = new DataOutputStream(byteArray)
+ dataFieldsWithComplexDataType(fields(i).name).asInstanceOf[ArrayDataType]
+ .writeByteArray(result.asInstanceOf[ArrayObject],
+ dataOutputStream,
+ badRecordLogHolder)
+ dataOutputStream.close()
+ data(i) = byteArray.toByteArray.asInstanceOf[AnyRef]
Review comment:
both are different asInstanceOf[], so method will have if, else again. not useful here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-583440865
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1882/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569674951
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1368/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376338167
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -179,6 +182,93 @@ object DataLoadProcessBuilderOnSpark {
updateLoadStatus(model, partialSuccessAccum)
}
+ def insertDataUsingGlobalSortWithInternalRow(sparkSession: SparkSession,
Review comment:
ok, I follow this guideline. Auto refactor will keep in the same line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376355884
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala
##########
@@ -316,16 +315,15 @@ private[sql] case class CarbonProjectForUpdateCommand(
val header = getHeader(carbonRelation, plan)
- CarbonLoadDataCommand(
+ CarbonInsertIntoWithDf(
Some(carbonRelation.identifier.getCarbonTableIdentifier.getDatabaseName),
carbonRelation.identifier.getCarbonTableIdentifier.getTableName,
- null,
- Seq(),
Map(("fileheader" -> header)),
false,
null,
- Some(dataFrame),
- Some(updateTableModel)).run(sparkSession)
+ dataFrame,
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376680696
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
Review comment:
ok done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569293083
Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1303/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586188469
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/285/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-579678870
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1787/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-570499381
Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1428/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581736552
@kumarvishal09 , @jackylk , @ravipesala , @QiangCai : PR is ready please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r377000735
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,518 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ var isInsertFromTable = false
+ logicalPlan.collect {
+ case _: LogicalRelation =>
+ isInsertFromTable = true
+ }
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle this in future, this must use new flow.
+ if (!isInsertFromTable || isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp,
+ tableName,
+ options,
+ isOverwriteTable,
+ dimFilesPath,
+ dataFrame,
+ inputSqlString,
+ None,
+ tableInfoOp,
+ internalOptions,
+ partition).process(sparkSession)
+ }
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf,
+ factPath,
+ optionsFinal, parentTablePath, table, isDataFrame = true, internalOptions, partition, options)
+
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(
+ carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+
+ var complexChildCount: Int = 0
+ var reArrangedIndex: Seq[Int] = Seq()
+ var selectedColumnSchema: Seq[ColumnSchema] = Seq()
+ var partitionIndex: Seq[Int] = Seq()
+
+ val columnSchema = tableInfoOp.get.getFactTable.getListOfColumns.asScala
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = mutable.Map[String, AnyRef]()
+ // Remove the thread local entries of previous configurations.
+ DataTypeUtil.setDataTypeConverter(new SparkDataTypeConverterImpl)
+ if (partition.nonEmpty) {
+ for (col <- partitionColumnSchema) {
+ if (partition(col.getColumnName.toLowerCase).isDefined) {
+ convertedStaticPartition(col.getColumnName.toLowerCase) =
+ CarbonScalaUtil.convertStaticPartitionToValues(partition(col.getColumnName.toLowerCase)
+ .get,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(col.getDataType),
+ timeStampFormat,
+ dateFormat)
+ }
+ }
+ }
+ val partitionColumnNames = if (partitionColumnSchema != null) {
+ partitionColumnSchema.map(x => x.getColumnName).toSet
+ } else {
+ null
+ }
+ // get invisible column indexes, alter table scenarios can have it before or after new column
+ // dummy measure will have ordinal -1 and it is invisible, ignore that column.
+ // alter table old columns are just invisible columns with proper ordinal
+ val invisibleIndex = columnSchema.filter(col => col.isInvisible && col.getSchemaOrdinal != -1)
+ .map(col => col.getSchemaOrdinal)
+ columnSchema.filterNot(col => col.isInvisible).foreach {
+ col =>
+ var skipPartitionColumn = false
+ if (col.getColumnName.contains(".")) {
+ // If the schema ordinal is -1,
+ // no need to consider it during shifting columns to derive new shifted ordinal
+ if (col.getSchemaOrdinal != -1) {
+ complexChildCount = complexChildCount + 1
+ }
+ } else {
+ // get number of invisible index count before this column
+ val invisibleIndexCount = invisibleIndex.count(index => index < col.getSchemaOrdinal)
+ if (col.getDataType.isComplexType) {
+ // Calculate re-arrange index by ignoring the complex child count.
+ // As projection will have only parent columns
+ reArrangedIndex = reArrangedIndex :+
+ (col.getSchemaOrdinal - complexChildCount - invisibleIndexCount)
+ } else {
+ if (partitionColumnNames != null && partitionColumnNames.contains(col.getColumnName)) {
+ partitionIndex = partitionIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ skipPartitionColumn = true
+ } else {
+ reArrangedIndex = reArrangedIndex :+ (col.getSchemaOrdinal - invisibleIndexCount)
+ }
+ }
+ if (!skipPartitionColumn) {
+ selectedColumnSchema = selectedColumnSchema :+ col
+ }
+ }
+ }
+ if (partitionColumnSchema != null) {
+ // keep partition columns in the end
+ selectedColumnSchema = selectedColumnSchema ++ partitionColumnSchema
+ }
+ if (partitionIndex.nonEmpty) {
+ // keep partition columns in the end and in the original create order
+ reArrangedIndex = reArrangedIndex ++ partitionIndex.sortBy(x => x)
+ }
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
+ var oldProjectionList = p.projectList
+ if (!processedProject) {
+ if (partition.nonEmpty) {
+ // partition keyword is present in insert and
+ // select query partition projections may not be same as create order.
+ // So, bring to create table order
+ val dynamicPartition = partition.filterNot(entry => entry._2.isDefined)
+ var index = 0
+ val map = mutable.Map[String, Int]()
+ for (part <- dynamicPartition) {
+ map(part._1) = index
+ index = index + 1
+ }
+ var tempList = oldProjectionList.take(oldProjectionList.size - dynamicPartition.size)
+ val partitionList = oldProjectionList.takeRight(dynamicPartition.size)
+ val partitionSchema = table.getPartitionInfo.getColumnSchemaList.asScala
+ for (partitionCol <- partitionSchema) {
+ if (map.get(partitionCol.getColumnName).isDefined) {
+ tempList = tempList :+ partitionList(map(partitionCol.getColumnName))
+ }
+ }
+ oldProjectionList = tempList
+ }
+ if (reArrangedIndex.size != oldProjectionList.size) {
+ // for non-partition table columns must match
+ if (partition.isEmpty) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ if (reArrangedIndex.size - oldProjectionList.size != convertedStaticPartition.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are " +
+ s"different: need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ oldProjectionList.size } columns.")
+ } else {
+ // TODO: For partition case, remaining projections need to validate ?
+ }
+ }
+ }
+ var newProjectionList: Seq[NamedExpression] = Seq.empty
+ var i = 0
+ while (i < reArrangedIndex.size) {
+ // column schema is already has sortColumns-dimensions-measures. Collect the ordinal &
+ // re-arrange the projection in the same order
+ if (partition.nonEmpty &&
+ convertedStaticPartition.contains(selectedColumnSchema(i).getColumnName
+ .toLowerCase())) {
+ // If column schema present in partitionSchema means it is a static partition,
+ // then add a value literal expression in the project.
+ val value = convertedStaticPartition(selectedColumnSchema(i).getColumnName
+ .toLowerCase())
+ newProjectionList = newProjectionList :+
+ Alias(new Literal(value,
+ SparkDataTypeConverterImpl.convertCarbonToSparkDataType(
+ selectedColumnSchema(i).getDataType)), value.toString)(
+ NamedExpression.newExprId,
+ None,
+ None).asInstanceOf[NamedExpression]
+ } else {
+ // If column schema NOT present in partition column,
+ // get projection column mapping its ordinal.
+ if (partition.contains(selectedColumnSchema(i).getColumnName.toLowerCase())) {
+ // static partition + dynamic partition case,
+ // here dynamic partition ordinal will be more than projection size
+ newProjectionList = newProjectionList :+
+ oldProjectionList(
+ reArrangedIndex(i) - convertedStaticPartition.size)
+ } else {
+ newProjectionList = newProjectionList :+
+ oldProjectionList(reArrangedIndex(i))
+ }
+ }
+ i = i + 1
+ }
+ processedProject = true
+ Project(newProjectionList, p.child)
+ } else {
+ p
+ }
+ }
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // First system has to partition the data first and then call the load data
+ LOGGER.info(s"Initiating Direct Load for the Table : ($dbName.$tableName)")
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedSchemaLogicalRelation(reArrangedIndex: Seq[Int],
+ logicalRelation: LogicalRelation): LogicalRelation = {
+ if (reArrangedIndex.size != logicalRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ logicalRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](logicalRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](logicalRelation.schema.size)
+ val fields = logicalRelation.schema.fields
+ val output = logicalRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = logicalRelation.catalogTable
+ .get
+ .copy(schema = new StructType(reArrangedFields))
+ logicalRelation.copy(logicalRelation.relation,
+ reArrangedAttributes,
+ Some(catalogTable))
+ }
+
+ def getReArrangedSchemaHiveRelation(reArrangedIndex: Seq[Int],
+ hiveTableRelation: HiveTableRelation): HiveTableRelation = {
+ if (reArrangedIndex.size != hiveTableRelation.schema.size) {
+ throw new AnalysisException(
+ s"Cannot insert into table $tableName because the number of columns are different: " +
+ s"need ${ reArrangedIndex.size } columns, " +
+ s"but query has ${ hiveTableRelation.schema.size } columns.")
+ }
+ val reArrangedFields = new Array[StructField](hiveTableRelation.schema.size)
+ val reArrangedAttributes = new Array[AttributeReference](hiveTableRelation.schema.size)
+ val fields = hiveTableRelation.schema.fields
+ val output = hiveTableRelation.output
+ var i = 0
+ for (index <- reArrangedIndex) {
+ reArrangedFields(i) = fields(index)
+ reArrangedAttributes(i) = output(index)
+ i = i + 1
+ }
+ val catalogTable = hiveTableRelation.tableMeta.copy(schema = new StructType(reArrangedFields))
+ hiveTableRelation.copy(catalogTable,
+ reArrangedAttributes)
+ }
+
+ def insertData(loadParams: CarbonLoadParams): (Seq[Row], LoadMetadataDetails) = {
+ var rows = Seq.empty[Row]
+ val table = loadParams.carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
+ var loadResult : LoadMetadataDetails = null
+ if (table.isHivePartitionTable) {
+ rows = CommonLoadUtils.loadDataWithPartition(loadParams)
} else {
- Seq.empty
+ loadResult = CarbonDataRDDFactory.loadCarbonData(loadParams.sparkSession.sqlContext,
+ loadParams.carbonLoadModel,
+ loadParams.partitionStatus,
+ isOverwriteTable,
+ loadParams.hadoopConf,
+ None,
+ loadParams.scanResultRDD,
+ None,
+ operationContext)
}
+ (rows, loadResult)
}
override protected def opName: String = {
- if (overwrite) {
+ if (isOverwriteTable) {
"INSERT OVERWRITE"
} else {
"INSERT INTO"
}
}
+
+ private def isAlteredSchema(tableSchema: TableSchema): Boolean = {
+ if (tableInfoOp.get.getFactTable.getSchemaEvolution != null) {
+ for (entry: SchemaEvolutionEntry <- tableInfoOp
Review comment:
ok done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569303524
Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1313/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-586090371
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1983/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569616794
Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1352/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r374612386
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
+ ThreadLocalSessionInfo
+ .setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ val carbonProperty: CarbonProperties = CarbonProperties.getInstance()
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ carbonProperty.addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession, carbonProperty)
+ val optionsFinal: util.Map[String, String] =
+ CommonLoadUtils.getFinalLoadOptions(
+ carbonProperty, table, options)
Review comment:
No need to pass the carbonProperties object to these methods let them handle using CarbonProperties.getInstance().
Same with setNumberOfCoresWhileLoading
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r376195656
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoWithDf.scala
##########
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import java.text.SimpleDateFormat
+import java.util
+
+import org.apache.spark.sql.{AnalysisException, CarbonUtils, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.execution.command.{DataLoadTableFileMapping, UpdateTableModel}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.util.{CarbonReflectionUtils, CausedBy, SparkUtil}
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo}
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.{CarbonDataProcessorUtil, CarbonLoaderUtil}
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+
+/*
+* insert into with df, doesn't use logical plan
+*
+*/
+case class CarbonInsertIntoWithDf(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var dataFrame: DataFrame,
+ var inputSqlString: String = null,
+ var updateModel: Option[UpdateTableModel] = None,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext) {
+
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ def process(sparkSession: SparkSession): Seq[Row] = {
Review comment:
this function is too big, separate the logic into smaller and manageable functions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581733364
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1844/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3538: [CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378800899
##########
File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -17,85 +17,517 @@
package org.apache.spark.sql.execution.command.management
-import org.apache.spark.sql.{CarbonDatasourceHadoopRelation, Dataset, Row, SparkSession}
-import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LogicalPlan}
-import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataCommand}
-import org.apache.spark.storage.StorageLevel
-
-import org.apache.carbondata.common.logging.{LogService, LogServiceFactory}
-import org.apache.carbondata.core.constants.CarbonCommonConstants
-import org.apache.carbondata.core.util.{CarbonProperties, ThreadLocalSessionInfo}
-
-case class CarbonInsertIntoCommand(
- relation: CarbonDatasourceHadoopRelation,
- child: LogicalPlan,
- overwrite: Boolean,
- partition: Map[String, Option[String]])
+import java.text.SimpleDateFormat
+import java.util
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.{AnalysisException, CarbonEnv, CarbonToSparkAdapter, DataFrame, Dataset, Row, SparkSession}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeReference, Literal, NamedExpression}
+import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project}
+import org.apache.spark.sql.execution.command.{AtomicRunnableCommand, DataLoadTableFileMapping}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.types.{StructField, StructType}
+import org.apache.spark.util.CausedBy
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.converter.SparkDataTypeConverterImpl
+import org.apache.carbondata.core.datastore.impl.FileFactory
+import org.apache.carbondata.core.indexstore.PartitionSpec
+import org.apache.carbondata.core.metadata.schema.table.{CarbonTable, TableInfo, TableSchema}
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema
+import org.apache.carbondata.core.statusmanager.{LoadMetadataDetails, SegmentStatus, SegmentStatusManager}
+import org.apache.carbondata.core.util.{CarbonProperties, DataTypeUtil, ThreadLocalSessionInfo}
+import org.apache.carbondata.core.util.path.CarbonTablePath
+import org.apache.carbondata.events.exception.PreEventException
+import org.apache.carbondata.events.OperationContext
+import org.apache.carbondata.processing.loading.TableProcessingOperations
+import org.apache.carbondata.processing.loading.exception.NoRetryException
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel
+import org.apache.carbondata.processing.util.CarbonLoaderUtil
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.util.CarbonScalaUtil
+
+/*
+* insert into without df, by just using logical plan
+*
+*/
+case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
+ tableName: String,
+ options: Map[String, String],
+ isOverwriteTable: Boolean,
+ var logicalPlan: LogicalPlan,
+ var dimFilesPath: Seq[DataLoadTableFileMapping] = Seq(),
+ var inputSqlString: String = null,
+ var tableInfoOp: Option[TableInfo] = None,
+ var internalOptions: Map[String, String] = Map.empty,
+ var partition: Map[String, Option[String]] = Map.empty,
+ var operationContext: OperationContext = new OperationContext)
extends AtomicRunnableCommand {
- var loadCommand: CarbonLoadDataCommand = _
+ var table: CarbonTable = _
+
+ var logicalPartitionRelation: LogicalRelation = _
+
+ var sizeInBytes: Long = _
+
+ var currPartitions: util.List[PartitionSpec] = _
+
+ var parentTablePath: String = _
+
+ val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
+
+ var scanResultRdd: RDD[InternalRow] = _
+
+ var timeStampFormat: SimpleDateFormat = _
+
+ var dateFormat: SimpleDateFormat = _
+
+ var finalPartition: Map[String, Option[String]] = Map.empty
+
+ var isInsertIntoWithConverterFlow: Boolean = false
+
+ var dataFrame: DataFrame = _
override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
- setAuditTable(relation.carbonTable.getDatabaseName, relation.carbonTable.getTableName)
- val LOGGER = LogServiceFactory.getLogService(this.getClass.getName)
- def containsLimit(plan: LogicalPlan): Boolean = {
- plan find {
- case limit: GlobalLimit => true
- case other => false
- } isDefined
+ if (!tableInfoOp.isDefined) {
+ throw new RuntimeException(" table info must be present when logical relation exist")
}
-
+ // If logical plan is unresolved, need to convert it to resolved.
+ dataFrame = Dataset.ofRows(sparkSession, logicalPlan)
+ logicalPlan = dataFrame.queryExecution.analyzed
+ // Currently projection re-ordering is based on schema ordinal,
+ // for some scenarios in alter table scenario, schema ordinal logic cannot be applied.
+ // So, sending it to old flow
+ // TODO: Handle alter table in future, this also must use new flow.
+ if (CarbonProperties.isBadRecordHandlingEnabledForInsert ||
+ isAlteredSchema(tableInfoOp.get.getFactTable)) {
+ isInsertIntoWithConverterFlow = true
+ }
+ if (isInsertIntoWithConverterFlow) {
+ return Seq.empty
+ }
+ setAuditTable(CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession), tableName)
ThreadLocalSessionInfo
.setConfigurationToCurrentThread(sparkSession.sessionState.newHadoopConf())
- val isPersistEnabledUserValue = CarbonProperties.getInstance
- .getProperty(CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED,
- CarbonCommonConstants.CARBON_INSERT_PERSIST_ENABLED_DEFAULT)
- val isPersistRequired =
- isPersistEnabledUserValue.equalsIgnoreCase("true") || containsLimit(child)
- val df =
- if (isPersistRequired) {
- LOGGER.info("Persist enabled for Insert operation")
- Dataset.ofRows(sparkSession, child).persist(
- StorageLevel.fromString(
- CarbonProperties.getInstance.getInsertIntoDatasetStorageLevel))
+ val (sizeInBytes, table, dbName, logicalPartitionRelation, finalPartition) = CommonLoadUtils
+ .processMetadataCommon(
+ sparkSession,
+ databaseNameOp,
+ tableName,
+ tableInfoOp,
+ partition)
+ this.sizeInBytes = sizeInBytes
+ this.table = table
+ this.logicalPartitionRelation = logicalPartitionRelation
+ this.finalPartition = finalPartition
+ setAuditTable(dbName, tableName)
+ Seq.empty
+ }
+
+ override def processData(sparkSession: SparkSession): Seq[Row] = {
+ if (isInsertIntoWithConverterFlow) {
+ return CarbonInsertIntoWithDf(
+ databaseNameOp = databaseNameOp,
+ tableName = tableName,
+ options = options,
+ isOverwriteTable = isOverwriteTable,
+ dimFilesPath = dimFilesPath,
+ dataFrame = dataFrame,
+ inputSqlString = inputSqlString,
+ updateModel = None,
+ tableInfoOp = tableInfoOp,
+ internalOptions = internalOptions,
+ partition = partition).process(sparkSession)
+ }
+ val dbName = CarbonEnv.getDatabaseName(databaseNameOp)(sparkSession)
+ val hadoopConf = sparkSession.sessionState.newHadoopConf()
+ CarbonProperties.getInstance().addProperty("zookeeper.enable.lock", "false")
+ val factPath = ""
+ currPartitions = CommonLoadUtils.getCurrentParitions(sparkSession, table)
+ CommonLoadUtils.setNumberOfCoresWhileLoading(sparkSession)
+ val optionsFinal: util.Map[String, String] = CommonLoadUtils.getFinalLoadOptions(table, options)
+ val carbonLoadModel: CarbonLoadModel = CommonLoadUtils.prepareLoadModel(
+ hadoopConf = hadoopConf,
+ factPath = factPath,
+ optionsFinal = optionsFinal,
+ parentTablePath = parentTablePath,
+ table = table,
+ isDataFrame = true,
+ internalOptions = internalOptions,
+ partition = partition,
+ options = options)
+ val (tf, df) = CommonLoadUtils.getTimeAndDateFormatFromLoadModel(carbonLoadModel)
+ timeStampFormat = tf
+ dateFormat = df
+ val partitionInfo = tableInfoOp.get.getFactTable.getPartitionInfo
+ val partitionColumnSchema =
+ if (partitionInfo != null && partitionInfo.getColumnSchemaList.size() != 0) {
+ partitionInfo.getColumnSchemaList.asScala
+ } else {
+ null
+ }
+ val convertedStaticPartition = getConvertedStaticPartitionMap(partitionColumnSchema)
+ val (reArrangedIndex, selectedColumnSchema) = getReArrangedIndexAndSelectedSchema(
+ tableInfoOp.get,
+ partitionColumnSchema)
+ val newLogicalPlan = getReArrangedLogicalPlan(
+ reArrangedIndex,
+ selectedColumnSchema,
+ convertedStaticPartition)
+ scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
+ if (logicalPartitionRelation != null) {
+ logicalPartitionRelation =
+ getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ }
+ // Delete stale segment folders that are not in table status but are physically present in
+ // the Fact folder
+ LOGGER.info(s"Deleting stale folders if present for table $dbName.$tableName")
+ TableProcessingOperations.deletePartialLoadDataIfExist(table, false)
+ var isUpdateTableStatusRequired = false
+ val uuid = ""
+ try {
+ val (tableDataMaps, dataMapOperationContext) =
+ CommonLoadUtils.firePreLoadEvents(sparkSession,
+ carbonLoadModel,
+ uuid,
+ table,
+ isOverwriteTable,
+ operationContext)
+ // Clean up the old invalid segment data before creating a new entry for new load.
+ SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false, currPartitions)
+ // add the start entry for the new load in the table status file
+ if (!table.isHivePartitionTable) {
+ CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
+ carbonLoadModel,
+ isOverwriteTable)
+ isUpdateTableStatusRequired = true
+ }
+ if (isOverwriteTable) {
+ LOGGER.info(s"Overwrite of carbon table with $dbName.$tableName is in progress")
+ }
+ // Create table and metadata folders if not exist
+ if (carbonLoadModel.isCarbonTransactionalTable) {
+ val metadataDirectoryPath = CarbonTablePath.getMetadataPath(table.getTablePath)
+ if (!FileFactory.isFileExist(metadataDirectoryPath)) {
+ FileFactory.mkdirs(metadataDirectoryPath)
+ }
} else {
- Dataset.ofRows(sparkSession, child)
+ carbonLoadModel.setSegmentId(System.nanoTime().toString)
}
- val header = relation.tableSchema.get.fields.map(_.name).mkString(",")
- loadCommand = CarbonLoadDataCommand(
- databaseNameOp = Some(relation.carbonRelation.databaseName),
- tableName = relation.carbonRelation.tableName,
- factPathFromUser = null,
- dimFilesPath = Seq(),
- options = scala.collection.immutable.Map("fileheader" -> header),
- isOverwriteTable = overwrite,
- inputSqlString = null,
- dataFrame = Some(df),
- updateModel = None,
- tableInfoOp = None,
- internalOptions = Map.empty,
- partition = partition)
- val load = loadCommand.processMetadata(sparkSession)
- if (isPersistRequired) {
- df.unpersist()
- }
- load
+ val partitionStatus = SegmentStatus.SUCCESS
+ val loadParams = CarbonLoadParams(sparkSession,
+ tableName,
+ sizeInBytes,
+ isOverwriteTable,
+ carbonLoadModel,
+ hadoopConf,
+ logicalPartitionRelation,
+ dateFormat,
+ timeStampFormat,
+ options,
+ finalPartition,
+ currPartitions,
+ partitionStatus,
+ None,
+ Some(scanResultRdd),
+ None,
+ operationContext)
+ LOGGER.info("Sort Scope : " + carbonLoadModel.getSortScope)
+ val (rows, loadResult) = insertData(loadParams)
+ val info = CommonLoadUtils.makeAuditInfo(loadResult)
+ setAuditInfo(info)
+ CommonLoadUtils.firePostLoadEvents(sparkSession,
+ carbonLoadModel,
+ tableDataMaps,
+ dataMapOperationContext,
+ table,
+ operationContext)
+ } catch {
+ case CausedBy(ex: NoRetryException) =>
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", ex)
+ throw new RuntimeException(s"Dataload failure for $dbName.$tableName, ${ex.getMessage}")
+ // In case of event related exception
+ case preEventEx: PreEventException =>
+ LOGGER.error(s"Dataload failure for $dbName.$tableName", preEventEx)
+ throw new AnalysisException(preEventEx.getMessage)
+ case ex: Exception =>
+ LOGGER.error(ex)
+ // update the load entry in table status file for changing the status to marked for delete
+ if (isUpdateTableStatusRequired) {
+ CarbonLoaderUtil.updateTableStatusForFailure(carbonLoadModel, uuid)
+ }
+ throw ex
+ }
+ Seq.empty
}
- override def processData(sparkSession: SparkSession): Seq[Row] = {
- if (null != loadCommand) {
- val rows = loadCommand.processData(sparkSession)
- setAuditInfo(loadCommand.auditInfo)
- rows
+
+ def getReArrangedLogicalPlan(
+ reArrangedIndex: Seq[Int],
+ selectedColumnSchema: Seq[ColumnSchema],
+ convertedStaticPartition: mutable.Map[String, AnyRef]): LogicalPlan = {
+ var processedProject: Boolean = false
+ // check first node is the projection or not
+ logicalPlan match {
+ case _: Project =>
+ // project is already present as first node
+ case _ =>
+ // If project is not present, add the projection to re-arrange it
+ logicalPlan = Project(logicalPlan.output, logicalPlan)
+ }
+ // Re-arrange the project as per columnSchema
+ val newLogicalPlan = logicalPlan.transformDown {
+ // case logicalRelation: LogicalRelation =>
+ // getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalRelation)
+ // case hiveRelation: HiveTableRelation =>
+ // getReArrangedSchemaHiveRelation(reArrangedIndex, hiveRelation)
+ case p: Project =>
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-569699604
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1382/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-577157879
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1738/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573543047
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1622/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Optimize
insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-581720405
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/140/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r375086020
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/steps/InputProcessorStepWithNoConverterImpl.java
##########
@@ -334,6 +356,42 @@ private CarbonRowBatch getBatch() {
return newData;
}
+ private Object[] convertToNoDictionaryToBytesWithoutReArrange(Object[] data,
Review comment:
This method is almost same as 'convertToNoDictionaryToBytes'. Better to use the withoutReArrange to skip the rearrange in that method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3538:
[CARBONDATA-3637] Optimize insert into flow
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3538: [CARBONDATA-3637] Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378715601
##########
File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
##########
@@ -263,17 +265,35 @@ class NewDataFrameLoaderRDD[K, V](
carbonLoadModel.setPreFetch(false)
val recordReaders = mutable.Buffer[CarbonIterator[Array[AnyRef]]]()
- val partitionIterator = firstParent[DataLoadPartitionWrap[Row]].iterator(theSplit, context)
val serializer = SparkEnv.get.closureSerializer.newInstance()
var serializeBytes: Array[Byte] = null
- while(partitionIterator.hasNext) {
- val value = partitionIterator.next()
- if (serializeBytes == null) {
- serializeBytes = serializer.serialize[RDD[Row]](value.rdd).array()
+ if (isDataFrame) {
+ val partitionIterator = firstParent[DataLoadPartitionWrap[Row]].iterator(theSplit,
+ context)
+ while (partitionIterator.hasNext) {
+ val value = partitionIterator.next()
+ if (serializeBytes == null) {
+ serializeBytes = serializer.serialize[RDD[Row]](value.rdd).array()
+ }
+ recordReaders += new LazyRddIterator(serializer, serializeBytes, value.partition,
+ carbonLoadModel, context)
}
- recordReaders += new LazyRddIterator(serializer, serializeBytes, value.partition,
+ } else {
+ // For internal row, no need of converter and re-arrange step,
+ model.setLoadWithoutConverterWithoutReArrangeStep(true)
Review comment:
how about set it in driver side and remove 'isDataFrame' of NewDataFrameLoaderRDD
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate
Insert and load to later optimize insert.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573278489
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1594/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services