You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/01/16 14:34:11 UTC
[GitHub] [carbondata] kunal642 opened a new pull request #3583: [WIP]
Support CarbonOutputFormat in Hive
kunal642 opened a new pull request #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583871265
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/201/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r386766357
##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/AvroCarbonWriterTest.java
##########
@@ -29,6 +29,7 @@
import org.apache.carbondata.core.constants.CarbonCommonConstants;
import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.Field;
Review comment:
Seems not required to add?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-581264479
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/7/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-592453681
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/524/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584064356
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/209/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat edited a comment on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat edited a comment on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595163508
LGTM
But all the users of SDK, need to change code (need to reimport the Field!)
As it is major version. I guess it is ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593121753
@jackylk @ravipesala @ajantha-bhat @akashrn5 Please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r387500658
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/addsegment/AddSegmentTestCase.scala
##########
@@ -758,7 +758,9 @@ class AddSegmentTestCase extends QueryTest with BeforeAndAfterAll {
val writer = CarbonWriter.builder
.outputPath(externalSegmentPath)
.writtenBy("AddSegmentTestCase")
- .withCsvInput(new Schema(fields))
+ .withSchemaFile(CarbonTablePath.getSchemaFilePath(CarbonEnv
Review comment:
format it properly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583877338
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1903/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584053618
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/208/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r387500358
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/parser/GenericParser.java
##########
@@ -32,4 +32,6 @@
*/
E parse(Object data);
+ E parseRaw(Object data);
Review comment:
Add comment for this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-575986699
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1689/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat edited a comment on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat edited a comment on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595163508
LGTM
But all the users of SDK, need to change code (need to reimport the Field!)
As it is major version. I guess it is ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595142247
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/631/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589524355
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/385/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589515724
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/382/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584131694
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1914/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] asfgit closed pull request #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589522609
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/383/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-576152481
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1699/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r387501525
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/util/HiveCarbonUtil.java
##########
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.hive.util;
+
+import java.io.File;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.carbondata.common.Strings;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.compression.CompressorFactory;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.fileoperations.FileWriteOperation;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.Field;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.metadata.schema.SchemaEvolution;
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry;
+import org.apache.carbondata.core.metadata.schema.SchemaReader;
+import org.apache.carbondata.core.metadata.schema.partition.PartitionType;
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
+import org.apache.carbondata.core.metadata.schema.table.TableInfo;
+import org.apache.carbondata.core.metadata.schema.table.TableSchema;
+import org.apache.carbondata.core.metadata.schema.table.TableSchemaBuilder;
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.core.util.OutputFilesInfoHolder;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+import org.apache.carbondata.core.writer.ThriftWriter;
+import org.apache.carbondata.processing.loading.constants.DataLoadProcessorConstants;
+import org.apache.carbondata.processing.loading.model.CarbonDataLoadSchema;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
+import org.apache.carbondata.processing.util.TableOptionConstant;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaHook;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.log4j.Logger;
+
+public class HiveCarbonUtil {
+
+ private static final Logger LOGGER =
+ LogServiceFactory.getLogService(HiveCarbonUtil.class.getName());
+
+ public static CarbonLoadModel getCarbonLoadModel(Configuration tableProperties) {
+ String[] tableUniqueName = tableProperties.get("name").split("\\.");
+ String databaseName = tableUniqueName[0];
+ String tableName = tableUniqueName[1];
+ String tablePath = tableProperties.get("location");
+ String columns = tableProperties.get("columns");
+ String sortColumns = tableProperties.get("sort_columns");
+ String columnTypes = tableProperties.get("columns.types");
+ String partitionColumns = tableProperties.get("partition_columns");
+ String partitionColumnTypes = tableProperties.get("partition_columns.types");
+ if (partitionColumns != null) {
+ columns = columns + "," + partitionColumns;
+ columnTypes = columnTypes + ":" + partitionColumnTypes;
+ }
+ String[] columnTypeArray = splitSchemaStringToArray(columnTypes);
+ String complexDelim = tableProperties.get("complex_delimiter", "");
+ CarbonLoadModel carbonLoadModel =
+ getCarbonLoadModel(tableName, databaseName, tablePath, sortColumns, columns.split(","),
+ columnTypeArray, tableProperties);
+ carbonLoadModel.setCarbonTransactionalTable(true);
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().setTransactionalTable(true);
+ for (String delim : complexDelim.split(",")) {
+ carbonLoadModel.setComplexDelimiter(delim);
+ }
+ return carbonLoadModel;
+ }
+
+ public static CarbonLoadModel getCarbonLoadModel(Properties tableProperties,
+ Configuration configuration) {
+ String[] tableUniqueName = tableProperties.getProperty("name").split("\\.");
+ String databaseName = tableUniqueName[0];
+ String tableName = tableUniqueName[1];
+ String tablePath = tableProperties.getProperty("location");
+ String columns = tableProperties.getProperty("columns");
+ String sortColumns = tableProperties.getProperty("sort_columns");
+ String[] columnTypes = splitSchemaStringToArray(tableProperties.getProperty("columns.types"));
+ String complexDelim = tableProperties.getProperty("complex_delimiter", "");
+ CarbonLoadModel carbonLoadModel =
+ getCarbonLoadModel(tableName, databaseName, tablePath, sortColumns, columns.split(","),
+ columnTypes, configuration);
+ for (String delim : complexDelim.split(",")) {
+ carbonLoadModel.setComplexDelimiter(delim);
+ }
+ return carbonLoadModel;
+ }
+
+ public static CarbonLoadModel getCarbonLoadModel(String tableName, String databaseName,
Review comment:
can you use CarbonLoadModelBuilder to build it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-594530363
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2321/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583845230
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/199/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593767537
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/568/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk edited a comment on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk edited a comment on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595246355
If not absolutely necessary, better not to move the `Field` class in sdk to core, since this is SDK API that many users have used already. Otherwise they can not compile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-576542900
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1712/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584054237
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1910/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r386766389
##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java
##########
@@ -34,6 +34,7 @@
import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
import org.apache.carbondata.core.metadata.datatype.DecimalType;
+import org.apache.carbondata.core.metadata.datatype.Field;
Review comment:
Seems not required to add?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 edited a comment on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 edited a comment on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595627546
@ajantha-bhat @jackylk it is for a major version release, i think its ok to change some API for this version.
It will be better than duplicate code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583883007
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/202/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595646137
ok, please put a note in this class to notify the user
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388133509
##########
File path: integration/hive/pom.xml
##########
@@ -177,7 +188,7 @@
<!-- Note config is repeated in scalatest config -->
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
- <argLine>-Xmx3g -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m</argLine>
+ <argLine>-Xmx3g -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kunal</argLine>
Review comment:
please revert this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584084733
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1911/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595161020
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/633/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583729531
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/182/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-592529727
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2232/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388129048
##########
File path: core/src/main/java/org/apache/carbondata/core/metadata/datatype/Field.java
##########
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.carbondata.sdk.file;
+package org.apache.carbondata.core.metadata.datatype;
Review comment:
SDK guide need to update, many examples in guide have import statement of Field
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595163508
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r386766444
##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonSchemaReaderTest.java
##########
@@ -27,6 +27,8 @@
import org.apache.carbondata.core.datastore.filesystem.CarbonFileFilter;
import org.apache.carbondata.core.datastore.impl.FileFactory;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.Field;
Review comment:
Seems not required to add?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r386766389
##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CSVCarbonWriterTest.java
##########
@@ -34,6 +34,7 @@
import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
import org.apache.carbondata.core.metadata.datatype.DecimalType;
+import org.apache.carbondata.core.metadata.datatype.Field;
Review comment:
Seems not required to add?
Please check all testcases
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583851132
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1901/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388129048
##########
File path: core/src/main/java/org/apache/carbondata/core/metadata/datatype/Field.java
##########
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.carbondata.sdk.file;
+package org.apache.carbondata.core.metadata.datatype;
Review comment:
SDK guide need to update, many examples in guide have import statement of Field
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-585101243
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/253/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-592519836
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/532/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-596199562
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595064724
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2331/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388167177
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/addsegment/AddSegmentTestCase.scala
##########
@@ -758,7 +758,9 @@ class AddSegmentTestCase extends QueryTest with BeforeAndAfterAll {
val writer = CarbonWriter.builder
.outputPath(externalSegmentPath)
.writtenBy("AddSegmentTestCase")
- .withCsvInput(new Schema(fields))
+ .withSchemaFile(CarbonTablePath.getSchemaFilePath(CarbonEnv.getCarbonTable(None,
Review comment:
why this change ? now above fields() become unused. revert it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595143308
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2338/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-590035234
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/405/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-592460178
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2224/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583827775
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1892/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388142780
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/util/HiveCarbonUtil.java
##########
@@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.hive.util;
+
+import java.io.File;
+import java.io.IOException;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.carbondata.common.Strings;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.compression.CompressorFactory;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.fileoperations.FileWriteOperation;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.Field;
+import org.apache.carbondata.core.metadata.datatype.StructField;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.metadata.schema.SchemaEvolution;
+import org.apache.carbondata.core.metadata.schema.SchemaEvolutionEntry;
+import org.apache.carbondata.core.metadata.schema.SchemaReader;
+import org.apache.carbondata.core.metadata.schema.partition.PartitionType;
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
+import org.apache.carbondata.core.metadata.schema.table.TableInfo;
+import org.apache.carbondata.core.metadata.schema.table.TableSchema;
+import org.apache.carbondata.core.metadata.schema.table.TableSchemaBuilder;
+import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.core.util.OutputFilesInfoHolder;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+import org.apache.carbondata.core.writer.ThriftWriter;
+import org.apache.carbondata.processing.loading.constants.DataLoadProcessorConstants;
+import org.apache.carbondata.processing.loading.model.CarbonDataLoadSchema;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
+import org.apache.carbondata.processing.util.TableOptionConstant;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.HiveMetaHook;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.log4j.Logger;
+
+public class HiveCarbonUtil {
+
+ private static final Logger LOGGER =
+ LogServiceFactory.getLogService(HiveCarbonUtil.class.getName());
+
+ public static CarbonLoadModel getCarbonLoadModel(Configuration tableProperties) {
+ String[] tableUniqueName = tableProperties.get("name").split("\\.");
+ String databaseName = tableUniqueName[0];
+ String tableName = tableUniqueName[1];
+ String tablePath = tableProperties.get("location");
+ String columns = tableProperties.get("columns");
+ String sortColumns = tableProperties.get("sort_columns");
+ String columnTypes = tableProperties.get("columns.types");
+ String partitionColumns = tableProperties.get("partition_columns");
+ String partitionColumnTypes = tableProperties.get("partition_columns.types");
+ if (partitionColumns != null) {
+ columns = columns + "," + partitionColumns;
+ columnTypes = columnTypes + ":" + partitionColumnTypes;
+ }
+ String[] columnTypeArray = splitSchemaStringToArray(columnTypes);
+ String complexDelim = tableProperties.get("complex_delimiter", "");
+ CarbonLoadModel carbonLoadModel =
+ getCarbonLoadModel(tableName, databaseName, tablePath, sortColumns, columns.split(","),
+ columnTypeArray, tableProperties);
+ carbonLoadModel.setCarbonTransactionalTable(true);
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().setTransactionalTable(true);
+ for (String delim : complexDelim.split(",")) {
+ carbonLoadModel.setComplexDelimiter(delim);
+ }
+ return carbonLoadModel;
+ }
+
+ public static CarbonLoadModel getCarbonLoadModel(Properties tableProperties,
+ Configuration configuration) {
+ String[] tableUniqueName = tableProperties.getProperty("name").split("\\.");
+ String databaseName = tableUniqueName[0];
+ String tableName = tableUniqueName[1];
+ String tablePath = tableProperties.getProperty("location");
+ String columns = tableProperties.getProperty("columns");
+ String sortColumns = tableProperties.getProperty("sort_columns");
+ String[] columnTypes = splitSchemaStringToArray(tableProperties.getProperty("columns.types"));
+ String complexDelim = tableProperties.getProperty("complex_delimiter", "");
+ CarbonLoadModel carbonLoadModel =
+ getCarbonLoadModel(tableName, databaseName, tablePath, sortColumns, columns.split(","),
+ columnTypes, configuration);
+ for (String delim : complexDelim.split(",")) {
+ carbonLoadModel.setComplexDelimiter(delim);
+ }
+ return carbonLoadModel;
+ }
+
+ public static CarbonLoadModel getCarbonLoadModel(String tableName, String databaseName,
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-581274029
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/125/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388148584
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -88,12 +89,23 @@ private static void populateCarbonTable(Configuration configuration, String path
}
}
}
- if (null != validInputPath) {
- AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
- .from(validInputPath, getDatabaseName(configuration), getTableName(configuration));
+ if (null != paths) {
// read the schema file to get the absoluteTableIdentifier having the correct table id
// persisted in the schema
- CarbonTable carbonTable = SchemaReader.readCarbonTableFromStore(absoluteTableIdentifier);
+ CarbonTable carbonTable;
+ AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
+ .from(validInputPath, getDatabaseName(configuration), getTableName(configuration));
+ String schemaPath = CarbonTablePath.getSchemaFilePath(validInputPath);
+ if (FileFactory.getCarbonFile(schemaPath).exists()) {
+ // read the schema file to get the absoluteTableIdentifier having the correct table id
+ // persisted in the schema
+ carbonTable = SchemaReader.readCarbonTableFromStore(absoluteTableIdentifier);
+ } else {
+ // InferSchema from data file
+ carbonTable = CarbonTable.buildFromTableInfo(SchemaReader
+ .inferSchema(absoluteTableIdentifier, false));
+ carbonTable.setTransactionalTable(true);
Review comment:
If shema doesn't exists, will it be non-transactional table ?
can you add comment why it is set as transactional ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388167171
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
}
@Override
- public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf)
- throws IOException {
- org.apache.hadoop.mapreduce.JobContext jobContext = Job.getInstance(jobConf);
- super.checkOutputSpecs(jobContext);
+ public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf) throws IOException {
}
@Override
public FileSinkOperator.RecordWriter getHiveRecordWriter(JobConf jc, Path finalOutPath,
Class<? extends Writable> valueClass, boolean isCompressed, Properties tableProperties,
- Progressable progress) {
- return null;
+ Progressable progress) throws IOException {
+ CarbonLoadModel carbonLoadModel = null;
+ String encodedString = jc.get(LOAD_MODEL);
+ if (encodedString != null) {
+ carbonLoadModel =
+ (CarbonLoadModel) ObjectSerializationUtil.convertStringToObject(encodedString);
+ }
+ if (carbonLoadModel == null) {
+ carbonLoadModel = HiveCarbonUtil.getCarbonLoadModel(tableProperties, jc);
+ } else {
+ for (Map.Entry<Object, Object> entry : tableProperties.entrySet()) {
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().getTableInfo().getFactTable()
+ .getTableProperties().put(entry.getKey().toString().toLowerCase(),
+ entry.getValue().toString().toLowerCase());
+ }
+ }
+ String tablePath = FileFactory.getCarbonFile(carbonLoadModel.getTablePath()).getAbsolutePath();
+ TaskAttemptID taskAttemptID = TaskAttemptID.forName(jc.get("mapred.task.id"));
+ TaskAttemptContextImpl context = new TaskAttemptContextImpl(jc, taskAttemptID);
+ final boolean isHivePartitionedTable =
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().isHivePartitionTable();
+ PartitionInfo partitionInfo =
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().getPartitionInfo();
+ final int partitionColumn =
+ partitionInfo != null ? partitionInfo.getColumnSchemaList().size() : 0;
+ String finalOutputPath = FileFactory.getCarbonFile(finalOutPath.toString()).getAbsolutePath();
+ if (carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().isHivePartitionTable()) {
+ carbonLoadModel.getOutputFilesInfoHolder().addToPartitionPath(finalOutputPath);
+ context.getConfiguration().set("carbon.outputformat.writepath", finalOutputPath);
+ }
+ CarbonTableOutputFormat.setLoadModel(context.getConfiguration(), carbonLoadModel);
+ org.apache.hadoop.mapreduce.RecordWriter<NullWritable, ObjectArrayWritable> re =
+ super.getRecordWriter(context);
+ return new FileSinkOperator.RecordWriter() {
+ @Override
+ public void write(Writable writable) throws IOException {
+ try {
+ ObjectArrayWritable objectArrayWritable = new ObjectArrayWritable();
+ if (isHivePartitionedTable) {
+ Object[] actualRow = ((CarbonHiveRow) writable).getData();
+ Object[] newData = Arrays.copyOf(actualRow, actualRow.length + partitionColumn);
+ String[] partitionValues = finalOutputPath.substring(tablePath.length()).split("/");
Review comment:
/ in partition would be converted to '%2' when creating the partition folder, so it wount fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388213194
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
}
@Override
- public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf)
- throws IOException {
- org.apache.hadoop.mapreduce.JobContext jobContext = Job.getInstance(jobConf);
- super.checkOutputSpecs(jobContext);
+ public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf) throws IOException {
}
@Override
public FileSinkOperator.RecordWriter getHiveRecordWriter(JobConf jc, Path finalOutPath,
Class<? extends Writable> valueClass, boolean isCompressed, Properties tableProperties,
- Progressable progress) {
- return null;
+ Progressable progress) throws IOException {
+ CarbonLoadModel carbonLoadModel = null;
+ String encodedString = jc.get(LOAD_MODEL);
+ if (encodedString != null) {
+ carbonLoadModel =
+ (CarbonLoadModel) ObjectSerializationUtil.convertStringToObject(encodedString);
+ }
+ if (carbonLoadModel == null) {
+ carbonLoadModel = HiveCarbonUtil.getCarbonLoadModel(tableProperties, jc);
+ } else {
+ for (Map.Entry<Object, Object> entry : tableProperties.entrySet()) {
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().getTableInfo().getFactTable()
+ .getTableProperties().put(entry.getKey().toString().toLowerCase(),
+ entry.getValue().toString().toLowerCase());
+ }
+ }
+ String tablePath = FileFactory.getCarbonFile(carbonLoadModel.getTablePath()).getAbsolutePath();
+ TaskAttemptID taskAttemptID = TaskAttemptID.forName(jc.get("mapred.task.id"));
+ TaskAttemptContextImpl context = new TaskAttemptContextImpl(jc, taskAttemptID);
+ final boolean isHivePartitionedTable =
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().isHivePartitionTable();
+ PartitionInfo partitionInfo =
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().getPartitionInfo();
+ final int partitionColumn =
+ partitionInfo != null ? partitionInfo.getColumnSchemaList().size() : 0;
+ String finalOutputPath = FileFactory.getCarbonFile(finalOutPath.toString()).getAbsolutePath();
+ if (carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().isHivePartitionTable()) {
+ carbonLoadModel.getOutputFilesInfoHolder().addToPartitionPath(finalOutputPath);
+ context.getConfiguration().set("carbon.outputformat.writepath", finalOutputPath);
+ }
+ CarbonTableOutputFormat.setLoadModel(context.getConfiguration(), carbonLoadModel);
+ org.apache.hadoop.mapreduce.RecordWriter<NullWritable, ObjectArrayWritable> re =
+ super.getRecordWriter(context);
+ return new FileSinkOperator.RecordWriter() {
+ @Override
+ public void write(Writable writable) throws IOException {
+ try {
+ ObjectArrayWritable objectArrayWritable = new ObjectArrayWritable();
+ if (isHivePartitionedTable) {
+ Object[] actualRow = ((CarbonHiveRow) writable).getData();
+ Object[] newData = Arrays.copyOf(actualRow, actualRow.length + partitionColumn);
+ String[] partitionValues = finalOutputPath.substring(tablePath.length()).split("/");
Review comment:
yeah ok, I remember now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r387500926
##########
File path: integration/hive/src/test/java/org/apache/carbondata/hive/HiveTestUtils.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import java.io.File;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.carbondata.hive.test.server.HiveEmbeddedServer2;
+
+import org.junit.Assert;
+
+/**
+ * A utility class to start and stop the Hive Embedded Server.
+ */
+public abstract class HiveTestUtils {
+
+ private static Connection connection;
+
+ private static HiveEmbeddedServer2 hiveEmbeddedServer2;
+
+ public HiveTestUtils() {
+ }
+
+ private static void setup() {
+ try {
+ File rootPath = new File(HiveTestUtils.class.getResource("/").getPath() + "../../../..");
+ String targetLoc = rootPath.getAbsolutePath() + "/integration/hive/target/warehouse";
+ hiveEmbeddedServer2 = new HiveEmbeddedServer2();
+ hiveEmbeddedServer2.start(targetLoc);
+ int port = hiveEmbeddedServer2.getFreePort();
+ connection = DriverManager.getConnection("jdbc:hive2://localhost:" + port + "/default", "", "");
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ static Connection getConnection() {
+ if (connection == null) {
+ setup();
+ tearDown();
+ }
+ return connection;
+ }
+
+ public static void tearDown() {
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ try {
+ connection.close();
+ hiveEmbeddedServer2.stop();
+ } catch (SQLException e) {
+ throw new RuntimeException("Unable to close Hive Embedded Server", e);
+ }
+ }));
+ }
+
+ public String getFieldValue(ResultSet rs, String field) throws Exception {
+ while (rs.next()) {
+ System.out.println(rs.getString(1));
+ System.out.println("-- " + rs.getString(2));
+ if (rs.getString(1).toLowerCase().contains(field.toLowerCase())) {
+ return rs.getString(2);
+ }
+ }
+ return "";
+ }
+
+ public boolean checkAnswer(ResultSet actual, ResultSet expected) throws SQLException {
+ Assert.assertEquals("Row Count Mismatch: ", expected.getFetchSize(), actual.getFetchSize());
+ while(expected.next()) {
Review comment:
add space before (
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-592423693
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2221/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r387500358
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/parser/GenericParser.java
##########
@@ -32,4 +32,6 @@
*/
E parse(Object data);
+ E parseRaw(Object data);
Review comment:
Add comment for this. What is the difference with `parse`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388206618
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/addsegment/AddSegmentTestCase.scala
##########
@@ -758,7 +758,9 @@ class AddSegmentTestCase extends QueryTest with BeforeAndAfterAll {
val writer = CarbonWriter.builder
.outputPath(externalSegmentPath)
.writtenBy("AddSegmentTestCase")
- .withCsvInput(new Schema(fields))
+ .withSchemaFile(CarbonTablePath.getSchemaFilePath(CarbonEnv.getCarbonTable(None,
Review comment:
reverted
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388156847
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
}
@Override
- public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf)
- throws IOException {
- org.apache.hadoop.mapreduce.JobContext jobContext = Job.getInstance(jobConf);
- super.checkOutputSpecs(jobContext);
+ public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf) throws IOException {
}
@Override
public FileSinkOperator.RecordWriter getHiveRecordWriter(JobConf jc, Path finalOutPath,
Class<? extends Writable> valueClass, boolean isCompressed, Properties tableProperties,
- Progressable progress) {
- return null;
+ Progressable progress) throws IOException {
+ CarbonLoadModel carbonLoadModel = null;
+ String encodedString = jc.get(LOAD_MODEL);
+ if (encodedString != null) {
+ carbonLoadModel =
+ (CarbonLoadModel) ObjectSerializationUtil.convertStringToObject(encodedString);
+ }
+ if (carbonLoadModel == null) {
+ carbonLoadModel = HiveCarbonUtil.getCarbonLoadModel(tableProperties, jc);
+ } else {
+ for (Map.Entry<Object, Object> entry : tableProperties.entrySet()) {
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().getTableInfo().getFactTable()
+ .getTableProperties().put(entry.getKey().toString().toLowerCase(),
+ entry.getValue().toString().toLowerCase());
+ }
+ }
+ String tablePath = FileFactory.getCarbonFile(carbonLoadModel.getTablePath()).getAbsolutePath();
+ TaskAttemptID taskAttemptID = TaskAttemptID.forName(jc.get("mapred.task.id"));
+ TaskAttemptContextImpl context = new TaskAttemptContextImpl(jc, taskAttemptID);
+ final boolean isHivePartitionedTable =
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().isHivePartitionTable();
+ PartitionInfo partitionInfo =
+ carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().getPartitionInfo();
+ final int partitionColumn =
+ partitionInfo != null ? partitionInfo.getColumnSchemaList().size() : 0;
+ String finalOutputPath = FileFactory.getCarbonFile(finalOutPath.toString()).getAbsolutePath();
+ if (carbonLoadModel.getCarbonDataLoadSchema().getCarbonTable().isHivePartitionTable()) {
+ carbonLoadModel.getOutputFilesInfoHolder().addToPartitionPath(finalOutputPath);
+ context.getConfiguration().set("carbon.outputformat.writepath", finalOutputPath);
+ }
+ CarbonTableOutputFormat.setLoadModel(context.getConfiguration(), carbonLoadModel);
+ org.apache.hadoop.mapreduce.RecordWriter<NullWritable, ObjectArrayWritable> re =
+ super.getRecordWriter(context);
+ return new FileSinkOperator.RecordWriter() {
+ @Override
+ public void write(Writable writable) throws IOException {
+ try {
+ ObjectArrayWritable objectArrayWritable = new ObjectArrayWritable();
+ if (isHivePartitionedTable) {
+ Object[] actualRow = ((CarbonHiveRow) writable).getData();
+ Object[] newData = Arrays.copyOf(actualRow, actualRow.length + partitionColumn);
+ String[] partitionValues = finalOutputPath.substring(tablePath.length()).split("/");
Review comment:
what if the partition value itself contains "/" or "=" ? can happen if partition table is string column !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583370516
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1881/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388154014
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
}
@Override
- public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf)
- throws IOException {
- org.apache.hadoop.mapreduce.JobContext jobContext = Job.getInstance(jobConf);
- super.checkOutputSpecs(jobContext);
+ public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf) throws IOException {
}
@Override
public FileSinkOperator.RecordWriter getHiveRecordWriter(JobConf jc, Path finalOutPath,
Class<? extends Writable> valueClass, boolean isCompressed, Properties tableProperties,
- Progressable progress) {
- return null;
+ Progressable progress) throws IOException {
+ CarbonLoadModel carbonLoadModel = null;
+ String encodedString = jc.get(LOAD_MODEL);
+ if (encodedString != null) {
+ carbonLoadModel =
+ (CarbonLoadModel) ObjectSerializationUtil.convertStringToObject(encodedString);
+ }
+ if (carbonLoadModel == null) {
+ carbonLoadModel = HiveCarbonUtil.getCarbonLoadModel(tableProperties, jc);
Review comment:
better to keep it in `else` case of `if (encodedString != null)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388164466
##########
File path: integration/hive/src/test/java/org/apache/carbondata/hive/HiveCarbonTest.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.SchemaReader;
+import org.apache.carbondata.core.util.CarbonProperties;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class HiveCarbonTest extends HiveTestUtils {
+
+ private static Statement statement;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT_DEFAULT, "false");
+ CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "false");
+ CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME, "hive");
Review comment:
Please set back these values to default at the end of test case, to avoid impact on existing test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595063366
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/625/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388152804
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
}
@Override
- public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf)
- throws IOException {
- org.apache.hadoop.mapreduce.JobContext jobContext = Job.getInstance(jobConf);
- super.checkOutputSpecs(jobContext);
+ public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf) throws IOException {
}
Review comment:
can add some comment, why no functionality required here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-585109872
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/254/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589506071
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2081/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388142709
##########
File path: integration/hive/pom.xml
##########
@@ -177,7 +188,7 @@
<!-- Note config is repeated in scalatest config -->
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
- <argLine>-Xmx3g -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m</argLine>
+ <argLine>-Xmx3g -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kunal</argLine>
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589524824
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2086/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583955161
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/204/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589505865
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/379/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583723271
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1883/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583827649
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/190/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388134906
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/CarbonHiveRecordReader.java
##########
@@ -106,6 +86,7 @@ private void initialize(InputSplit inputSplit, Configuration conf) throws IOExce
int columnId = 0;
for (int j = 0; j < arraySelectedColId.length; j++) {
columnId = Integer.parseInt(arraySelectedColId[j]);
+
Review comment:
revert this and remove local variable `columnId`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r386767350
##########
File path: core/src/main/java/org/apache/carbondata/core/metadata/datatype/Field.java
##########
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.carbondata.sdk.file;
+package org.apache.carbondata.core.metadata.datatype;
Review comment:
Why move this package? Originally, this class is intended stay in SDK module so that user only uses API in carbondata-sdk module instead of calling functions in carbondata-core module
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-594461313
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/606/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593813062
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2273/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593954531
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/585/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388158515
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -88,12 +89,23 @@ private static void populateCarbonTable(Configuration configuration, String path
}
}
}
- if (null != validInputPath) {
- AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
- .from(validInputPath, getDatabaseName(configuration), getTableName(configuration));
+ if (null != paths) {
// read the schema file to get the absoluteTableIdentifier having the correct table id
// persisted in the schema
- CarbonTable carbonTable = SchemaReader.readCarbonTableFromStore(absoluteTableIdentifier);
+ CarbonTable carbonTable;
+ AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
+ .from(validInputPath, getDatabaseName(configuration), getTableName(configuration));
+ String schemaPath = CarbonTablePath.getSchemaFilePath(validInputPath);
+ if (FileFactory.getCarbonFile(schemaPath).exists()) {
+ // read the schema file to get the absoluteTableIdentifier having the correct table id
+ // persisted in the schema
+ carbonTable = SchemaReader.readCarbonTableFromStore(absoluteTableIdentifier);
+ } else {
+ // InferSchema from data file
+ carbonTable = CarbonTable.buildFromTableInfo(SchemaReader
+ .inferSchema(absoluteTableIdentifier, false));
+ carbonTable.setTransactionalTable(true);
Review comment:
it should be false.. changed it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-575655145
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1682/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584229487
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/222/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r386795859
##########
File path: core/src/main/java/org/apache/carbondata/core/metadata/datatype/Field.java
##########
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.carbondata.sdk.file;
+package org.apache.carbondata.core.metadata.datatype;
Review comment:
Actually Field class already has the logic to parse complex child types, therefore instead of duplicating the code its better to move this class to core and use in both flows.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589567554
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/387/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388142737
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/CarbonHiveRecordReader.java
##########
@@ -106,6 +86,7 @@ private void initialize(InputSplit inputSplit, Configuration conf) throws IOExce
int columnId = 0;
for (int j = 0; j < arraySelectedColId.length; j++) {
columnId = Integer.parseInt(arraySelectedColId[j]);
+
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593788995
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/569/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388161829
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
}
@Override
- public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf)
- throws IOException {
- org.apache.hadoop.mapreduce.JobContext jobContext = Job.getInstance(jobConf);
- super.checkOutputSpecs(jobContext);
+ public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf) throws IOException {
}
Review comment:
not needed for us
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583883191
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1904/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584109035
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/212/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388167385
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
}
@Override
- public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf)
- throws IOException {
- org.apache.hadoop.mapreduce.JobContext jobContext = Job.getInstance(jobConf);
- super.checkOutputSpecs(jobContext);
+ public void checkOutputSpecs(FileSystem fileSystem, JobConf jobConf) throws IOException {
}
@Override
public FileSinkOperator.RecordWriter getHiveRecordWriter(JobConf jc, Path finalOutPath,
Class<? extends Writable> valueClass, boolean isCompressed, Properties tableProperties,
- Progressable progress) {
- return null;
+ Progressable progress) throws IOException {
+ CarbonLoadModel carbonLoadModel = null;
+ String encodedString = jc.get(LOAD_MODEL);
+ if (encodedString != null) {
+ carbonLoadModel =
+ (CarbonLoadModel) ObjectSerializationUtil.convertStringToObject(encodedString);
+ }
+ if (carbonLoadModel == null) {
+ carbonLoadModel = HiveCarbonUtil.getCarbonLoadModel(tableProperties, jc);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593299916
please rebase and update the detail description
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-575492926
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1676/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388159518
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
Review comment:
it do not matter because this API is not called
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595719080
can we still keep same package name inside core? so user need not have to change code ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583967776
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1906/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-594529435
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/614/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583727207
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/181/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-575182754
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1669/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583995968
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1908/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat edited a comment on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat edited a comment on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595719080
can we still keep same package name inside core? so user need not have to change code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589582588
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2088/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-579729895
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1793/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-590046822
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2106/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-594467878
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2313/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-576235325
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1704/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388142864
##########
File path: processing/src/main/java/org/apache/carbondata/processing/loading/parser/GenericParser.java
##########
@@ -32,4 +32,6 @@
*/
E parse(Object data);
+ E parseRaw(Object data);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593766907
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2272/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-584254883
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1924/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589514661
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/381/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589514877
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2083/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-579731431
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1790/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595769202
@ajantha-bhat it dosent make sense to make a package names 'sdk' in core module. I think it is okay like this, we can put this as a breaking change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388142806
##########
File path: integration/hive/src/test/java/org/apache/carbondata/hive/HiveTestUtils.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import java.io.File;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+import org.apache.carbondata.hive.test.server.HiveEmbeddedServer2;
+
+import org.junit.Assert;
+
+/**
+ * A utility class to start and stop the Hive Embedded Server.
+ */
+public abstract class HiveTestUtils {
+
+ private static Connection connection;
+
+ private static HiveEmbeddedServer2 hiveEmbeddedServer2;
+
+ public HiveTestUtils() {
+ }
+
+ private static void setup() {
+ try {
+ File rootPath = new File(HiveTestUtils.class.getResource("/").getPath() + "../../../..");
+ String targetLoc = rootPath.getAbsolutePath() + "/integration/hive/target/warehouse";
+ hiveEmbeddedServer2 = new HiveEmbeddedServer2();
+ hiveEmbeddedServer2.start(targetLoc);
+ int port = hiveEmbeddedServer2.getFreePort();
+ connection = DriverManager.getConnection("jdbc:hive2://localhost:" + port + "/default", "", "");
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ static Connection getConnection() {
+ if (connection == null) {
+ setup();
+ tearDown();
+ }
+ return connection;
+ }
+
+ public static void tearDown() {
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ try {
+ connection.close();
+ hiveEmbeddedServer2.stop();
+ } catch (SQLException e) {
+ throw new RuntimeException("Unable to close Hive Embedded Server", e);
+ }
+ }));
+ }
+
+ public String getFieldValue(ResultSet rs, String field) throws Exception {
+ while (rs.next()) {
+ System.out.println(rs.getString(1));
+ System.out.println("-- " + rs.getString(2));
+ if (rs.getString(1).toLowerCase().contains(field.toLowerCase())) {
+ return rs.getString(2);
+ }
+ }
+ return "";
+ }
+
+ public boolean checkAnswer(ResultSet actual, ResultSet expected) throws SQLException {
+ Assert.assertEquals("Row Count Mismatch: ", expected.getFetchSize(), actual.getFetchSize());
+ while(expected.next()) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ydvpankaj99 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
ydvpankaj99 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-581271072
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595161465
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2340/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-576119537
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1697/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583719152
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/180/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-576023794
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1693/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589686328
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/393/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583735058
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1885/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-592422981
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/522/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583971855
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/205/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583371523
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/178/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-577503076
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1747/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-585135292
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1957/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388142834
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/addsegment/AddSegmentTestCase.scala
##########
@@ -758,7 +758,9 @@ class AddSegmentTestCase extends QueryTest with BeforeAndAfterAll {
val writer = CarbonWriter.builder
.outputPath(externalSegmentPath)
.writtenBy("AddSegmentTestCase")
- .withCsvInput(new Schema(fields))
+ .withSchemaFile(CarbonTablePath.getSchemaFilePath(CarbonEnv
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589711199
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2094/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3583: [CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388152290
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonOutputFormat.java
##########
@@ -18,43 +18,115 @@
package org.apache.carbondata.hive;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
import java.util.Properties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.PartitionInfo;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
+import org.apache.carbondata.core.util.ThreadLocalSessionInfo;
import org.apache.carbondata.hadoop.api.CarbonTableOutputFormat;
+import org.apache.carbondata.hadoop.internal.ObjectArrayWritable;
+import org.apache.carbondata.hive.util.HiveCarbonUtil;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.FileSinkOperator;
import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.OutputFormat;
import org.apache.hadoop.mapred.RecordWriter;
-import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.Progressable;
-/**
- * TODO : To extend CarbonOutputFormat
- */
public class MapredCarbonOutputFormat<T> extends CarbonTableOutputFormat
- implements HiveOutputFormat<Void, T> {
+ implements HiveOutputFormat<Void, T>, OutputFormat<Void, T> {
@Override
public RecordWriter<Void, T> getRecordWriter(FileSystem fileSystem, JobConf jobConf, String s,
- Progressable progressable) {
- return null;
+ Progressable progressable) throws IOException {
+ throw new RuntimeException("Should never be used");
Review comment:
better to throw unsupported or more meaningful exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583981872
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/206/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-596199554
ok, please remember to mention this in 2.0's release note
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r386766407
##########
File path: sdk/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java
##########
@@ -33,6 +33,7 @@
import org.apache.carbondata.core.datastore.impl.FileFactory;
import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.Field;
Review comment:
Seems not required to add?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-581274095
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1829/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-589677737
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-593993693
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2291/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3583: [WIP] Support
CarbonOutputFormat in Hive
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3583: [WIP] Support CarbonOutputFormat in Hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-583727305
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1884/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595246355
Better not to move the `Field` class in sdk to core, since this is SDK API that many users have used already. Otherwise they can not compile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on issue #3583: [CARBONDATA-3687]
Support writing non-transactional carbondata files through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on issue #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#issuecomment-595627546
@ajantha-bhat @jackylk it is for a major version release, i think its ok to change some API for this version.
It will be better than duplicate code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3583:
[CARBONDATA-3687] Support writing non-transactional carbondata files
through hive
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3583: [CARBONDATA-3687] Support writing non-transactional carbondata files through hive
URL: https://github.com/apache/carbondata/pull/3583#discussion_r388206653
##########
File path: integration/hive/src/test/java/org/apache/carbondata/hive/HiveCarbonTest.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import java.sql.ResultSet;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.filesystem.CarbonFile;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.metadata.schema.SchemaReader;
+import org.apache.carbondata.core.util.CarbonProperties;
+import org.apache.carbondata.core.util.path.CarbonTablePath;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class HiveCarbonTest extends HiveTestUtils {
+
+ private static Statement statement;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+ CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_OFFHEAP_SORT_DEFAULT, "false");
+ CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "false");
+ CarbonProperties.getInstance().addProperty(CarbonCommonConstants.CARBON_WRITTEN_BY_APPNAME, "hive");
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services