You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/08/05 10:14:29 UTC
[GitHub] [carbondata] nihal0107 opened a new pull request #3878: [WIP]fixed insert into select hive cmd
nihal0107 opened a new pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467938413
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/util/HiveCarbonUtil.java
##########
@@ -155,7 +155,7 @@ public static CarbonLoadModel getCarbonLoadModel(String tableName, String databa
return loadModel;
}
- private static TableInfo getTableInfo(String tableName, String databaseName, String location,
+ public static TableInfo getTableInfo(String tableName, String databaseName, String location,
Review comment:
revert this change after moving getCarbonTable to HiveCarbonUtil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467937442
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -202,7 +212,7 @@ protected void setFilterPredicates(Configuration configuration, CarbonTable carb
}
private QueryModel getQueryModel(Configuration configuration, String path)
- throws IOException, InvalidConfigurationException {
+ throws IOException, InvalidConfigurationException, SQLException {
Review comment:
Which method is throwing SQLException?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-673935828
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [WIP]fixed insert into select hive cmd
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-669112179
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1876/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359209
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -202,7 +212,7 @@ protected void setFilterPredicates(Configuration configuration, CarbonTable carb
}
private QueryModel getQueryModel(Configuration configuration, String path)
- throws IOException, InvalidConfigurationException {
+ throws IOException, InvalidConfigurationException, SQLException {
Review comment:
getTableInfo() is throwing SQLException which is called from getCarbonTable().
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359455
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
.build();
}
+ private static CarbonTable getCarbonTable(Configuration tableProperties)
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [WIP]fixed insert into select hive cmd
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-669112822
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3615/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359349
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
.build();
}
+ private static CarbonTable getCarbonTable(Configuration tableProperties)
+ throws InvalidConfigurationException, IOException, SQLException {
+ String[] tableUniqueName = tableProperties.get("name").split("\\.");
+ String databaseName = tableUniqueName[0];
+ String tableName = tableUniqueName[1];
+ String tablePath = tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION);
+ String columns = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMNS);
+ String sortColumns = tableProperties.get("sort_columns");
+ String columnTypes = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMN_TYPES);
+ String partitionColumns =
+ tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMNS);
+ String partitionColumnTypes =
+ tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMN_TYPES);
+ if (partitionColumns != null) {
+ columns = columns + "," + partitionColumns;
+ columnTypes = columnTypes + ":" + partitionColumnTypes;
+ }
+ String[] columnTypeArray = HiveCarbonUtil.splitSchemaStringToArray(columnTypes);
+
+ AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
+ .from(tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION),
+ getDatabaseName(tableProperties), getTableName(tableProperties));
+ String schemaPath =
+ CarbonTablePath.getSchemaFilePath(absoluteTableIdentifier.getTablePath(), tableProperties);
+
+ CarbonTable carbonTable;
+ String carbonDataFile = CarbonUtil.getFilePathExternalFilePath(schemaPath, tableProperties);
+ if (carbonDataFile == null) {
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467937194
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
.build();
}
+ private static CarbonTable getCarbonTable(Configuration tableProperties)
+ throws InvalidConfigurationException, IOException, SQLException {
+ String[] tableUniqueName = tableProperties.get("name").split("\\.");
+ String databaseName = tableUniqueName[0];
+ String tableName = tableUniqueName[1];
+ String tablePath = tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION);
+ String columns = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMNS);
+ String sortColumns = tableProperties.get("sort_columns");
+ String columnTypes = tableProperties.get(hive_metastoreConstants.META_TABLE_COLUMN_TYPES);
+ String partitionColumns =
+ tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMNS);
+ String partitionColumnTypes =
+ tableProperties.get(hive_metastoreConstants.META_TABLE_PARTITION_COLUMN_TYPES);
+ if (partitionColumns != null) {
+ columns = columns + "," + partitionColumns;
+ columnTypes = columnTypes + ":" + partitionColumnTypes;
+ }
+ String[] columnTypeArray = HiveCarbonUtil.splitSchemaStringToArray(columnTypes);
+
+ AbsoluteTableIdentifier absoluteTableIdentifier = AbsoluteTableIdentifier
+ .from(tableProperties.get(hive_metastoreConstants.META_TABLE_LOCATION),
+ getDatabaseName(tableProperties), getTableName(tableProperties));
+ String schemaPath =
+ CarbonTablePath.getSchemaFilePath(absoluteTableIdentifier.getTablePath(), tableProperties);
+
+ CarbonTable carbonTable;
+ String carbonDataFile = CarbonUtil.getFilePathExternalFilePath(schemaPath, tableProperties);
+ if (carbonDataFile == null) {
Review comment:
This method is called only when carbondataFile does not exist. Refer: https://github.com/apache/carbondata/pull/3878/files#diff-e4da0735e7ef96dca3a7bfedb5a8039dR96
Please remove this uncessary listing and directly use the CarbonTable.buildFromTableInfo() to build carbon table.
No need for inferSchema code also
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671826659
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3693/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r468359293
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/util/HiveCarbonUtil.java
##########
@@ -155,7 +155,7 @@ public static CarbonLoadModel getCarbonLoadModel(String tableName, String databa
return loadModel;
}
- private static TableInfo getTableInfo(String tableName, String databaseName, String location,
+ public static TableInfo getTableInfo(String tableName, String databaseName, String location,
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671090835
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1929/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671090661
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3668/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#issuecomment-671828688
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1954/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on a change in pull request #3878: [CARBONDATA-3947]Fixed Hive read/write operation for Insert into Select operation.
Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3878:
URL: https://github.com/apache/carbondata/pull/3878#discussion_r467935312
##########
File path: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java
##########
@@ -213,6 +223,45 @@ private QueryModel getQueryModel(Configuration configuration, String path)
.build();
}
+ private static CarbonTable getCarbonTable(Configuration tableProperties)
Review comment:
Please move this method to HiveCarbonUtil class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org