You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/02/12 10:39:17 UTC
[GitHub] [carbondata] ajantha-bhat opened a new pull request #3615: [WIP]
Send insert stage, compaction to new insert into flow
ajantha-bhat opened a new pull request #3615: [WIP] Send insert stage, compaction to new insert into flow
URL: https://github.com/apache/carbondata/pull/3615
This PR depends on #3538
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615: [WIP]
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP] [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589082225
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/372/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383378326
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
##########
@@ -728,7 +728,11 @@ object CommonLoadUtils {
}
val updatedRdd: RDD[InternalRow] = rdd.map { internalRow =>
for (index <- timeStampIndex) {
- internalRow.setLong(index, internalRow.getLong(index) / 1000)
+ if (internalRow.getLong(index) == 0) {
+ internalRow.setNullAt(index)
+ } else {
+ internalRow.setLong(index, internalRow.getLong(index) / 1000)
Review comment:
It is a time stamp local granularity , Let me define it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590153543
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/414/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587571377
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/346/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590636742
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589498137
@QiangCai , @kunal642 , @jackylk :PR is ready. please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615: [WIP] Send
insert stage, compaction to new insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP] Send insert stage, compaction to new insert into flow
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-585151986
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/262/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590424436
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/442/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383369387
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -176,8 +176,21 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
convertedStaticPartition)
scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
if (logicalPartitionRelation != null) {
- logicalPartitionRelation =
- getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ if (selectedColumnSchema.length != logicalPartitionRelation.output.length) {
+ throw new RuntimeException(" schema length doesn't match partition length")
+ }
+ var isAlreadyReArranged = true
+ var index = 0
+ for (col: ColumnSchema <- selectedColumnSchema) {
Review comment:
Please use lambda function instead of `for` loop, findFirst?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] asfgit closed pull request #3615: [CARBONDATA-3637]
Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590163981
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2115/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590464854
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2142/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383657139
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
##########
@@ -728,7 +728,13 @@ object CommonLoadUtils {
}
val updatedRdd: RDD[InternalRow] = rdd.map { internalRow =>
for (index <- timeStampIndex) {
- internalRow.setLong(index, internalRow.getLong(index) / 1000)
+ if (internalRow.getLong(index) == 0) {
Review comment:
because timestamp is not direct dictionary, only date is direct dictionary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-586549542
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/298/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590642360
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/448/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590690777
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/450/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] QiangCai commented on a change in pull request #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383615563
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
##########
@@ -728,7 +728,13 @@ object CommonLoadUtils {
}
val updatedRdd: RDD[InternalRow] = rdd.map { internalRow =>
for (index <- timeStampIndex) {
- internalRow.setLong(index, internalRow.getLong(index) / 1000)
+ if (internalRow.getLong(index) == 0) {
Review comment:
why is 0, not DIRECT_DICT_VALUE_NULL?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587551993
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2045/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383369776
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
##########
@@ -728,7 +728,11 @@ object CommonLoadUtils {
}
val updatedRdd: RDD[InternalRow] = rdd.map { internalRow =>
for (index <- timeStampIndex) {
- internalRow.setLong(index, internalRow.getLong(index) / 1000)
+ if (internalRow.getLong(index) == 0) {
+ internalRow.setNullAt(index)
+ } else {
+ internalRow.setLong(index, internalRow.getLong(index) / 1000)
Review comment:
What does 1000 stands for? It is magic number
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383661618
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -176,8 +176,21 @@ case class CarbonInsertIntoCommand(databaseNameOp: Option[String],
convertedStaticPartition)
scanResultRdd = sparkSession.sessionState.executePlan(newLogicalPlan).toRdd
if (logicalPartitionRelation != null) {
- logicalPartitionRelation =
- getReArrangedSchemaLogicalRelation(reArrangedIndex, logicalPartitionRelation)
+ if (selectedColumnSchema.length != logicalPartitionRelation.output.length) {
+ throw new RuntimeException(" schema length doesn't match partition length")
+ }
+ var isAlreadyReArranged = true
+ var index = 0
+ for (col: ColumnSchema <- selectedColumnSchema) {
Review comment:
done. used lambda and exists
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-586552805
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2002/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3615: [CARBONDATA-3637]
Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590921776
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590466281
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587515359
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/343/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587486283
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/340/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589501574
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/378/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383371226
##########
File path: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java
##########
@@ -99,9 +99,12 @@
noDictDimensionPages = new ColumnPage[model.getNoDictionaryCount()];
int tmpNumDictDimIdx = 0;
int tmpNumNoDictDimIdx = 0;
- for (int i = 0; i < dictDimensionPages.length + noDictDimensionPages.length; i++) {
+ for (int i = 0; i < tableSpec.getNumDimensions(); i++) {
TableSpec.DimensionSpec spec = tableSpec.getDimensionSpec(i);
- ColumnType columnType = tableSpec.getDimensionSpec(i).getColumnType();
+ if (spec.getSchemaDataType().isComplexType()) {
+ // partition columns are placed at the end. so, might present after complex columns
Review comment:
Do you mean to skip all complex column and go to the last dimension?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589519466
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2080/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590150487
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590719018
@jackylk, @QiangCai : Handled the comments. Build passed. PR is ready.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590451473
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/446/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615: [WIP]
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP] [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589153305
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2074/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587038629
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2024/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615: [WIP] Send
insert stage, compaction to new insert into flow
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP] Send insert stage, compaction to new insert into flow
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-585168279
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1965/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587016618
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/322/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587411471
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2037/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590711800
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2150/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request
#3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on a change in pull request #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#discussion_r383377457
##########
File path: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java
##########
@@ -99,9 +99,12 @@
noDictDimensionPages = new ColumnPage[model.getNoDictionaryCount()];
int tmpNumDictDimIdx = 0;
int tmpNumNoDictDimIdx = 0;
- for (int i = 0; i < dictDimensionPages.length + noDictDimensionPages.length; i++) {
+ for (int i = 0; i < tableSpec.getNumDimensions(); i++) {
TableSpec.DimensionSpec spec = tableSpec.getDimensionSpec(i);
- ColumnType columnType = tableSpec.getDimensionSpec(i).getColumnType();
+ if (spec.getSchemaDataType().isComplexType()) {
+ // partition columns are placed at the end. so, might present after complex columns
Review comment:
yes, initially also it was skipping. I will make it more easy to understand
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615: [WIP]
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP] [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589463680
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/376/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587387340
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/335/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587628230
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2048/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3615: [WIP]
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3615: [WIP] [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589458443
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3615:
[WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage
command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3615: [WIP][CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-587559618
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615: [WIP]
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [WIP] [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-589480162
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2078/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3615:
[CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3615: [CARBONDATA-3637] Use optimized insert flow for MV and insert stage command
URL: https://github.com/apache/carbondata/pull/3615#issuecomment-590661153
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2148/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services