You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/10/13 15:13:48 UTC
[GitHub] [carbondata] marchpure opened a new pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
marchpure opened a new pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981
…rt overwrite partition
### Why is this PR needed?
If update/delete some records on one partition, then inserts overwrite another partition. Deleted records before will be queried out.
### What changes were proposed in this PR?
When insert overwrite (or drop) partition, the updatestatusfilename in tablestatus will be tampered, which is fixed in this PR.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-716406302
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708201912
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4422/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713148889
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2790/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#discussion_r508334402
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##########
@@ -723,6 +723,15 @@ class MergeTestCase extends QueryTest with BeforeAndAfterAll {
assert(getDeleteDeltaFileCount("target", "0") == 0)
checkAnswer(sql("select count(*) from target"), Seq(Row(3)))
checkAnswer(sql("select * from target order by key"), Seq(Row("c", "200"), Row("d", "3"), Row("e", "100")))
+
+ // insert overwrite a partition. make sure the merge executed before still works.
+ sql(
+ """insert overwrite table target
+ | partition (value=3)
+ | select * from target where value = 100""".stripMargin)
+ checkAnswer(sql("select * from target"), Seq(Row("c", "200"), Row("e", "3"), Row("e", "100")))
Review comment:
```suggestion
checkAnswer(sql("select * from target order by key,value"), Seq(Row("c", "200"), Row("e", "100"), Row("e", "3")))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-725199629
@marchpure : This is already handled in #3999 , need to close this PR ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-709770210
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2718/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-716460174
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2934/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707956430
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2658/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713276469
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4552/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-717398581
@marchpure as from the description, i understood that the table status file after insert overwrite do not contain the `updatetablestatusfile ` name. So why do we need to add boolean variable, and send this boolean from many places. This may lead to confusion later also. So why dont we take care it just while updating the table status in case of insert overwrite?
I mean to say, while updating the status after insert overwrite, we can first copy the existing meta and then prepare new entry and update case can be handled there, this way, it will lead to less changes and less confusion later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure closed pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure closed pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-715532823
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4674/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708143674
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4418/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on a change in pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
Kejian-Li commented on a change in pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#discussion_r504458957
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
##########
@@ -69,6 +69,60 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
sql("""drop table iud.zerorows""")
}
+ test("update and insert overwrite partition") {
+ sql("""drop table if exists iud.updateinpartition""")
+ sql(
+ """CREATE TABLE iud.updateinpartition (id STRING, sales INT)
+ | PARTITIONED BY (dtm STRING)
+ | STORED AS carbondata""".stripMargin)
+ sql(
+ s"""load data local
+ | inpath '$resourcesPath/IUD/updateinpartition.csv' into table updateinpartition""".stripMargin)
+ sql(
+ """update iud.updateinpartition u set (u.sales) = (u.sales + 1) where id='001'""".stripMargin)
+ sql(
+ """update iud.updateinpartition u set (u.sales) = (u.sales + 2) where id='011'""".stripMargin)
+
+ // delete data from a partition, make sure the update executed before still works.
+ sql("""delete from updateinpartition where dtm=20200908 and id='012'""".stripMargin)
+ checkAnswer(
+ sql("""select sales from iud.updateinpartition where id='001'""".stripMargin), Seq(Row(1))
+ )
+ checkAnswer(
+ sql("""select sales from iud.updateinpartition where id='011'""".stripMargin), Seq(Row(2))
+ )
+ checkAnswer(
+ sql("""select sales from iud.updateinpartition where id='012'""".stripMargin), Seq()
+ )
+
+ // insert overwrite a partition. make sure the update executed before still works.
+ sql(
+ """insert overwrite table iud.updateinpartition
+ | partition (dtm=20200908)
+ | select * from iud.updateinpartition where dtm = 20200907""".stripMargin)
+ checkAnswer(
+ sql(
+ """select sales from iud.updateinpartition
+ | where dtm=20200908 and id='001'""".stripMargin), Seq(Row(1))
+ )
+ checkAnswer(
+ sql(
+ """select sales from iud.updateinpartition
+ | where dtm=20200908 and id='001'""".stripMargin), Seq(Row(1))
+ )
Review comment:
Are they duplicated?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-717333719
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4709/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708186433
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2668/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713222723
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-711489234
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2745/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713265288
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2801/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-717259421
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708473030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708143909
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2664/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-716977104
@marchpure please correct the PR title and please give detailed explanation of the issue and the fix in the PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707889628
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2656/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Klaus-xjp commented on a change in pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
Klaus-xjp commented on a change in pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#discussion_r504369446
##########
File path: integration/spark/src/test/resources/IUD/updateinpartition.csv
##########
@@ -0,0 +1,21 @@
+id,sales,dtm
+001,0,20200907
+002,0,20200907
+003,0,20200907
+004,0,20200907
+005,0,20200907
+006,0,20200907
+007,0,20200907
+008,0,20200907
+009,0,20200907
+010,0,20200907
+011,0,20200908
+012,0,20200908
+013,0,20200908
+014,0,20200908
+015,0,20200908
+016,0,20200908
+017,0,20200908
+018,0,20200908
+019,0,20200908
+020,0,20200908
Review comment:
new line at the end of file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-709698383
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#discussion_r512875192
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -276,14 +278,13 @@ public static boolean updateTableMetadataStatus(Set<Segment> updatedSegmentsList
SegmentStatusManager.readLoadMetadata(metaDataFilepath);
for (LoadMetadataDetails loadMetadata : listOfLoadFolderDetailsArray) {
+ if (isUpdateStatusFileUpdateRequired &&
+ loadMetadata.getLoadName().equalsIgnoreCase("0")) {
+ loadMetadata.setUpdateStatusFileName(
+ CarbonUpdateUtil.getUpdateStatusFileName(updatedTimeStamp));
+ }
if (isTimestampUpdateRequired) {
- // we are storing the link between the 2 status files in the segment 0 only.
Review comment:
why this condition is moved up? and why not `isTimestampUpdateRequired`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-711487305
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4499/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707894635
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4410/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708349031
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2679/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-715371390
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4671/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-711459583
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-716406534
> please do rebase
I have do rebase according to your suggestion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-716466945
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4691/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-715535323
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2917/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707818001
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707816761
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2654/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708102326
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708340588
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4431/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708100450
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4415/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708375305
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708097990
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2661/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707955314
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4412/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-717398850
> > @marchpure please correct the PR title and please give detailed explanation of the issue and the fix in the PR description.
>
> Yes. I have modified the PR desc as your suggestion
PR title is still not yet formatted, please check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-717328681
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2952/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-708148791
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
QiangCai commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-714450676
please do rebase
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-709765901
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4472/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-715375742
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2915/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on a change in pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on a change in pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#discussion_r510863739
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##########
@@ -723,6 +723,15 @@ class MergeTestCase extends QueryTest with BeforeAndAfterAll {
assert(getDeleteDeltaFileCount("target", "0") == 0)
checkAnswer(sql("select count(*) from target"), Seq(Row(3)))
checkAnswer(sql("select * from target order by key"), Seq(Row("c", "200"), Row("d", "3"), Row("e", "100")))
+
+ // insert overwrite a partition. make sure the merge executed before still works.
+ sql(
+ """insert overwrite table target
+ | partition (value=3)
+ | select * from target where value = 100""".stripMargin)
+ checkAnswer(sql("select * from target"), Seq(Row("c", "200"), Row("e", "3"), Row("e", "100")))
Review comment:
I have modifed code according to your suggestion
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala
##########
@@ -69,6 +69,60 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
sql("""drop table iud.zerorows""")
}
+ test("update and insert overwrite partition") {
+ sql("""drop table if exists iud.updateinpartition""")
+ sql(
+ """CREATE TABLE iud.updateinpartition (id STRING, sales INT)
+ | PARTITIONED BY (dtm STRING)
+ | STORED AS carbondata""".stripMargin)
+ sql(
+ s"""load data local
+ | inpath '$resourcesPath/IUD/updateinpartition.csv' into table updateinpartition""".stripMargin)
+ sql(
+ """update iud.updateinpartition u set (u.sales) = (u.sales + 1) where id='001'""".stripMargin)
+ sql(
+ """update iud.updateinpartition u set (u.sales) = (u.sales + 2) where id='011'""".stripMargin)
+
+ // delete data from a partition, make sure the update executed before still works.
+ sql("""delete from updateinpartition where dtm=20200908 and id='012'""".stripMargin)
+ checkAnswer(
+ sql("""select sales from iud.updateinpartition where id='001'""".stripMargin), Seq(Row(1))
+ )
+ checkAnswer(
+ sql("""select sales from iud.updateinpartition where id='011'""".stripMargin), Seq(Row(2))
+ )
+ checkAnswer(
+ sql("""select sales from iud.updateinpartition where id='012'""".stripMargin), Seq()
+ )
+
+ // insert overwrite a partition. make sure the update executed before still works.
+ sql(
+ """insert overwrite table iud.updateinpartition
+ | partition (dtm=20200908)
+ | select * from iud.updateinpartition where dtm = 20200907""".stripMargin)
+ checkAnswer(
+ sql(
+ """select sales from iud.updateinpartition
+ | where dtm=20200908 and id='001'""".stripMargin), Seq(Row(1))
+ )
+ checkAnswer(
+ sql(
+ """select sales from iud.updateinpartition
+ | where dtm=20200908 and id='001'""".stripMargin), Seq(Row(1))
+ )
Review comment:
I have modifed code according to your suggestion
##########
File path: integration/spark/src/test/resources/IUD/updateinpartition.csv
##########
@@ -0,0 +1,21 @@
+id,sales,dtm
+001,0,20200907
+002,0,20200907
+003,0,20200907
+004,0,20200907
+005,0,20200907
+006,0,20200907
+007,0,20200907
+008,0,20200907
+009,0,20200907
+010,0,20200907
+011,0,20200908
+012,0,20200908
+013,0,20200908
+014,0,20200908
+015,0,20200908
+016,0,20200908
+017,0,20200908
+018,0,20200908
+019,0,20200908
+020,0,20200908
Review comment:
I have modifed code according to your suggestion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-713208327
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4544/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3981: [CARBONDATA-4027] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-707816884
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4408/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3981: [CARBONDATA-4031] Incorrect query result after Update/Delete and Inse…
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3981:
URL: https://github.com/apache/carbondata/pull/3981#issuecomment-716981130
> @marchpure please correct the PR title and please give detailed explanation of the issue and the fix in the PR description.
Yes. I have modified the PR desc as your suggestion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org