You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/10/13 17:21:54 UTC
[GitHub] [carbondata] marchpure opened a new pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
marchpure opened a new pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982
### Why is this PR needed?
1. CREATE TABLE droppartition (id STRING, sales STRING) PARTITIONED BY (dtm STRING)STORED AS carbondata
2.
insert into droppartition values ('01', '0', '20200907'),('03', '0', '20200908'),
insert overwrite table droppartition partition (dtm=20200908) select * from droppartition where dtm = 20200907;
insert overwrite table droppartition partition (dtm=20200909) select * from droppartition where dtm = 20200907;
3. alter table droppartition drop partition (dtm=20200909)
the dirctionary "20200908" was deleted.
The rootcause: drop partition will clean stale data, but delete the parent dirctionary contains the stale data, leading to normal data loss.
### What changes were proposed in this PR?
Fix this issue by limit to clean stale data, not the whole dirctionary
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708880660
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2697/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-709674010
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-709705817
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4469/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708162643
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2667/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-714889033
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4649/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-712604515
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4523/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713302522
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4557/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713189481
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4543/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-714891090
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2893/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
QiangCai commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-714860028
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713513371
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713320386
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2807/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-709263365
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4462/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-707952856
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2657/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708285816
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2674/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708092784
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2662/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Kejian-Li commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
Kejian-Li commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708816282
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-712602226
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2769/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
QiangCai commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-715048761
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-707956564
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4411/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] marchpure commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
marchpure commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-712560057
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-713178626
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2789/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#discussion_r510680548
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/partition/CarbonAlterTableDropHivePartitionCommand.scala
##########
@@ -181,9 +182,11 @@ case class CarbonAlterTableDropHivePartitionCommand(
OperationListenerBus.getInstance().fireEvent(postStatusEvent, operationContext)
IndexStoreManager.getInstance().clearIndex(table.getAbsoluteTableIdentifier)
+ tobeCleanSegs.addAll(tobeUpdatedSegs)
+ tobeCleanSegs.addAll(tobeDeletedSegs)
Review comment:
add all twice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-709265320
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2708/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708090696
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4416/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-709706349
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2715/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708163126
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4421/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
QiangCai commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708382061
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708287330
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4428/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3982: [CARBONDATA-4032] Fix drop partition command clean data issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3982:
URL: https://github.com/apache/carbondata/pull/3982#issuecomment-708879401
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4451/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org