You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2022/01/04 11:12:13 UTC
[GitHub] [carbondata] vikramahuja1001 opened a new pull request #4246: [WIP] Fix clean files removing wrong delta files
vikramahuja1001 opened a new pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246
### Why is this PR needed?
In the case where there are multiple delete delta files in a partition in a partition table, some delta files were being ignored and deleted, thus changing the value during the query
### What changes were proposed in this PR?
Fixed the logic which checks which delta file to delete. Now checking the deltaStartTime and comparing it with deltaEndTime to check consider all the delta files during clean files.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- Yes, one test case has been added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1004969731
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6194/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005004640
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/585/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r780048924
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -732,11 +732,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
.collect(Collectors.toList()));
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
- for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
- }
+
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in this case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ Arrays.stream(updateDetails).forEach(block -> {
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.removeIf(filter -> filter.getName().endsWith(block
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1006602812
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4454/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r779473613
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -733,9 +733,19 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in this case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
+ .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
+ .collect(Collectors.toList()).forEach(totalDeltaFiles::remove);
+ } else {
+ block.getDeltaFileStamps().stream().forEach(fileName -> totalDeltaFiles
+ .removeIf(filter -> filter.getName().endsWith(fileName +
+ CarbonCommonConstants.DELETE_DELTA_FILE_EXT)));
+ }
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005710711
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4452/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r778866192
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -733,9 +733,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in thios case more
Review comment:
```suggestion
// Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in this case more
```
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/cleanfiles/TestCleanFilesCommandPartitionTable.scala
##########
@@ -359,6 +359,88 @@ class TestCleanFilesCommandPartitionTable extends QueryTest with BeforeAndAfterA
sql("drop table if exists partition_hc")
}
+ test("test clean files after IUD Horizontal Compaction when" +
+ " CarbonCommonConstants.DELETE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION > 1") {
+
+ CarbonProperties.getInstance().
+ addProperty(CarbonCommonConstants.DELETE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION, "3")
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_CLEAN_FILES_FORCE_ALLOWED, "true")
+ sql("drop table if exists origintable")
+
+ sql(
+ """
+ | CREATE TABLE origintable
+ | (id Int,
+ | vin String,
+ | logdate Date,
+ | phonenumber Long,
+ | area String,
+ | salary Int) PARTITIONED BY(country String)
+ | STORED AS carbondata
+ """.stripMargin)
+
+ val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+ val testData = s"$rootPath/integration/spark/src/test/resources/" +
+ s"partition_data_example.csv"
+
+ sql(
+ s"""
+ LOAD DATA LOCAL INPATH '$testData' into table origintable
+ """)
+
+ sql("delete from origintable where salary = 10000").show()
+ sql("delete from origintable where salary = 10001").show()
+ sql("delete from origintable where salary = 10003").show()
+ var preCleanFiles = sql("select * from origintable").count()
+ sql(s"CLEAN FILES FOR TABLE origintable OPTIONS('force'='true')").collect()
+ var postCleanFiles = sql("select * from origintable").count()
+ assert(preCleanFiles == postCleanFiles)
+ sql("delete from origintable where salary = 10005").show()
+
+ // verify if the horizontal compaction happened or not
+ val carbonTable = CarbonEnv.getCarbonTable(None, "origintable")(sqlContext
+ .sparkSession)
+ val partitionPath = carbonTable.getTablePath + "/country=China"
+ val deltaFilesPre = FileFactory.getCarbonFile(partitionPath).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.endsWith(CarbonCommonConstants.DELETE_DELTA_FILE_EXT)
+ }
+ })
+ assert(deltaFilesPre.size == 5)
+ val updateStatusFilesPre = FileFactory.getCarbonFile(CarbonTablePath.getMetadataPath(carbonTable
+ .getTablePath)).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.startsWith(CarbonCommonConstants.TABLEUPDATESTATUS_FILENAME)
+ }
+ })
+ assert(updateStatusFilesPre.size == 3)
+
+ preCleanFiles = sql("select * from origintable").count()
+ sql(s"CLEAN FILES FOR TABLE origintable OPTIONS('force'='true')").collect()
+ postCleanFiles = sql("select * from origintable").count()
+ assert(preCleanFiles == postCleanFiles)
+
+ val deltaFilesPost = FileFactory.getCarbonFile(partitionPath).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.endsWith(CarbonCommonConstants.DELETE_DELTA_FILE_EXT)
+ }
+ })
+ assert(deltaFilesPost.size == 1)
+ val updateStatusFilesPost = FileFactory.getCarbonFile(CarbonTablePath
+ .getMetadataPath(carbonTable.getTablePath)).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.startsWith(CarbonCommonConstants.TABLEUPDATESTATUS_FILENAME)
+ }
+ })
+ assert(updateStatusFilesPost.size == 1)
+
+ sql("drop table if exists origintable")
+ CarbonProperties.getInstance()
+ .removeProperty(CarbonCommonConstants.DELETE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION)
Review comment:
set back to default value, instead of removing
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -733,9 +733,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in thios case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
+ .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
+ .collect(Collectors.toList()).forEach(totalDeltaFiles::remove);
+ } else {
+ for (String deltaFile: block.getDeltaFileStamps()) {
+ totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(
+ deltaFile + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
+ .collect(Collectors.toList()).forEach(totalDeltaFiles::remove);
Review comment:
for every `deltaFile`, you are iterating over `totalDeltaFiles` to find and delete file, it will add more time complexity. Instead for all `totalDeltaFiles`, filter the files `block.getDeltaFileStamps()` and then remove in one shot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1004735850
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4449/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r780049079
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -732,11 +732,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
.collect(Collectors.toList()));
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
- for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
- }
+
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in this case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ Arrays.stream(updateDetails).forEach(block -> {
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.removeIf(filter -> filter.getName().endsWith(block
+ .getDeleteDeltaEndTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT));
+ } else {
+ block.getDeltaFileStamps().stream().forEach(fileName -> totalDeltaFiles
+ .removeIf(filter -> filter.getName().endsWith(fileName +
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ydvpankaj99 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
ydvpankaj99 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005609819
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r779352216
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -733,9 +733,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in thios case more
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r779352407
##########
File path: integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/cleanfiles/TestCleanFilesCommandPartitionTable.scala
##########
@@ -359,6 +359,88 @@ class TestCleanFilesCommandPartitionTable extends QueryTest with BeforeAndAfterA
sql("drop table if exists partition_hc")
}
+ test("test clean files after IUD Horizontal Compaction when" +
+ " CarbonCommonConstants.DELETE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION > 1") {
+
+ CarbonProperties.getInstance().
+ addProperty(CarbonCommonConstants.DELETE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION, "3")
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_CLEAN_FILES_FORCE_ALLOWED, "true")
+ sql("drop table if exists origintable")
+
+ sql(
+ """
+ | CREATE TABLE origintable
+ | (id Int,
+ | vin String,
+ | logdate Date,
+ | phonenumber Long,
+ | area String,
+ | salary Int) PARTITIONED BY(country String)
+ | STORED AS carbondata
+ """.stripMargin)
+
+ val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+ val testData = s"$rootPath/integration/spark/src/test/resources/" +
+ s"partition_data_example.csv"
+
+ sql(
+ s"""
+ LOAD DATA LOCAL INPATH '$testData' into table origintable
+ """)
+
+ sql("delete from origintable where salary = 10000").show()
+ sql("delete from origintable where salary = 10001").show()
+ sql("delete from origintable where salary = 10003").show()
+ var preCleanFiles = sql("select * from origintable").count()
+ sql(s"CLEAN FILES FOR TABLE origintable OPTIONS('force'='true')").collect()
+ var postCleanFiles = sql("select * from origintable").count()
+ assert(preCleanFiles == postCleanFiles)
+ sql("delete from origintable where salary = 10005").show()
+
+ // verify if the horizontal compaction happened or not
+ val carbonTable = CarbonEnv.getCarbonTable(None, "origintable")(sqlContext
+ .sparkSession)
+ val partitionPath = carbonTable.getTablePath + "/country=China"
+ val deltaFilesPre = FileFactory.getCarbonFile(partitionPath).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.endsWith(CarbonCommonConstants.DELETE_DELTA_FILE_EXT)
+ }
+ })
+ assert(deltaFilesPre.size == 5)
+ val updateStatusFilesPre = FileFactory.getCarbonFile(CarbonTablePath.getMetadataPath(carbonTable
+ .getTablePath)).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.startsWith(CarbonCommonConstants.TABLEUPDATESTATUS_FILENAME)
+ }
+ })
+ assert(updateStatusFilesPre.size == 3)
+
+ preCleanFiles = sql("select * from origintable").count()
+ sql(s"CLEAN FILES FOR TABLE origintable OPTIONS('force'='true')").collect()
+ postCleanFiles = sql("select * from origintable").count()
+ assert(preCleanFiles == postCleanFiles)
+
+ val deltaFilesPost = FileFactory.getCarbonFile(partitionPath).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.endsWith(CarbonCommonConstants.DELETE_DELTA_FILE_EXT)
+ }
+ })
+ assert(deltaFilesPost.size == 1)
+ val updateStatusFilesPost = FileFactory.getCarbonFile(CarbonTablePath
+ .getMetadataPath(carbonTable.getTablePath)).listFiles(new CarbonFileFilter {
+ override def accept(file: CarbonFile): Boolean = {
+ file.getName.startsWith(CarbonCommonConstants.TABLEUPDATESTATUS_FILENAME)
+ }
+ })
+ assert(updateStatusFilesPost.size == 1)
+
+ sql("drop table if exists origintable")
+ CarbonProperties.getInstance()
+ .removeProperty(CarbonCommonConstants.DELETE_DELTAFILE_COUNT_THRESHOLD_IUD_COMPACTION)
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r779609742
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -732,11 +732,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
.collect(Collectors.toList()));
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
- for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
- }
+
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in this case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ Arrays.stream(updateDetails).forEach(block -> {
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.removeIf(filter -> filter.getName().endsWith(block
Review comment:
rename `filter` to `deltaFile` or `carbonFile`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ydvpankaj99 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
ydvpankaj99 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005417486
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1008632473
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/592/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005751472
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/587/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1011807077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005499739
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6195/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r779459763
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -733,9 +733,19 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in this case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
+ .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
+ .collect(Collectors.toList()).forEach(totalDeltaFiles::remove);
+ } else {
+ block.getDeltaFileStamps().stream().forEach(fileName -> totalDeltaFiles
+ .removeIf(filter -> filter.getName().endsWith(fileName +
+ CarbonCommonConstants.DELETE_DELTA_FILE_EXT)));
+ }
Review comment:
instead of streaming for each `block`, can merge and do single looping?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1007184271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r779352493
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -733,9 +733,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in thios case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
+ .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
+ .collect(Collectors.toList()).forEach(totalDeltaFiles::remove);
+ } else {
+ for (String deltaFile: block.getDeltaFileStamps()) {
+ totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(
+ deltaFile + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
+ .collect(Collectors.toList()).forEach(totalDeltaFiles::remove);
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005703408
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6196/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1007190228
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1007184271
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4455/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1004846649
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/584/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1004982034
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4450/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005515693
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4451/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#discussion_r779609866
##########
File path: core/src/main/java/org/apache/carbondata/core/mutate/CarbonUpdateUtil.java
##########
@@ -732,11 +732,21 @@ public static long cleanUpDeltaFiles(CarbonTable table, boolean isDryRun) throws
.collect(Collectors.toList()));
}
SegmentUpdateDetails[] updateDetails = updateStatusManager.readLoadMetadata();
- for (SegmentUpdateDetails block : updateDetails) {
- totalDeltaFiles.stream().filter(fileName -> fileName.getName().endsWith(block
- .getDeleteDeltaStartTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT))
- .collect(Collectors.toList()).forEach(fileName -> totalDeltaFiles.remove(fileName));
- }
+
+ // Case 1: When deleteDeltaStartTimestamp = deleteDeltaEndTimestamp. in this case only 1
+ // delta file is present and deltaFileStamps is NULL
+ // Case 2: When deleteDeltaStartTimestamp != deleteDeltaEndTimestamp. in this case more
+ // than 1 delta files are present, then can blindly read deltaFilesStamps variable
+ Arrays.stream(updateDetails).forEach(block -> {
+ if (block.getDeleteDeltaStartTimestamp().equals(block.getDeleteDeltaEndTimestamp())) {
+ totalDeltaFiles.removeIf(filter -> filter.getName().endsWith(block
+ .getDeleteDeltaEndTimestamp() + CarbonCommonConstants.DELETE_DELTA_FILE_EXT));
+ } else {
+ block.getDeltaFileStamps().stream().forEach(fileName -> totalDeltaFiles
+ .removeIf(filter -> filter.getName().endsWith(fileName +
Review comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1007251385
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/590/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1006458471
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/588/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1008606702
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1008624059
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4457/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1008546624
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1006598074
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6198/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1006448293
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4453/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1005487596
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/586/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1006617101
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/589/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1004797152
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6193/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4246: [WIP] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246#issuecomment-1006430019
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6197/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4246: [CARBONDATA-4320] Fix clean files removing wrong delta files
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4246:
URL: https://github.com/apache/carbondata/pull/4246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org