You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/12/08 10:21:00 UTC
[GitHub] [carbondata] akashrn5 opened a new pull request #4242: [WIP]Improve load overwrite performance for partition tables
akashrn5 opened a new pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-988779046
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6153/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#discussion_r775794190
##########
File path: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonOutputCommitter.java
##########
@@ -316,31 +326,38 @@ private void commitJobForPartition(JobContext context, boolean overwriteSet,
* of all segment files.
*/
private String overwritePartitions(CarbonLoadModel loadModel, LoadMetadataDetails newMetaEntry,
- String uuid) throws IOException {
+ String uuid, List<String> partitionList, List<PartitionSpec> currentPartitionsOfTable)
+ throws IOException {
CarbonTable table = loadModel.getCarbonDataLoadSchema().getCarbonTable();
- SegmentFileStore fileStore = new SegmentFileStore(loadModel.getTablePath(),
- loadModel.getSegmentId() + "_" + loadModel.getFactTimeStamp()
- + CarbonTablePath.SEGMENT_EXT);
- List<PartitionSpec> partitionSpecs = fileStore.getPartitionSpecs();
-
- if (partitionSpecs != null && partitionSpecs.size() > 0) {
- List<Segment> validSegments =
- new SegmentStatusManager(table.getAbsoluteTableIdentifier())
- .getValidAndInvalidSegments(table.isMV()).getValidSegments();
- String uniqueId = String.valueOf(System.currentTimeMillis());
- List<String> toBeUpdatedSegments = new ArrayList<>();
- List<String> toBeDeletedSegments = new ArrayList<>();
- // First drop the partitions from partition mapper files of each segment
- for (Segment segment : validSegments) {
- new SegmentFileStore(table.getTablePath(), segment.getSegmentFileName()).dropPartitions(
- segment, partitionSpecs, uniqueId, toBeDeletedSegments, toBeUpdatedSegments);
+ if (partitionList != null && partitionList.size() > 0) {
+ // check if any partitions overlaps
+ List<String> overlappingPartitions = currentPartitionsOfTable.stream()
+ .map(partitionSpec -> partitionSpec.getLocation().toString())
+ .filter(partitionList::contains).collect(Collectors.toList());
+ if (!overlappingPartitions.isEmpty()) {
+ List<LoadMetadataDetails> validLoadMetadataDetails =
+ loadModel.getLoadMetadataDetails().stream().filter(
+ loadMetadataDetail -> !loadMetadataDetail.getLoadName()
+ .equalsIgnoreCase(newMetaEntry.getLoadName())).collect(Collectors.toList());
+ String uniqueId = String.valueOf(System.currentTimeMillis());
+ List<String> toBeUpdatedSegments = new ArrayList<>();
Review comment:
maybe for this list we can give the max length as number of segments while instantiating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-989567500
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-989632280
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/546/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-989027304
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/545/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-989605584
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6155/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-989014941
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4411/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-988778055
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4410/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#discussion_r775953047
##########
File path: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonOutputCommitter.java
##########
@@ -316,31 +326,38 @@ private void commitJobForPartition(JobContext context, boolean overwriteSet,
* of all segment files.
*/
private String overwritePartitions(CarbonLoadModel loadModel, LoadMetadataDetails newMetaEntry,
- String uuid) throws IOException {
+ String uuid, List<String> partitionList, List<PartitionSpec> currentPartitionsOfTable)
+ throws IOException {
CarbonTable table = loadModel.getCarbonDataLoadSchema().getCarbonTable();
- SegmentFileStore fileStore = new SegmentFileStore(loadModel.getTablePath(),
- loadModel.getSegmentId() + "_" + loadModel.getFactTimeStamp()
- + CarbonTablePath.SEGMENT_EXT);
- List<PartitionSpec> partitionSpecs = fileStore.getPartitionSpecs();
-
- if (partitionSpecs != null && partitionSpecs.size() > 0) {
- List<Segment> validSegments =
- new SegmentStatusManager(table.getAbsoluteTableIdentifier())
- .getValidAndInvalidSegments(table.isMV()).getValidSegments();
- String uniqueId = String.valueOf(System.currentTimeMillis());
- List<String> toBeUpdatedSegments = new ArrayList<>();
- List<String> toBeDeletedSegments = new ArrayList<>();
- // First drop the partitions from partition mapper files of each segment
- for (Segment segment : validSegments) {
- new SegmentFileStore(table.getTablePath(), segment.getSegmentFileName()).dropPartitions(
- segment, partitionSpecs, uniqueId, toBeDeletedSegments, toBeUpdatedSegments);
+ if (partitionList != null && partitionList.size() > 0) {
+ // check if any partitions overlaps
+ List<String> overlappingPartitions = currentPartitionsOfTable.stream()
+ .map(partitionSpec -> partitionSpec.getLocation().toString())
+ .filter(partitionList::contains).collect(Collectors.toList());
+ if (!overlappingPartitions.isEmpty()) {
+ List<LoadMetadataDetails> validLoadMetadataDetails =
+ loadModel.getLoadMetadataDetails().stream().filter(
+ loadMetadataDetail -> !loadMetadataDetail.getLoadName()
+ .equalsIgnoreCase(newMetaEntry.getLoadName())).collect(Collectors.toList());
+ String uniqueId = String.valueOf(System.currentTimeMillis());
+ List<String> toBeUpdatedSegments = new ArrayList<>();
Review comment:
anyways, this was old code just added in a condition, but yeah, initialized with an initial capacity of `validLoadMetadataDetails`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-989001098
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6154/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-1002212357
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4447/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-1002212246
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6191/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [WIP]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-988783291
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/544/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
vikramahuja1001 commented on a change in pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#discussion_r775794279
##########
File path: hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonOutputCommitter.java
##########
@@ -316,31 +326,38 @@ private void commitJobForPartition(JobContext context, boolean overwriteSet,
* of all segment files.
*/
private String overwritePartitions(CarbonLoadModel loadModel, LoadMetadataDetails newMetaEntry,
- String uuid) throws IOException {
+ String uuid, List<String> partitionList, List<PartitionSpec> currentPartitionsOfTable)
+ throws IOException {
CarbonTable table = loadModel.getCarbonDataLoadSchema().getCarbonTable();
- SegmentFileStore fileStore = new SegmentFileStore(loadModel.getTablePath(),
- loadModel.getSegmentId() + "_" + loadModel.getFactTimeStamp()
- + CarbonTablePath.SEGMENT_EXT);
- List<PartitionSpec> partitionSpecs = fileStore.getPartitionSpecs();
-
- if (partitionSpecs != null && partitionSpecs.size() > 0) {
- List<Segment> validSegments =
- new SegmentStatusManager(table.getAbsoluteTableIdentifier())
- .getValidAndInvalidSegments(table.isMV()).getValidSegments();
- String uniqueId = String.valueOf(System.currentTimeMillis());
- List<String> toBeUpdatedSegments = new ArrayList<>();
- List<String> toBeDeletedSegments = new ArrayList<>();
- // First drop the partitions from partition mapper files of each segment
- for (Segment segment : validSegments) {
- new SegmentFileStore(table.getTablePath(), segment.getSegmentFileName()).dropPartitions(
- segment, partitionSpecs, uniqueId, toBeDeletedSegments, toBeUpdatedSegments);
+ if (partitionList != null && partitionList.size() > 0) {
+ // check if any partitions overlaps
+ List<String> overlappingPartitions = currentPartitionsOfTable.stream()
+ .map(partitionSpec -> partitionSpec.getLocation().toString())
+ .filter(partitionList::contains).collect(Collectors.toList());
+ if (!overlappingPartitions.isEmpty()) {
+ List<LoadMetadataDetails> validLoadMetadataDetails =
+ loadModel.getLoadMetadataDetails().stream().filter(
+ loadMetadataDetail -> !loadMetadataDetail.getLoadName()
+ .equalsIgnoreCase(newMetaEntry.getLoadName())).collect(Collectors.toList());
+ String uniqueId = String.valueOf(System.currentTimeMillis());
+ List<String> toBeUpdatedSegments = new ArrayList<>();
+ List<String> toBeDeletedSegments = new ArrayList<>();
Review comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] kunal642 commented on pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-1002510505
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4242: [CARBONDATA-4318]Improve load overwrite performance for partition tables
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4242:
URL: https://github.com/apache/carbondata/pull/4242#issuecomment-1002205435
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org