You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "askwang (via GitHub)" <gi...@apache.org> on 2024/09/26 09:33:13 UTC
[PR] [core] Sort the expired partitions and add max_expires param [paimon]
askwang opened a new pull request, #4266:
URL: https://github.com/apache/paimon/pull/4266
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
- The result of expire_partitions is out of order, sort it.
- Add max_expires param to limit the expired partitions number at one time.
before:
after:
<img width="1067" alt="image" src="https://github.com/user-attachments/assets/e6cf7ef8-d978-47de-a198-024a7f109fd6">
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Sort the expired partitions and add max_expires param [paimon]
Posted by "askwang (via GitHub)" <gi...@apache.org>.
askwang commented on PR #4266:
URL: https://github.com/apache/paimon/pull/4266#issuecomment-2376893231
@wwj6591812 thanks for review, addressed comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Sort the expired partitions and add max_expires param [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4266:
URL: https://github.com/apache/paimon/pull/4266#discussion_r1776864995
##########
paimon-core/src/main/java/org/apache/paimon/operation/PartitionExpire.java:
##########
@@ -153,4 +163,14 @@ private void deleteMetastorePartitions(List<Map<String, String>> partitions) {
});
}
}
+
+ private List<Map<String, String>> convertToPartitionString(List<List<String>> partValues) {
+ return partValues.stream()
+ .map(values -> String.join(",", values))
Review Comment:
private final String xxx = ",";
##########
paimon-core/src/main/java/org/apache/paimon/operation/PartitionExpire.java:
##########
@@ -125,14 +133,16 @@ List<Map<String, String>> expire(LocalDateTime now, long commitIdentifier) {
private List<Map<String, String>> doExpire(
LocalDateTime expireDateTime, long commitIdentifier) {
- List<Map<String, String>> expired = new ArrayList<>();
+ List<List<String>> partValues = new ArrayList<>();
for (PartitionEntry partition : strategy.selectExpiredPartitions(scan, expireDateTime)) {
Object[] array = strategy.convertPartition(partition.partition());
- Map<String, String> partString = strategy.toPartitionString(array);
- expired.add(partString);
- LOG.info("Expire Partition: {}", partString);
+ partValues.add(strategy.toPartitionValue(array));
}
- if (!expired.isEmpty()) {
+
Review Comment:
Use follow code could init partValues with capacity.
List<PartitionEntry> partitionEntries = strategy.selectExpiredPartitions(scan, expireDateTime);
List<List<String>> partValues = new ArrayList<>(partitionEntries.size());
for (PartitionEntry partition : partitionEntries) {
Object[] array = strategy.convertPartition(partition.partition());
partValues.add(strategy.toPartitionValue(array));
}
##########
paimon-core/src/main/java/org/apache/paimon/partition/PartitionExpireStrategy.java:
##########
@@ -49,6 +50,14 @@ public Map<String, String> toPartitionString(Object[] array) {
return map;
}
+ public List<String> toPartitionValue(Object[] array) {
+ List<String> list = new ArrayList<>();
Review Comment:
Change "List<String> list = new ArrayList<>();" to "List<String> list = new ArrayList<>(partitionKeys.size());".
##########
paimon-core/src/main/java/org/apache/paimon/operation/PartitionExpire.java:
##########
@@ -125,14 +133,16 @@ List<Map<String, String>> expire(LocalDateTime now, long commitIdentifier) {
private List<Map<String, String>> doExpire(
LocalDateTime expireDateTime, long commitIdentifier) {
- List<Map<String, String>> expired = new ArrayList<>();
+ List<List<String>> partValues = new ArrayList<>();
for (PartitionEntry partition : strategy.selectExpiredPartitions(scan, expireDateTime)) {
Object[] array = strategy.convertPartition(partition.partition());
- Map<String, String> partString = strategy.toPartitionString(array);
- expired.add(partString);
- LOG.info("Expire Partition: {}", partString);
+ partValues.add(strategy.toPartitionValue(array));
}
- if (!expired.isEmpty()) {
+
+ List<Map<String, String>> expired = new ArrayList<>();
+ if (!partValues.isEmpty()) {
+ expired = convertToPartitionString(partValues);
+ LOG.info("Expire Partition: {}", expired);
Review Comment:
Change "Expire Partition" to "Expire Partitions".
##########
paimon-core/src/main/java/org/apache/paimon/operation/PartitionExpire.java:
##########
@@ -125,14 +133,16 @@ List<Map<String, String>> expire(LocalDateTime now, long commitIdentifier) {
private List<Map<String, String>> doExpire(
LocalDateTime expireDateTime, long commitIdentifier) {
- List<Map<String, String>> expired = new ArrayList<>();
+ List<List<String>> partValues = new ArrayList<>();
Review Comment:
Change "partValues" to "expiredPartValues"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Sort the expired partitions and add max_expires param [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #4266:
URL: https://github.com/apache/paimon/pull/4266#issuecomment-2377046610
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org