You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LinMingQiang (via GitHub)" <gi...@apache.org> on 2025/01/11 14:04:26 UTC
[PR] [core] Supports skipping specified actions during writing. [paimon]
LinMingQiang opened a new pull request, #4884:
URL: https://github.com/apache/paimon/pull/4884
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1914168078
##########
docs/layouts/shortcodes/generated/core_configuration.html:
##########
@@ -1050,6 +1050,12 @@
<td>Integer</td>
<td>Write batch size for any file format if it supports.</td>
</tr>
+ <tr>
+ <td><h5>write.skip-actions</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>This parameter only works when write-only is false., You can specify which actions to skip during the write process.<br />1. 'partition-expire': Skip the action of partition expiration.<br />2. 'snapshot-expire': Skip the action of snapshot expiration.<br />3. 'create-tag': Skip automatic tag creation.<br />Both can be configured at the same time: 'partition-expire,snapshot-expire,create-tag'.</td>
Review Comment:
Maybe you can add a `write-actions`? Default is 'all'. The values can be:
`snapshot-expire`, `partition-expire`, `compaction`, `tag-automatic-creation`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1912463173
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3163,4 +3206,26 @@ public enum MaterializedTableRefreshStatus {
ACTIVATED,
SUSPENDED
}
+
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
Review Comment:
Here is just an enumeration of all action types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1912463304
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2216,6 +2238,27 @@ public boolean writeOnly() {
return options.get(WRITE_ONLY);
}
+ public Set<WriteAction> writeSkippingActions() {
+ String str = options.get(WRITE_SKIP_ACTIONS);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1915907067
##########
docs/layouts/shortcodes/generated/core_configuration.html:
##########
@@ -1050,6 +1050,12 @@
<td>Integer</td>
<td>Write batch size for any file format if it supports.</td>
</tr>
+ <tr>
+ <td><h5>write.skip-actions</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>This parameter only works when write-only is false., You can specify which actions to skip during the write process.<br />1. 'partition-expire': Skip the action of partition expiration.<br />2. 'snapshot-expire': Skip the action of snapshot expiration.<br />3. 'create-tag': Skip automatic tag creation.<br />Both can be configured at the same time: 'partition-expire,snapshot-expire,create-tag'.</td>
Review Comment:
Ok, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1912355288
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -442,6 +443,25 @@ public class CoreOptions implements Serializable {
"If set to true, compactions and snapshot expiration will be skipped. "
+ "This option is used along with dedicated compact jobs.");
+ public static final ConfigOption<String> WRITE_SKIP_ACTIONS =
+ key("write.skip-actions")
+ .stringType()
Review Comment:
enumType?
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -442,6 +443,25 @@ public class CoreOptions implements Serializable {
"If set to true, compactions and snapshot expiration will be skipped. "
+ "This option is used along with dedicated compact jobs.");
+ public static final ConfigOption<String> WRITE_SKIP_ACTIONS =
+ key("write.skip-actions")
+ .stringType()
+ .noDefaultValue()
+ .withDescription(
+ Description.builder()
+ .text(
+ "This parameter only works when write-only is false., You can specify which actions to skip during the write process.")
+ .linebreak()
+ .text("1. 'partition-expire': skipping partition expire.")
Review Comment:
skipping -> skip?
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2216,6 +2236,27 @@ public boolean writeOnly() {
return options.get(WRITE_ONLY);
}
+ public HashSet<WriteAction> writeSkippingActions() {
Review Comment:
public Set<WriteAction> writeSkippingActions(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1912450915
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3163,4 +3206,26 @@ public enum MaterializedTableRefreshStatus {
ACTIVATED,
SUSPENDED
}
+
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
Review Comment:
Maybe we can add a option for all actions?
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2216,6 +2238,27 @@ public boolean writeOnly() {
return options.get(WRITE_ONLY);
}
+ public Set<WriteAction> writeSkippingActions() {
+ String str = options.get(WRITE_SKIP_ACTIONS);
Review Comment:
Here no default value，check isPresent？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on PR #4884:
URL: https://github.com/apache/paimon/pull/4884#issuecomment-2586043595
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1951894438
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3226,4 +3311,31 @@ public String toString() {
return value;
}
}
+
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // All write actions will be performed, this is the default behavior.
+ ALL("all"),
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
+ SNAPSHOT_EXPIRE("snapshot-expire"),
+ TAG_AUTOMATIC_CREATION("tag-automatic-creation"),
+
+ // Actions during writing.
+ MINOR_COMPACT("minor-compact"),
Review Comment:
We can only keep `full-compact` and `minor-compact` action can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4884:
URL: https://github.com/apache/paimon/pull/4884#issuecomment-2638685157
@JingsongLi PTAL, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #4884:
URL: https://github.com/apache/paimon/pull/4884#issuecomment-2585651065
Thanks for @LinMingQiang prepare this pr.
This is a good job, I recently received a similar request from some Paimon user in my company.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1950146698
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3226,4 +3311,31 @@ public String toString() {
return value;
}
}
+
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // All write actions will be performed, this is the default behavior.
+ ALL("all"),
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
+ SNAPSHOT_EXPIRE("snapshot-expire"),
+ TAG_AUTOMATIC_CREATION("tag-automatic-creation"),
+
+ // Actions during writing.
+ MINOR_COMPACT("minor-compact"),
Review Comment:
It is strange here, in streaming mode, it is hard to say just doing minor compact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4884:
URL: https://github.com/apache/paimon/pull/4884#issuecomment-4426244237
This pull request has had no activity for 90 days. If you'd like to keep it open, please push a new commit or leave a comment. Thanks for the contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang closed pull request #4884: [core] Supports skipping specified actions during writing.
URL: https://github.com/apache/paimon/pull/4884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1951894438
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3226,4 +3311,31 @@ public String toString() {
return value;
}
}
+
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // All write actions will be performed, this is the default behavior.
+ ALL("all"),
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
+ SNAPSHOT_EXPIRE("snapshot-expire"),
+ TAG_AUTOMATIC_CREATION("tag-automatic-creation"),
+
+ // Actions during writing.
+ MINOR_COMPACT("minor-compact"),
Review Comment:
We can only keep `full-compact` and `minor-compact` action can be removed. `minor-compact` keep the original default logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1915904843
##########
docs/layouts/shortcodes/generated/core_configuration.html:
##########
@@ -1050,6 +1050,12 @@
<td>Integer</td>
<td>Write batch size for any file format if it supports.</td>
</tr>
+ <tr>
+ <td><h5>write.skip-actions</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>This parameter only works when write-only is false., You can specify which actions to skip during the write process.<br />1. 'partition-expire': Skip the action of partition expiration.<br />2. 'snapshot-expire': Skip the action of snapshot expiration.<br />3. 'create-tag': Skip automatic tag creation.<br />Both can be configured at the same time: 'partition-expire,snapshot-expire,create-tag'.</td>
Review Comment:
maybe full or minor should be separate option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #4884: [core] Supports only execute specified actions during the writing process.
URL: https://github.com/apache/paimon/pull/4884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1912688885
##########
docs/layouts/shortcodes/generated/core_configuration.html:
##########
@@ -1050,6 +1050,12 @@
<td>Integer</td>
<td>Write batch size for any file format if it supports.</td>
</tr>
+ <tr>
+ <td><h5>write.skip-actions</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>This parameter only works when write-only is false., You can specify which actions to skip during the write process.<br />1. 'partition-expire': Skip the action of partition expiration.<br />2. 'snapshot-expire': Skip the action of snapshot expiration.<br />3. 'create-tag': Skip automatic tag creation.<br />Both can be configured at the same time: 'partition-expire,snapshot-expire,create-tag'.</td>
Review Comment:
I prefer to add a separate enable conf for each such post-commit behavior, with the default set to true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1994996331
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3227,6 +3271,56 @@ public String toString() {
}
}
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // All write actions will be performed, this is the default behavior.
+ ALL("all"),
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
+ SNAPSHOT_EXPIRE("snapshot-expire"),
+ TAG_AUTOMATIC_CREATION("tag-automatic-creation"),
+
+ // Actions during writing.
+ FULL_COMPACT("full-compact");
Review Comment:
Should I add parameter `write-action.compact.enable` separately to control whether to execute compaction?
In my original conception, `compact` is one of the `write-actions` types like the others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1950146933
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -2241,6 +2271,61 @@ public boolean writeOnly() {
return options.get(WRITE_ONLY);
}
+ public Set<WriteAction> writeActions() {
+ if (writeOnly()) {
+ return new HashSet<>(0);
+ }
+ return writeActions(options.get(WRITE_ACTIONS));
+ }
+
+ public static Set<WriteAction> writeActions(String str) {
+ if (StringUtils.isNullOrWhitespaceOnly(str)) {
+ return new HashSet<>(0);
+ }
+ return Arrays.stream(str.split(","))
+ .map(action -> WriteAction.valueOf(action.toUpperCase().replace('-', '_')))
+ .collect(Collectors.toCollection(HashSet::new));
+ }
+
+ public boolean doPartitionExpireAction(Set<WriteAction> doWriteActions) {
+ return doAllWriteActions(doWriteActions)
+ || doWriteActions.contains(WriteAction.PARTITION_EXPIRE);
+ }
+
+ public boolean doSnapshotExpireAction(Set<WriteAction> doWriteActions) {
Review Comment:
All these methods can be in `WriteAction`, they are just static methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on PR #4884:
URL: https://github.com/apache/paimon/pull/4884#issuecomment-2903113994
Do you intend to control the behavior of expire in compact action?
Based on our experience, the expire behavior is quite heavy and may cause the checkpoint to timeout during compact commit.
We expect to use separate and independent compact and expire job to handle these things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1914414510
##########
docs/layouts/shortcodes/generated/core_configuration.html:
##########
@@ -1050,6 +1050,12 @@
<td>Integer</td>
<td>Write batch size for any file format if it supports.</td>
</tr>
+ <tr>
+ <td><h5>write.skip-actions</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>This parameter only works when write-only is false., You can specify which actions to skip during the write process.<br />1. 'partition-expire': Skip the action of partition expiration.<br />2. 'snapshot-expire': Skip the action of snapshot expiration.<br />3. 'create-tag': Skip automatic tag creation.<br />Both can be configured at the same time: 'partition-expire,snapshot-expire,create-tag'.</td>
Review Comment:
Does this `compact` mean `full` or `minor` or both, or should it be separated into two actions `full_compact` and `minor_compact`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1912441021
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -442,6 +443,25 @@ public class CoreOptions implements Serializable {
"If set to true, compactions and snapshot expiration will be skipped. "
+ "This option is used along with dedicated compact jobs.");
+ public static final ConfigOption<String> WRITE_SKIP_ACTIONS =
+ key("write.skip-actions")
+ .stringType()
Review Comment:
This cannot be an enumType because this is a multi-option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports skipping specified actions during writing. [paimon]
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1912702370
##########
docs/layouts/shortcodes/generated/core_configuration.html:
##########
@@ -1050,6 +1050,12 @@
<td>Integer</td>
<td>Write batch size for any file format if it supports.</td>
</tr>
+ <tr>
+ <td><h5>write.skip-actions</h5></td>
+ <td style="word-wrap: break-word;">(none)</td>
+ <td>String</td>
+ <td>This parameter only works when write-only is false., You can specify which actions to skip during the write process.<br />1. 'partition-expire': Skip the action of partition expiration.<br />2. 'snapshot-expire': Skip the action of snapshot expiration.<br />3. 'create-tag': Skip automatic tag creation.<br />Both can be configured at the same time: 'partition-expire,snapshot-expire,create-tag'.</td>
Review Comment:
Thanks for review, If the parameters are separated, will it be more cumbersome to use ? For my needs, I need to skip `partition-expire` & `snapshot-expire` & `create tag` but retain `compact`, so I may need to configure three parameters to meet my needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r1957640148
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3227,6 +3271,56 @@ public String toString() {
}
}
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // All write actions will be performed, this is the default behavior.
+ ALL("all"),
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
+ SNAPSHOT_EXPIRE("snapshot-expire"),
+ TAG_AUTOMATIC_CREATION("tag-automatic-creation"),
+
+ // Actions during writing.
+ FULL_COMPACT("full-compact");
Review Comment:
I cannot get why you want to add this. Full compact should be separate option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Supports only execute specified actions during the writing process. [paimon]
Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on code in PR #4884:
URL: https://github.com/apache/paimon/pull/4884#discussion_r2108769538
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3227,6 +3271,56 @@ public String toString() {
}
}
+ /** Actions performed during table writing. */
+ public enum WriteAction {
+
+ // All write actions will be performed, this is the default behavior.
+ ALL("all"),
+
+ // Actions during commit.
+ PARTITION_EXPIRE("partition-expire"),
+ SNAPSHOT_EXPIRE("snapshot-expire"),
+ TAG_AUTOMATIC_CREATION("tag-automatic-creation"),
+
+ // Actions during writing.
+ FULL_COMPACT("full-compact");
Review Comment:
Is it more reasonable to directly use "compact"?
From the implementation code, this parameter only controls the compact behavior, while whether to perform a full compact or a minor compact is controlled by other parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org