You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2025/12/09 13:07:58 UTC
[PR] [core] add audit fields into partitions sys table [paimon]
XiaoHongbo-Hope opened a new pull request, #6784:
URL: https://github.com/apache/paimon/pull/6784
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Add audit fields into partitions sys table
<!-- Linking this pull request to the issue -->
<!-- What is the purpose of the change -->
### Tests
- `PartitionsTableTest.testPartitionAuditFields` and `PartitionsTableTest.testPartitionAuditFieldsNull`
- `RestCatalogTest.testReadPartitionsTable`
<!-- List UT and IT cases to verify this change -->
### API and Format
For rest catalog, partitions sys table should be like this:
<img width="1666" height="316" alt="image" src="https://github.com/user-attachments/assets/2defcfda-d1db-4e07-bceb-176c1eea81c7" />
For other catalog, partitions sys table should be like this:
<img width="1346" height="326" alt="image" src="https://github.com/user-attachments/assets/5a293529-2783-4e22-b0e0-576dce62a8fd" />
<!-- Does this change affect API or storage format -->
### Documentation
`system-tables.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2619268337
##########
paimon-api/src/main/java/org/apache/paimon/partition/Partition.java:
##########
@@ -25,38 +25,127 @@
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import javax.annotation.Nullable;
+
import java.util.Map;
import java.util.Objects;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_BY;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_BY;
+
/** Represent a partition, including statistics and done flag. */
@JsonIgnoreProperties(ignoreUnknown = true)
@Public
public class Partition extends PartitionStatistics {
- private static final long serialVersionUID = 2L;
+ private static final long serialVersionUID = 3L;
public static final String FIELD_DONE = "done";
+ public static final String FIELD_OPTIONS = "options";
@JsonProperty(FIELD_DONE)
private final boolean done;
+ @Nullable
Review Comment:
> `@JsonInclude(JsonInclude.Include.NON_NULL)` for all new fields.
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2619074404
##########
docs/static/rest-catalog-open-api.yaml:
##########
@@ -2990,6 +2990,17 @@ components:
format: int64
done:
type: boolean
+ createdAt:
+ type: integer
+ format: int64
+ createdBy:
+ type: string
+ updatedBy:
Review Comment:
> Why not add updatedAt?
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2603014809
##########
paimon-core/src/main/java/org/apache/paimon/table/system/PartitionsTable.java:
##########
@@ -250,6 +254,14 @@ private InternalRow toRow(
.append(partitionValueString);
}
+ String createdBy = fileStoreTable.options().get(FIELD_CREATED_BY);
Review Comment:
This is wrong, we should get real partition createdBy from catalog server.
We should modify rest protocol to add these information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2619260866
##########
paimon-api/src/main/java/org/apache/paimon/partition/Partition.java:
##########
@@ -25,38 +25,127 @@
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import javax.annotation.Nullable;
+
import java.util.Map;
import java.util.Objects;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_BY;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_BY;
+
/** Represent a partition, including statistics and done flag. */
@JsonIgnoreProperties(ignoreUnknown = true)
@Public
public class Partition extends PartitionStatistics {
- private static final long serialVersionUID = 2L;
+ private static final long serialVersionUID = 3L;
public static final String FIELD_DONE = "done";
+ public static final String FIELD_OPTIONS = "options";
@JsonProperty(FIELD_DONE)
private final boolean done;
+ @Nullable
Review Comment:
`@JsonInclude(JsonInclude.Include.NON_NULL)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2619004640
##########
docs/static/rest-catalog-open-api.yaml:
##########
@@ -2990,6 +2990,17 @@ components:
format: int64
done:
type: boolean
+ createdAt:
+ type: integer
+ format: int64
+ createdBy:
+ type: string
+ updatedBy:
Review Comment:
> Why not add updatedAt?
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2618869397
##########
docs/static/rest-catalog-open-api.yaml:
##########
@@ -2990,6 +2990,17 @@ components:
format: int64
done:
type: boolean
+ createdAt:
+ type: integer
+ format: int64
+ createdBy:
+ type: string
+ updatedBy:
Review Comment:
Why not add updatedAt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #6784:
URL: https://github.com/apache/paimon/pull/6784
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2619260866
##########
paimon-api/src/main/java/org/apache/paimon/partition/Partition.java:
##########
@@ -25,38 +25,127 @@
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import javax.annotation.Nullable;
+
import java.util.Map;
import java.util.Objects;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_BY;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_BY;
+
/** Represent a partition, including statistics and done flag. */
@JsonIgnoreProperties(ignoreUnknown = true)
@Public
public class Partition extends PartitionStatistics {
- private static final long serialVersionUID = 2L;
+ private static final long serialVersionUID = 3L;
public static final String FIELD_DONE = "done";
+ public static final String FIELD_OPTIONS = "options";
@JsonProperty(FIELD_DONE)
private final boolean done;
+ @Nullable
Review Comment:
`@JsonInclude(JsonInclude.Include.NON_NULL)` for all new fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2619320312
##########
paimon-api/src/main/java/org/apache/paimon/partition/Partition.java:
##########
@@ -25,38 +25,127 @@
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonProperty;
+import javax.annotation.Nullable;
+
import java.util.Map;
import java.util.Objects;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_CREATED_BY;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_AT;
+import static org.apache.paimon.rest.responses.AuditRESTResponse.FIELD_UPDATED_BY;
+
/** Represent a partition, including statistics and done flag. */
@JsonIgnoreProperties(ignoreUnknown = true)
@Public
public class Partition extends PartitionStatistics {
- private static final long serialVersionUID = 2L;
+ private static final long serialVersionUID = 3L;
public static final String FIELD_DONE = "done";
+ public static final String FIELD_OPTIONS = "options";
@JsonProperty(FIELD_DONE)
private final boolean done;
+ @Nullable
Review Comment:
> `@JsonInclude(JsonInclude.Include.NON_NULL)` for all new fields.
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] add audit fields into partitions sys table [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #6784:
URL: https://github.com/apache/paimon/pull/6784#discussion_r2603091553
##########
paimon-core/src/main/java/org/apache/paimon/table/system/PartitionsTable.java:
##########
@@ -250,6 +254,14 @@ private InternalRow toRow(
.append(partitionValueString);
}
+ String createdBy = fileStoreTable.options().get(FIELD_CREATED_BY);
Review Comment:
> This is wrong, we should get real partition createdBy from catalog server.
>
> We should modify rest protocol to add these information.
Got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org