You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "flyrain (via GitHub)" <gi...@apache.org> on 2025/02/08 01:54:11 UTC
[PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
flyrain opened a new pull request, #969:
URL: https://github.com/apache/polaris/pull/969
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1955748949
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
Review Comment:
I guess you don't mean namespaces but just "locations"?
I think having one policy map to one location is probably okay for now; I'm not sure if/how we plan to handle overlapping locations though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "gurukaraje (via GitHub)" <gi...@apache.org>.
gurukaraje commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1949775577
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
+ "type": "number",
+ "description": "A Unix timestamp. Remove orphan files created before this timestamp."
+ },
+ "location": {
+ "type": "string",
+ "description": "Customized directory other than table location to look for files in."
+ },
+ "config": {
+ "type": "object",
+ "description": "A map containing custom configuration properties. Please note that interoperability is not guaranteed.",
+ "additionalProperties": {
+ "type": ["string", "number", "boolean"]
+ }
+ }
+ },
+ "required": ["enable"],
+ "additionalProperties": false,
+ "examples": [
+ {
+ "version": "2025-02-03",
+ "enable": true,
+ "older_than": 1707315296,
Review Comment:
how about keeping all the 3 policies consistent with the format? metadata compaction and snapshot-expiry seem to follow the format:
`"enable": true
"config" : {
"key1" : value1
"key2" : value2
}`
But orphan file deletion has some of the properties `older_than` and `location` out side of config:
`"enable": true,
"older_than": 1707315296,
"location": "s3://my-bucket/my-table-location",
"config": {
"prefix_mismatch_mode": "ignore",
"my_key": "my_value"
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #969:
URL: https://github.com/apache/polaris/pull/969#issuecomment-2670131155
I believe we have reached a general consensus on policy management:
1. Design Document: The design document [1] was published over two months ago, triggering a lot of discussions. Stakeholders from various companies—including representatives from Dremio—actively contributed, and all feedback has been incorporated without any outstanding blockers.
2. Review Sessions: We conducted multiple review sessions with key stakeholders such as @jbonofre, @omarsmak, @RussellSpitzer and several others. Through these discussions, we aligned on the overall approach.
3. PR #945 was received strong support, not only from committer @eric-maynard but also from @omarsmak, one of the stakeholders from Dremio and @HonahX, confirming broad agreement.
1. https://docs.google.com/document/d/1Vuhw5b9-6KAol2vU3HUs9FJwcgWtiVVXMYhLtGmz53s/edit?tab=t.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1950099168
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
+ "type": "number",
+ "description": "A Unix timestamp. Remove orphan files created before this timestamp."
+ },
+ "location": {
+ "type": "string",
+ "description": "Customized directory other than table location to look for files in."
+ },
+ "config": {
+ "type": "object",
+ "description": "A map containing custom configuration properties. Please note that interoperability is not guaranteed.",
+ "additionalProperties": {
+ "type": ["string", "number", "boolean"]
+ }
+ }
+ },
+ "required": ["enable"],
+ "additionalProperties": false,
+ "examples": [
+ {
+ "version": "2025-02-03",
+ "enable": true,
+ "older_than": 1707315296,
Review Comment:
The config map is used for customized properties. For example, a TMS may provide an optional feature that only a subset of tables need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain merged PR #969:
URL: https://github.com/apache/polaris/pull/969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1956489461
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
Review Comment:
We could support that. One use case is that table files might be stored in different locations based on the `write.data.path` and/or `write.metadata.path` settings. This is generally not recommended though, due to issues like it makes credential vending harder. Are there any other use cases you have in mind, @ashvina?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1950101466
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
+ "type": "string",
+ "description": "Specifies a custom directory to search for files instead of the default table location. Use with caution—if set to a broad location (e.g., s3://my-bucket instead of s3://my-bucket/my-table-location), all unreferenced files in that path may be permanently deleted, including files from other tables. Following best practices, tables should be stored in separate locations to avoid accidental data loss."
Review Comment:
This hard to read in an IDE like IntelliJ as it is a single long line. Json doesn't support a way to break one line to multiple lines. This makes me think we may use the format `yaml` instead of `json`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1956515947
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
+ "type": "string",
+ "description": "Specifies a custom directory to search for files instead of the default table location. Use with caution—if set to a broad location (e.g., s3://my-bucket instead of s3://my-bucket/my-table-location), all unreferenced files in that path may be permanently deleted, including files from other tables. Following best practices, tables should be stored in separate locations to avoid accidental data loss."
Review Comment:
> put line breaks in the string
One of my versions was that :). It didn't help much, esp. in the IDE.
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
+ "type": "string",
+ "description": "Specifies a custom directory to search for files instead of the default table location. Use with caution—if set to a broad location (e.g., s3://my-bucket instead of s3://my-bucket/my-table-location), all unreferenced files in that path may be permanently deleted, including files from other tables. Following best practices, tables should be stored in separate locations to avoid accidental data loss."
Review Comment:
> put line breaks in the string
One of my versions was that :). It didn't help much, esp. in the IDE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1957031208
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
Review Comment:
Made it a string array type in the new commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1955752599
##########
polaris-core/src/main/resources/schemas/policies/system/metadata-compaction/2025-02-03.json:
##########
@@ -0,0 +1,37 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json",
Review Comment:
Does the JSON magically land at the location specified in the ID somehow? Or do we always need a followup PR?
Also, it looks a little funny to use dates here given that the date in the PR may not align with the date the schema actually becomes effective. In the worst case, we could merge two versions in one day. Maybe just an incrementing number is easier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1965250864
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,50 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version"
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
Review Comment:
I'd vote for the config map
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on PR #969:
URL: https://github.com/apache/polaris/pull/969#issuecomment-2674476085
> I believe we have reached a general consensus on policy management:
>
> 1. Design Document: The design document [1] was published over two months ago, triggering a lot of discussions. Stakeholders from various companies—including representatives from Dremio—actively contributed, and all feedback has been incorporated without any outstanding blockers.
>
> 2. Review Sessions: We conducted multiple review sessions with key stakeholders such as @jbonofre, @omarsmak, @RussellSpitzer and several others. Through these discussions, we aligned on the overall approach.
>
> 3. PR [Add data compaction policy schema #945](https://github.com/apache/polaris/pull/945) was received strong support, not only from committer @eric-maynard but also from @omarsmak, one of the stakeholders from Dremio and @HonahX, confirming broad agreement.
>
>
> [1] https://docs.google.com/document/d/1Vuhw5b9-6KAol2vU3HUs9FJwcgWtiVVXMYhLtGmz53s/edit?tab=t.0
(Not sure how the linked google doc is related to this PR)
The points I raised are about:
1. The changes committed to the production code base are not used, there is just no code that uses those - and won't be used soon. Those belong to a topic that's overall still WIP - hence I object merging it into `main` at this point.
2. These files land in a place that makes it extremely hard for arbitrary consumers to consume those. I do not think that those should live there.
3. #945 was merged without giving all contributors enough time to review.
I propose to move the work to a feature branch and go from there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1955750315
##########
polaris-core/src/main/resources/schemas/policies/system/snapshot-retention/2025-02-03.json:
##########
@@ -0,0 +1,39 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/snapshot-retention/2025-02-03.json",
+ "title": "Snapshot Retention Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table snapshot retention.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
Review Comment:
nit: `.` is probably not needed here; these aren't sentences
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1955749830
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
Review Comment:
1. What does `Inheritable` mean here?
2. `Polaris` seems redundant, all of these are going to be Polaris schemas right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1950102629
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
Review Comment:
Good point, made the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1956491580
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
Review Comment:
Overlapping locations is very dangerous. No orphan file removal should happen in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1955747412
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
+ "type": "string",
+ "description": "Specifies a custom directory to search for files instead of the default table location. Use with caution—if set to a broad location (e.g., s3://my-bucket instead of s3://my-bucket/my-table-location), all unreferenced files in that path may be permanently deleted, including files from other tables. Following best practices, tables should be stored in separate locations to avoid accidental data loss."
Review Comment:
The alternative is just to put line breaks in the string, and then preprocess it anywhere we want to strip out the whitespace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #969:
URL: https://github.com/apache/polaris/pull/969#issuecomment-2676335522
Update the doc link.
> The changes committed to the production code base are not used, there is just no code that uses those - and won't be used soon. Those belong to a topic that's overall still WIP - hence I object merging it into main at this point.
It's a common and recommended way to break a big feature into multiple small PRs, which not only makes the process iterating faster, but also improve review/code quality. A separate branch is only necessary when a feature introduces breaking changes, which is not the case for this PR.
> These files land in a place that makes it extremely hard for arbitrary consumers to consume those. I do not think that those should live there.
I'm surprised by this claim. It's a common practice to put non-code files in the dir `resources`, it's pretty easy for any Java code to consume. For example, #938 introduced a way to consume these schemas. We also plan to publish them in the website for other tools to consume.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1956558246
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
Review Comment:
A inheritable policy means it can be applied to the under layer entities. For example, all tables under a namespace get the policies if it is assigned to the namespace.
I'm OK to remove it or keep it. Keeping it provides a complete view for anyone who read the schema, but without too much context of Polaris.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "gurukaraje (via GitHub)" <gi...@apache.org>.
gurukaraje commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1949774336
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
Review Comment:
It might be easier to specify the file age for orphaned files in days instead of timestamp? I.e.
`"max_orphan_file_age_in_days" : 30`
instead of
`"older_than": 1707315296`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "leangjonathan (via GitHub)" <gi...@apache.org>.
leangjonathan commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1949712370
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
+ "type": "number",
+ "description": "A Unix timestamp. Remove orphan files created before this timestamp."
+ },
+ "location": {
+ "type": "string",
+ "description": "Customized directory other than table location to look for files in."
Review Comment:
I think this should have a warning that if you specify locations other than the table base location for example `s3://my-bucket` instead of `s3://my-bucket/my-table-location`, all files not referenced by the table will purged including potentially other table files if those files are stored in the specified path. I think this aligns with best practice notes that one shouldn't store tables under the same location
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1964910684
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,50 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version"
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
Review Comment:
> remove orphan policy can be even expressed in more than in just file age.
Can you name them? We can put them into schema if they are commonly used, otherwise, the config map would be the best place to be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1956579878
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
Review Comment:
Are all policies inheritable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1956636458
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
Review Comment:
No. For example, column masking policy are not inheritable, which makes more sense, and also that's what most engines do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #969:
URL: https://github.com/apache/polaris/pull/969#issuecomment-2664838241
> Adding a note here as we might need a couple more days to review this one. Please do not merge yet.
@snazy, this PR was filed 10 days ago, and it's a small one. Can you review it in a timely manner?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on PR #969:
URL: https://github.com/apache/polaris/pull/969#issuecomment-2668443493
My concern with this change, and it's quickly merged predecessor #945, is that AFAIK we don't have a consensus on policies in general. And specifically where these files should live, how those are accessed by users and how those are used by implementations.
I don't object on draft-code and draft-PRs, but I think that we should have a consensus _first_ on those rather big topics _before_ merging anything. Sure, this change is rather "small" or "innocent", but the topic is big.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1966572460
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,50 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version"
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
Review Comment:
If no extra field is suggested, we could keep it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "leangjonathan (via GitHub)" <gi...@apache.org>.
leangjonathan commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1949801808
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
Review Comment:
+1. Timestamp probably only makes sense for engines during operation runtime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "gurukaraje (via GitHub)" <gi...@apache.org>.
gurukaraje commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1949775577
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
+ "type": "number",
+ "description": "A Unix timestamp. Remove orphan files created before this timestamp."
+ },
+ "location": {
+ "type": "string",
+ "description": "Customized directory other than table location to look for files in."
+ },
+ "config": {
+ "type": "object",
+ "description": "A map containing custom configuration properties. Please note that interoperability is not guaranteed.",
+ "additionalProperties": {
+ "type": ["string", "number", "boolean"]
+ }
+ }
+ },
+ "required": ["enable"],
+ "additionalProperties": false,
+ "examples": [
+ {
+ "version": "2025-02-03",
+ "enable": true,
+ "older_than": 1707315296,
Review Comment:
how about keeping all the 3 policies consistent with the format? metadata compaction and snapshot-expiry seem to follow the format:
```
"enable": true
"config" : {
"key1" : value1
"key2" : value2
}
```
But orphan file deletion has some of the properties `older_than` and `location` out side of config:
```
"enable": true,
"older_than": 1707315296,
"location": "s3://my-bucket/my-table-location",
"config": {
"prefix_mismatch_mode": "ignore",
"my_key": "my_value"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "ashvina (via GitHub)" <gi...@apache.org>.
ashvina commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1955650269
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
+ "type": "number",
+ "description": "Specifies the maximum age (in days) for orphaned files before they are eligible for removal."
+ },
+ "location": {
Review Comment:
What do you think about making this property multi-value? This way, a user could support adding paths for multiple "namespaces".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1956563321
##########
polaris-core/src/main/resources/schemas/policies/system/metadata-compaction/2025-02-03.json:
##########
@@ -0,0 +1,37 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/metadata-compaction/2025-02-03.json",
Review Comment:
Unfortunately no. I hope I can publish it once for all based on the directory structure.
I want to keep them as the same for date as these are first batch, should be fine as nobody is using it. Once we release them at 1.0. We should follow the date schema strictly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1950102097
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,47 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal.",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version."
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "older_than": {
+ "type": "number",
+ "description": "A Unix timestamp. Remove orphan files created before this timestamp."
+ },
+ "location": {
+ "type": "string",
+ "description": "Customized directory other than table location to look for files in."
Review Comment:
Added a waning message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add policies for metadata compaction, orphan file removal and snapshot retention [polaris]
Posted by "omarsmak (via GitHub)" <gi...@apache.org>.
omarsmak commented on code in PR #969:
URL: https://github.com/apache/polaris/pull/969#discussion_r1964062880
##########
polaris-core/src/main/resources/schemas/policies/system/orphan-file-removal/2025-02-03.json:
##########
@@ -0,0 +1,50 @@
+{
+ "license": "Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)",
+ "$id": "https://polaris.apache.org/schemas/policies/system/orphan-file-removal/2025-02-03.json",
+ "title": "Orphan File Removal Policy",
+ "description": "Inheritable Polaris policy schema for Iceberg table orphan file removal",
+ "type": "object",
+ "properties": {
+ "version": {
+ "type": "string",
+ "const": "2025-02-03",
+ "description": "Schema version"
+ },
+ "enable": {
+ "type": "boolean",
+ "description": "Enable or disable orphan file removal."
+ },
+ "max_orphan_file_age_in_days": {
Review Comment:
@flyrain this also struggles me a bit, remove orphan policy can be even expressed in more than in just file age. We don't we opt for the `config` key similar to the other policies?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org