MonkeyCanCode opened a new pull request, #2898: URL: https://github.com/apache/polaris/pull/2898 Fix for https://github.com/apache/polaris/issues/2844. There are a lot more features we have in Polaris but not documented. This PR is for adding all of the features we have in polaris (including the deprecation as they are still valid but with deprecation warning). Also, I formatted the markdown table for easy readability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
MonkeyCanCode merged PR #2898: URL: https://github.com/apache/polaris/pull/2898 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
adutra commented on code in PR #2898:
URL: https://github.com/apache/polaris/pull/2898#discussion_r2465224178
##########
site/content/in-dev/unreleased/configuration.md:
##########
@@ -81,51 +81,88 @@ read-only mode, as Polaris only reads the configuration file once, at startup.
## Polaris Configuration Options Reference
-| Configuration Property | Default Value | Description |
-|----------------------------------------------------------------------------------------|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
-| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
-| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
-| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
-| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
-| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
-| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
-| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
-| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
-| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | Flag to enforce check if credential rotation. |
-| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `GCS`, `AZURE` | Define the catalog supported storage. Supported values are `S3`, `GCS`, `AZURE`, `FILE`. |
-| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
-| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
-| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
-| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
-| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
-| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
-| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
-| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
-| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
-| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
-| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
-| `polaris.log.mdc.aid` | `polaris` | Define the log context (e.g. MDC) AID. |
-| `polaris.log.mdc.sid` | `polaris-service` | Define the log context (e.g. MDC) SID. |
-| `polaris.rate-limiter.filter.type` | `no-op` | Define the Polaris rate limiter. Supported values are `no-op`, `token-bucket`. |
-| `polaris.rate-limiter.token-bucket.type` | `default` | Define the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.requests-per-second` | `9999` | Define the number of requests per second for the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.window` | `PT10S` | Define the window type for the token bucket rate limiter. |
-| `polaris.metrics.tags.<tag-name>=<tag-value>` | `application=Polaris` | Define arbitrary metric tags to include in every request. |
-| `polaris.metrics.realm-id-tag.api-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in API metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in HTTP request metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-max-cardinality` | `100` | The maximum cardinality for the `realm_id` tag in HTTP request metrics. |
-| `polaris.tasks.max-concurrent-tasks` | `100` | Define the max number of concurrent tasks. |
-| `polaris.tasks.max-queued-tasks` | `1000` | Define the max number of tasks in queue. |
-| `polaris.config.rollback.compaction.on-conflicts.enabled` | `false` | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end. |
-| `polaris.event-listener.type` | `no-op` | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`. |
-| `polaris.event-listener.aws-cloudwatch.log-group` | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener. |
-| `polaris.event-listener.aws-cloudwatch.log-stream` | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required. |
-| `polaris.event-listener.aws-cloudwatch.region` | `us-east-1` | Define the AWS region for the CloudWatch event listener. |
-| `polaris.event-listener.aws-cloudwatch.synchronous-mode` | `false` | Define whether log events are sent to CloudWatch synchronously. When set to true, events are sent synchronously which may impact performance but ensures immediate delivery. When false (default), events are sent asynchronously for better performance. |
+| Configuration Property | Default Value | Description |
+|----------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
+| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
+| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
+| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
+| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
+| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
+| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
+| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
+| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
+| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | If set to true, require that principals must rotate their credentials before being used for anything else. |
+| `polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `false` | If set to true, skip credential-subscoping indirection entirely whenever trying to obtain storage credentials for instantiating a FileIO. If 'true', no attempt is made to use StorageConfigs to generate table-specific storage credentials, but instead the default fallthrough of table-level credential properties or else provider-specific APPLICATION_DEFAULT credential-loading will be used for the FileIO. Typically this setting is used in single-tenant server deployments that don't rely on "credential-vending" and can use server-default environment variables or credential config files for all storage access, or in test/dev scenarios. |
+| `polaris.features."ALLOW_SETTING_S3_ENDPOINTS"` | `true` | If set to true (default), Polaris will permit S3 storage configurations to have custom endpoints. If set to false, Polaris will not accept catalog create and update requests that contain S3 endpoint properties. |
+| `polaris.features."ALLOW_TABLE_LOCATION_OVERLAP"` | `false` | (DEPRECATION) If set to true, allow one table's location to reside within another table's location. This is only enforced within a given namespace. |
+| `polaris.features."ALLOW_NAMESPACE_LOCATION_OVERLAP"` | `false` | If set to true, allow one namespace's location to reside within another namespace's location. This is only enforced within a parent catalog or namespace. |
+| `polaris.features."ALLOW_EXTERNAL_METADATA_FILE_LOCATION"` | `false` | If set to true, allows metadata files to be located outside the default metadata directory. |
+| `polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"` | `false` | If set to true, allows catalog URLs to overlap. |
+| `polaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows unstructured table locations. |
+| `polaris.features."ALLOW_EXTERNAL_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows tables to have external locations outside the default structure. |
+| `polaris.features."ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING"` | `true` | (DEPRECATION) If set to true, allow credential vending for external catalogs. |
+| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `AZURE`, `GCS` | (DEPRECATION) The list of supported storage types for a catalog. |
+| `polaris.features."CLEANUP_ON_NAMESPACE_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a namespace is dropped. |
+| `polaris.features."CLEANUP_ON_CATALOG_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a catalog is dropped. |
+| `polaris.features."DROP_WITH_PURGE_ENABLED"` | `false` | (DEPRECATION) If set to true, allows tables to be dropped with the purge parameter set to true. |
+| `polaris.features."PURGE_VIEW_METADATA_ON_DROP"` | `true` | If set to true, Polaris will attempt to delete view metadata files when a view is dropped. |
+| `polaris.features."STORAGE_CREDENTIAL_DURATION_SECONDS"` | `3600` | The duration of time that vended storage credentials are valid for. Support for longer (or shorter) durations is dependent on the storage provider. GCS current does not respect this value. |
+| `polaris.features."STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS"` | `1800` | How long to store storage credentials in the local cache. This should be less than STORAGE_CREDENTIAL_DURATION_SECONDS.key(). |
+| `polaris.features."MAX_METADATA_REFRESH_RETRIES"` | `2` | How many times to retry refreshing metadata when the previous error was retryable. |
+| `polaris.features."LIST_PAGINATION_ENABLED"` | `false` | If set to true, pagination for APIs like listTables is enabled. |
+| `polaris.features."ENABLE_GENERIC_TABLES"` | `true` | If true, the generic-tables endpoints are enabled. |
+| `polaris.features."ENTITY_CACHE_WEIGHER_TARGET"` | `104857600` | The maximum weight for the entity cache. This is a heuristic value without any particular unit of measurement. It roughly correlates with the total heap size of cached values. Fine-tuning requires experimentation in the specific deployment environment. |
+| `polaris.features."ENABLE_CATALOG_FEDERATION"` | `false` | If true, allows creating and using ExternalCatalogs containing ConnectionConfigInfos to perform federation to remote catalogs. |
+| `polaris.features."ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `false` | When enabled, allows RBAC operations to create synthetic entities for entities in federated catalogs that don't exist in the local metastore. |
+| `polaris.features."ENABLE_POLICY_STORE"` | `true` | If true, the policy-store endpoints are enabled. |
+| `polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"` | `ICEBERG_REST` | The list of supported catalog connection types for federation. |
+| `polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"` | `OAUTH`, `BEARER`, `SIGV4` | The list of supported authentication types for catalog federation. |
+| `polaris.features."ICEBERG_COMMIT_MAX_RETRIES"` | `4` | The max number of times to try committing to an Iceberg table. |
+| `polaris.features."ALLOW_SPECIFYING_FILE_IO_IMPL"` | `false` | Config key for whether to allow setting the FILE_IO_IMPL using catalog properties. Must only be enabled in dev/test environments, should not be in production systems. |
+| `polaris.features."ALLOW_INSECURE_STORAGE_TYPES"` | `false` | Allow usage of FileIO implementations that are considered insecure. Enabling this setting may expose the service to possibly severe security risks! This should only be set to 'true' for tests! |
+| `polaris.features."ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS"` | `false` | Rollback replace snapshots created by compaction which have polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback in their snapshot summary. |
+| `polaris.features."ADD_TRAILING_SLASH_TO_LOCATION"` | `true` | When set, the base location for a table or namespace will have `/` added as a suffix if not present. |
+| `polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"` | `false` | When set to true, Polaris will permit enabling the feature OPTIMIZED_SIBLING_CHECK for catalogs, this is done to prevent accidental enabling the feature in cases such as schema migrations, without backfill and hence leading to potential data integrity issues. This will be removed in 2.0.0 when polaris ships with the necessary migrations to backfill the index. |
+| `polaris.features."OPTIMIZED_SIBLING_CHECK"` | `false` | When set, an index is used to perform the sibling check between tables, views, and namespaces. New locations will be checked against previous ones based on components, so the new location /foo/bar/ will check for a sibling at /, /foo/ and /foo/bar/%. In order for this check to be correct, locations should end with a slash. See ADD_TRAILING_SLASH_TO_LOCATION for a way to enforce this when new locations are added. Only supported by the JDBC metastore. |
+| `polaris.features."DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED"` | `false` | When enabled, Iceberg tables and views created without a location specified will have a prefix applied to the location within the catalog's base location, rather than a location directly inside the parent namespace. Note that this requires ALLOW_EXTERNAL_TABLE_LOCATION to be enabled, but with OPTIMIZED_SIBLING_CHECK enabled it is still possible to enforce the uniqueness of table locations within a catalog. |
+| `polaris.features."ENABLE_CREDENTIAL_RESET"` | `true` | Flag to enable or disable the API to reset principal credentials. Defaults to enabled, but service providers may want to disable it. |
+| `polaris.features."ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `true` | If set to true (default), Polaris will allow setting or changing catalog property polaris.config.enable-sub-catalog-rbac-for-federated-catalogs.If set to false, Polaris will disallow setting or changing the above catalog property. |
+| `polaris.features."ALLOW_DROPPING_NON_EMPTY_PASSTHROUGH_FACADE_CATALOG"` | `false` | If enabled, allow dropping a passthrough-facade catalog even if it contains namespaces or tables. passthrough-facade catalogs may contain leftover entities when syncing with source catalog.In the short term these entities will be ignored, in the long term there will be method/background job to clean them up. |
+| `polaris.features."ENABLE_FINE_GRAINED_UPDATE_TABLE_PRIVILEGES"` | `true` | When true, enables finer grained update table privileges which are passed to the authorizer for update table operations. |
+| `polaris.features."ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING"` | `true` | If set to true (default), allow credential vending for external catalogs. Note this requires ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING to be true first. |
+| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
+| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
+| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
+| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
+| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
+| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
+| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
+| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
+| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
+| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
Review Comment:
Note: this can be done later if you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
MonkeyCanCode commented on code in PR #2898:
URL: https://github.com/apache/polaris/pull/2898#discussion_r2467729043
##########
site/content/in-dev/unreleased/configuration.md:
##########
@@ -81,51 +81,88 @@ read-only mode, as Polaris only reads the configuration file once, at startup.
## Polaris Configuration Options Reference
-| Configuration Property | Default Value | Description |
-|----------------------------------------------------------------------------------------|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
-| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
-| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
-| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
-| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
-| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
-| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
-| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
-| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
-| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | Flag to enforce check if credential rotation. |
-| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `GCS`, `AZURE` | Define the catalog supported storage. Supported values are `S3`, `GCS`, `AZURE`, `FILE`. |
-| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
-| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
-| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
-| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
-| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
-| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
-| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
-| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
-| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
-| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
-| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
-| `polaris.log.mdc.aid` | `polaris` | Define the log context (e.g. MDC) AID. |
-| `polaris.log.mdc.sid` | `polaris-service` | Define the log context (e.g. MDC) SID. |
-| `polaris.rate-limiter.filter.type` | `no-op` | Define the Polaris rate limiter. Supported values are `no-op`, `token-bucket`. |
-| `polaris.rate-limiter.token-bucket.type` | `default` | Define the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.requests-per-second` | `9999` | Define the number of requests per second for the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.window` | `PT10S` | Define the window type for the token bucket rate limiter. |
-| `polaris.metrics.tags.<tag-name>=<tag-value>` | `application=Polaris` | Define arbitrary metric tags to include in every request. |
-| `polaris.metrics.realm-id-tag.api-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in API metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in HTTP request metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-max-cardinality` | `100` | The maximum cardinality for the `realm_id` tag in HTTP request metrics. |
-| `polaris.tasks.max-concurrent-tasks` | `100` | Define the max number of concurrent tasks. |
-| `polaris.tasks.max-queued-tasks` | `1000` | Define the max number of tasks in queue. |
-| `polaris.config.rollback.compaction.on-conflicts.enabled` | `false` | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end. |
-| `polaris.event-listener.type` | `no-op` | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`. |
-| `polaris.event-listener.aws-cloudwatch.log-group` | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener. |
-| `polaris.event-listener.aws-cloudwatch.log-stream` | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required. |
-| `polaris.event-listener.aws-cloudwatch.region` | `us-east-1` | Define the AWS region for the CloudWatch event listener. |
-| `polaris.event-listener.aws-cloudwatch.synchronous-mode` | `false` | Define whether log events are sent to CloudWatch synchronously. When set to true, events are sent synchronously which may impact performance but ensures immediate delivery. When false (default), events are sent asynchronously for better performance. |
+| Configuration Property | Default Value | Description |
+|----------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
+| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
+| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
+| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
+| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
+| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
+| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
+| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
+| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
+| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | If set to true, require that principals must rotate their credentials before being used for anything else. |
+| `polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `false` | If set to true, skip credential-subscoping indirection entirely whenever trying to obtain storage credentials for instantiating a FileIO. If 'true', no attempt is made to use StorageConfigs to generate table-specific storage credentials, but instead the default fallthrough of table-level credential properties or else provider-specific APPLICATION_DEFAULT credential-loading will be used for the FileIO. Typically this setting is used in single-tenant server deployments that don't rely on "credential-vending" and can use server-default environment variables or credential config files for all storage access, or in test/dev scenarios. |
+| `polaris.features."ALLOW_SETTING_S3_ENDPOINTS"` | `true` | If set to true (default), Polaris will permit S3 storage configurations to have custom endpoints. If set to false, Polaris will not accept catalog create and update requests that contain S3 endpoint properties. |
+| `polaris.features."ALLOW_TABLE_LOCATION_OVERLAP"` | `false` | (DEPRECATION) If set to true, allow one table's location to reside within another table's location. This is only enforced within a given namespace. |
+| `polaris.features."ALLOW_NAMESPACE_LOCATION_OVERLAP"` | `false` | If set to true, allow one namespace's location to reside within another namespace's location. This is only enforced within a parent catalog or namespace. |
+| `polaris.features."ALLOW_EXTERNAL_METADATA_FILE_LOCATION"` | `false` | If set to true, allows metadata files to be located outside the default metadata directory. |
+| `polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"` | `false` | If set to true, allows catalog URLs to overlap. |
+| `polaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows unstructured table locations. |
+| `polaris.features."ALLOW_EXTERNAL_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows tables to have external locations outside the default structure. |
+| `polaris.features."ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING"` | `true` | (DEPRECATION) If set to true, allow credential vending for external catalogs. |
+| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `AZURE`, `GCS` | (DEPRECATION) The list of supported storage types for a catalog. |
+| `polaris.features."CLEANUP_ON_NAMESPACE_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a namespace is dropped. |
+| `polaris.features."CLEANUP_ON_CATALOG_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a catalog is dropped. |
+| `polaris.features."DROP_WITH_PURGE_ENABLED"` | `false` | (DEPRECATION) If set to true, allows tables to be dropped with the purge parameter set to true. |
+| `polaris.features."PURGE_VIEW_METADATA_ON_DROP"` | `true` | If set to true, Polaris will attempt to delete view metadata files when a view is dropped. |
+| `polaris.features."STORAGE_CREDENTIAL_DURATION_SECONDS"` | `3600` | The duration of time that vended storage credentials are valid for. Support for longer (or shorter) durations is dependent on the storage provider. GCS current does not respect this value. |
+| `polaris.features."STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS"` | `1800` | How long to store storage credentials in the local cache. This should be less than STORAGE_CREDENTIAL_DURATION_SECONDS.key(). |
+| `polaris.features."MAX_METADATA_REFRESH_RETRIES"` | `2` | How many times to retry refreshing metadata when the previous error was retryable. |
+| `polaris.features."LIST_PAGINATION_ENABLED"` | `false` | If set to true, pagination for APIs like listTables is enabled. |
+| `polaris.features."ENABLE_GENERIC_TABLES"` | `true` | If true, the generic-tables endpoints are enabled. |
+| `polaris.features."ENTITY_CACHE_WEIGHER_TARGET"` | `104857600` | The maximum weight for the entity cache. This is a heuristic value without any particular unit of measurement. It roughly correlates with the total heap size of cached values. Fine-tuning requires experimentation in the specific deployment environment. |
+| `polaris.features."ENABLE_CATALOG_FEDERATION"` | `false` | If true, allows creating and using ExternalCatalogs containing ConnectionConfigInfos to perform federation to remote catalogs. |
+| `polaris.features."ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `false` | When enabled, allows RBAC operations to create synthetic entities for entities in federated catalogs that don't exist in the local metastore. |
+| `polaris.features."ENABLE_POLICY_STORE"` | `true` | If true, the policy-store endpoints are enabled. |
+| `polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"` | `ICEBERG_REST` | The list of supported catalog connection types for federation. |
+| `polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"` | `OAUTH`, `BEARER`, `SIGV4` | The list of supported authentication types for catalog federation. |
+| `polaris.features."ICEBERG_COMMIT_MAX_RETRIES"` | `4` | The max number of times to try committing to an Iceberg table. |
+| `polaris.features."ALLOW_SPECIFYING_FILE_IO_IMPL"` | `false` | Config key for whether to allow setting the FILE_IO_IMPL using catalog properties. Must only be enabled in dev/test environments, should not be in production systems. |
+| `polaris.features."ALLOW_INSECURE_STORAGE_TYPES"` | `false` | Allow usage of FileIO implementations that are considered insecure. Enabling this setting may expose the service to possibly severe security risks! This should only be set to 'true' for tests! |
+| `polaris.features."ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS"` | `false` | Rollback replace snapshots created by compaction which have polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback in their snapshot summary. |
+| `polaris.features."ADD_TRAILING_SLASH_TO_LOCATION"` | `true` | When set, the base location for a table or namespace will have `/` added as a suffix if not present. |
+| `polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"` | `false` | When set to true, Polaris will permit enabling the feature OPTIMIZED_SIBLING_CHECK for catalogs, this is done to prevent accidental enabling the feature in cases such as schema migrations, without backfill and hence leading to potential data integrity issues. This will be removed in 2.0.0 when polaris ships with the necessary migrations to backfill the index. |
+| `polaris.features."OPTIMIZED_SIBLING_CHECK"` | `false` | When set, an index is used to perform the sibling check between tables, views, and namespaces. New locations will be checked against previous ones based on components, so the new location /foo/bar/ will check for a sibling at /, /foo/ and /foo/bar/%. In order for this check to be correct, locations should end with a slash. See ADD_TRAILING_SLASH_TO_LOCATION for a way to enforce this when new locations are added. Only supported by the JDBC metastore. |
+| `polaris.features."DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED"` | `false` | When enabled, Iceberg tables and views created without a location specified will have a prefix applied to the location within the catalog's base location, rather than a location directly inside the parent namespace. Note that this requires ALLOW_EXTERNAL_TABLE_LOCATION to be enabled, but with OPTIMIZED_SIBLING_CHECK enabled it is still possible to enforce the uniqueness of table locations within a catalog. |
+| `polaris.features."ENABLE_CREDENTIAL_RESET"` | `true` | Flag to enable or disable the API to reset principal credentials. Defaults to enabled, but service providers may want to disable it. |
+| `polaris.features."ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `true` | If set to true (default), Polaris will allow setting or changing catalog property polaris.config.enable-sub-catalog-rbac-for-federated-catalogs.If set to false, Polaris will disallow setting or changing the above catalog property. |
+| `polaris.features."ALLOW_DROPPING_NON_EMPTY_PASSTHROUGH_FACADE_CATALOG"` | `false` | If enabled, allow dropping a passthrough-facade catalog even if it contains namespaces or tables. passthrough-facade catalogs may contain leftover entities when syncing with source catalog.In the short term these entities will be ignored, in the long term there will be method/background job to clean them up. |
+| `polaris.features."ENABLE_FINE_GRAINED_UPDATE_TABLE_PRIVILEGES"` | `true` | When true, enables finer grained update table privileges which are passed to the authorizer for update table operations. |
+| `polaris.features."ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING"` | `true` | If set to true (default), allow credential vending for external catalogs. Note this requires ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING to be true first. |
+| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
+| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
+| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
+| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
+| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
+| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
+| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
+| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
+| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
+| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
Review Comment:
@adutra thanks for pointing it out. I was not checking if those are missing. They are now added. Please take another look when you have a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
MonkeyCanCode commented on code in PR #2898:
URL: https://github.com/apache/polaris/pull/2898#discussion_r2467603450
##########
site/content/in-dev/unreleased/configuration.md:
##########
@@ -81,51 +81,88 @@ read-only mode, as Polaris only reads the configuration file once, at startup.
## Polaris Configuration Options Reference
-| Configuration Property | Default Value | Description |
-|----------------------------------------------------------------------------------------|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
-| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
-| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
-| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
-| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
-| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
-| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
-| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
-| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
-| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | Flag to enforce check if credential rotation. |
-| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `GCS`, `AZURE` | Define the catalog supported storage. Supported values are `S3`, `GCS`, `AZURE`, `FILE`. |
-| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
-| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
-| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
-| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
-| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
-| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
-| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
-| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
-| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
-| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
-| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
-| `polaris.log.mdc.aid` | `polaris` | Define the log context (e.g. MDC) AID. |
-| `polaris.log.mdc.sid` | `polaris-service` | Define the log context (e.g. MDC) SID. |
-| `polaris.rate-limiter.filter.type` | `no-op` | Define the Polaris rate limiter. Supported values are `no-op`, `token-bucket`. |
-| `polaris.rate-limiter.token-bucket.type` | `default` | Define the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.requests-per-second` | `9999` | Define the number of requests per second for the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.window` | `PT10S` | Define the window type for the token bucket rate limiter. |
-| `polaris.metrics.tags.<tag-name>=<tag-value>` | `application=Polaris` | Define arbitrary metric tags to include in every request. |
-| `polaris.metrics.realm-id-tag.api-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in API metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in HTTP request metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-max-cardinality` | `100` | The maximum cardinality for the `realm_id` tag in HTTP request metrics. |
-| `polaris.tasks.max-concurrent-tasks` | `100` | Define the max number of concurrent tasks. |
-| `polaris.tasks.max-queued-tasks` | `1000` | Define the max number of tasks in queue. |
-| `polaris.config.rollback.compaction.on-conflicts.enabled` | `false` | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end. |
-| `polaris.event-listener.type` | `no-op` | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`. |
-| `polaris.event-listener.aws-cloudwatch.log-group` | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener. |
-| `polaris.event-listener.aws-cloudwatch.log-stream` | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required. |
-| `polaris.event-listener.aws-cloudwatch.region` | `us-east-1` | Define the AWS region for the CloudWatch event listener. |
-| `polaris.event-listener.aws-cloudwatch.synchronous-mode` | `false` | Define whether log events are sent to CloudWatch synchronously. When set to true, events are sent synchronously which may impact performance but ensures immediate delivery. When false (default), events are sent asynchronously for better performance. |
+| Configuration Property | Default Value | Description |
+|----------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
+| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
+| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
+| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
+| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
+| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
+| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
+| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
+| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
+| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | If set to true, require that principals must rotate their credentials before being used for anything else. |
+| `polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `false` | If set to true, skip credential-subscoping indirection entirely whenever trying to obtain storage credentials for instantiating a FileIO. If 'true', no attempt is made to use StorageConfigs to generate table-specific storage credentials, but instead the default fallthrough of table-level credential properties or else provider-specific APPLICATION_DEFAULT credential-loading will be used for the FileIO. Typically this setting is used in single-tenant server deployments that don't rely on "credential-vending" and can use server-default environment variables or credential config files for all storage access, or in test/dev scenarios. |
+| `polaris.features."ALLOW_SETTING_S3_ENDPOINTS"` | `true` | If set to true (default), Polaris will permit S3 storage configurations to have custom endpoints. If set to false, Polaris will not accept catalog create and update requests that contain S3 endpoint properties. |
+| `polaris.features."ALLOW_TABLE_LOCATION_OVERLAP"` | `false` | (DEPRECATION) If set to true, allow one table's location to reside within another table's location. This is only enforced within a given namespace. |
+| `polaris.features."ALLOW_NAMESPACE_LOCATION_OVERLAP"` | `false` | If set to true, allow one namespace's location to reside within another namespace's location. This is only enforced within a parent catalog or namespace. |
+| `polaris.features."ALLOW_EXTERNAL_METADATA_FILE_LOCATION"` | `false` | If set to true, allows metadata files to be located outside the default metadata directory. |
+| `polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"` | `false` | If set to true, allows catalog URLs to overlap. |
+| `polaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows unstructured table locations. |
+| `polaris.features."ALLOW_EXTERNAL_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows tables to have external locations outside the default structure. |
+| `polaris.features."ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING"` | `true` | (DEPRECATION) If set to true, allow credential vending for external catalogs. |
+| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `AZURE`, `GCS` | (DEPRECATION) The list of supported storage types for a catalog. |
+| `polaris.features."CLEANUP_ON_NAMESPACE_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a namespace is dropped. |
+| `polaris.features."CLEANUP_ON_CATALOG_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a catalog is dropped. |
+| `polaris.features."DROP_WITH_PURGE_ENABLED"` | `false` | (DEPRECATION) If set to true, allows tables to be dropped with the purge parameter set to true. |
+| `polaris.features."PURGE_VIEW_METADATA_ON_DROP"` | `true` | If set to true, Polaris will attempt to delete view metadata files when a view is dropped. |
+| `polaris.features."STORAGE_CREDENTIAL_DURATION_SECONDS"` | `3600` | The duration of time that vended storage credentials are valid for. Support for longer (or shorter) durations is dependent on the storage provider. GCS current does not respect this value. |
+| `polaris.features."STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS"` | `1800` | How long to store storage credentials in the local cache. This should be less than STORAGE_CREDENTIAL_DURATION_SECONDS.key(). |
+| `polaris.features."MAX_METADATA_REFRESH_RETRIES"` | `2` | How many times to retry refreshing metadata when the previous error was retryable. |
+| `polaris.features."LIST_PAGINATION_ENABLED"` | `false` | If set to true, pagination for APIs like listTables is enabled. |
+| `polaris.features."ENABLE_GENERIC_TABLES"` | `true` | If true, the generic-tables endpoints are enabled. |
+| `polaris.features."ENTITY_CACHE_WEIGHER_TARGET"` | `104857600` | The maximum weight for the entity cache. This is a heuristic value without any particular unit of measurement. It roughly correlates with the total heap size of cached values. Fine-tuning requires experimentation in the specific deployment environment. |
+| `polaris.features."ENABLE_CATALOG_FEDERATION"` | `false` | If true, allows creating and using ExternalCatalogs containing ConnectionConfigInfos to perform federation to remote catalogs. |
+| `polaris.features."ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `false` | When enabled, allows RBAC operations to create synthetic entities for entities in federated catalogs that don't exist in the local metastore. |
+| `polaris.features."ENABLE_POLICY_STORE"` | `true` | If true, the policy-store endpoints are enabled. |
+| `polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"` | `ICEBERG_REST` | The list of supported catalog connection types for federation. |
+| `polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"` | `OAUTH`, `BEARER`, `SIGV4` | The list of supported authentication types for catalog federation. |
+| `polaris.features."ICEBERG_COMMIT_MAX_RETRIES"` | `4` | The max number of times to try committing to an Iceberg table. |
+| `polaris.features."ALLOW_SPECIFYING_FILE_IO_IMPL"` | `false` | Config key for whether to allow setting the FILE_IO_IMPL using catalog properties. Must only be enabled in dev/test environments, should not be in production systems. |
+| `polaris.features."ALLOW_INSECURE_STORAGE_TYPES"` | `false` | Allow usage of FileIO implementations that are considered insecure. Enabling this setting may expose the service to possibly severe security risks! This should only be set to 'true' for tests! |
+| `polaris.features."ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS"` | `false` | Rollback replace snapshots created by compaction which have polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback in their snapshot summary. |
+| `polaris.features."ADD_TRAILING_SLASH_TO_LOCATION"` | `true` | When set, the base location for a table or namespace will have `/` added as a suffix if not present. |
+| `polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"` | `false` | When set to true, Polaris will permit enabling the feature OPTIMIZED_SIBLING_CHECK for catalogs, this is done to prevent accidental enabling the feature in cases such as schema migrations, without backfill and hence leading to potential data integrity issues. This will be removed in 2.0.0 when polaris ships with the necessary migrations to backfill the index. |
+| `polaris.features."OPTIMIZED_SIBLING_CHECK"` | `false` | When set, an index is used to perform the sibling check between tables, views, and namespaces. New locations will be checked against previous ones based on components, so the new location /foo/bar/ will check for a sibling at /, /foo/ and /foo/bar/%. In order for this check to be correct, locations should end with a slash. See ADD_TRAILING_SLASH_TO_LOCATION for a way to enforce this when new locations are added. Only supported by the JDBC metastore. |
+| `polaris.features."DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED"` | `false` | When enabled, Iceberg tables and views created without a location specified will have a prefix applied to the location within the catalog's base location, rather than a location directly inside the parent namespace. Note that this requires ALLOW_EXTERNAL_TABLE_LOCATION to be enabled, but with OPTIMIZED_SIBLING_CHECK enabled it is still possible to enforce the uniqueness of table locations within a catalog. |
+| `polaris.features."ENABLE_CREDENTIAL_RESET"` | `true` | Flag to enable or disable the API to reset principal credentials. Defaults to enabled, but service providers may want to disable it. |
+| `polaris.features."ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `true` | If set to true (default), Polaris will allow setting or changing catalog property polaris.config.enable-sub-catalog-rbac-for-federated-catalogs.If set to false, Polaris will disallow setting or changing the above catalog property. |
+| `polaris.features."ALLOW_DROPPING_NON_EMPTY_PASSTHROUGH_FACADE_CATALOG"` | `false` | If enabled, allow dropping a passthrough-facade catalog even if it contains namespaces or tables. passthrough-facade catalogs may contain leftover entities when syncing with source catalog.In the short term these entities will be ignored, in the long term there will be method/background job to clean them up. |
+| `polaris.features."ENABLE_FINE_GRAINED_UPDATE_TABLE_PRIVILEGES"` | `true` | When true, enables finer grained update table privileges which are passed to the authorizer for update table operations. |
+| `polaris.features."ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING"` | `true` | If set to true (default), allow credential vending for external catalogs. Note this requires ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING to be true first. |
+| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
+| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
Review Comment:
Oh yeah, I was only adding `polaris.features.*` earlier. This now added as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
adutra commented on code in PR #2898:
URL: https://github.com/apache/polaris/pull/2898#discussion_r2465183689
##########
site/content/in-dev/unreleased/configuration.md:
##########
@@ -81,51 +81,88 @@ read-only mode, as Polaris only reads the configuration file once, at startup.
## Polaris Configuration Options Reference
-| Configuration Property | Default Value | Description |
-|----------------------------------------------------------------------------------------|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
-| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
-| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
-| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
-| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
-| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
-| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
-| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
-| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
-| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | Flag to enforce check if credential rotation. |
-| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `GCS`, `AZURE` | Define the catalog supported storage. Supported values are `S3`, `GCS`, `AZURE`, `FILE`. |
-| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
-| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
-| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
-| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
-| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
-| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
-| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
-| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
-| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
-| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
-| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
-| `polaris.log.mdc.aid` | `polaris` | Define the log context (e.g. MDC) AID. |
-| `polaris.log.mdc.sid` | `polaris-service` | Define the log context (e.g. MDC) SID. |
-| `polaris.rate-limiter.filter.type` | `no-op` | Define the Polaris rate limiter. Supported values are `no-op`, `token-bucket`. |
-| `polaris.rate-limiter.token-bucket.type` | `default` | Define the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.requests-per-second` | `9999` | Define the number of requests per second for the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.window` | `PT10S` | Define the window type for the token bucket rate limiter. |
-| `polaris.metrics.tags.<tag-name>=<tag-value>` | `application=Polaris` | Define arbitrary metric tags to include in every request. |
-| `polaris.metrics.realm-id-tag.api-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in API metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in HTTP request metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-max-cardinality` | `100` | The maximum cardinality for the `realm_id` tag in HTTP request metrics. |
-| `polaris.tasks.max-concurrent-tasks` | `100` | Define the max number of concurrent tasks. |
-| `polaris.tasks.max-queued-tasks` | `1000` | Define the max number of tasks in queue. |
-| `polaris.config.rollback.compaction.on-conflicts.enabled` | `false` | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end. |
-| `polaris.event-listener.type` | `no-op` | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`. |
-| `polaris.event-listener.aws-cloudwatch.log-group` | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener. |
-| `polaris.event-listener.aws-cloudwatch.log-stream` | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required. |
-| `polaris.event-listener.aws-cloudwatch.region` | `us-east-1` | Define the AWS region for the CloudWatch event listener. |
-| `polaris.event-listener.aws-cloudwatch.synchronous-mode` | `false` | Define whether log events are sent to CloudWatch synchronously. When set to true, events are sent synchronously which may impact performance but ensures immediate delivery. When false (default), events are sent asynchronously for better performance. |
+| Configuration Property | Default Value | Description |
+|----------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
+| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
+| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
+| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
+| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
+| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
+| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
+| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
+| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
+| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | If set to true, require that principals must rotate their credentials before being used for anything else. |
+| `polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `false` | If set to true, skip credential-subscoping indirection entirely whenever trying to obtain storage credentials for instantiating a FileIO. If 'true', no attempt is made to use StorageConfigs to generate table-specific storage credentials, but instead the default fallthrough of table-level credential properties or else provider-specific APPLICATION_DEFAULT credential-loading will be used for the FileIO. Typically this setting is used in single-tenant server deployments that don't rely on "credential-vending" and can use server-default environment variables or credential config files for all storage access, or in test/dev scenarios. |
+| `polaris.features."ALLOW_SETTING_S3_ENDPOINTS"` | `true` | If set to true (default), Polaris will permit S3 storage configurations to have custom endpoints. If set to false, Polaris will not accept catalog create and update requests that contain S3 endpoint properties. |
+| `polaris.features."ALLOW_TABLE_LOCATION_OVERLAP"` | `false` | (DEPRECATION) If set to true, allow one table's location to reside within another table's location. This is only enforced within a given namespace. |
+| `polaris.features."ALLOW_NAMESPACE_LOCATION_OVERLAP"` | `false` | If set to true, allow one namespace's location to reside within another namespace's location. This is only enforced within a parent catalog or namespace. |
+| `polaris.features."ALLOW_EXTERNAL_METADATA_FILE_LOCATION"` | `false` | If set to true, allows metadata files to be located outside the default metadata directory. |
+| `polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"` | `false` | If set to true, allows catalog URLs to overlap. |
+| `polaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows unstructured table locations. |
+| `polaris.features."ALLOW_EXTERNAL_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows tables to have external locations outside the default structure. |
+| `polaris.features."ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING"` | `true` | (DEPRECATION) If set to true, allow credential vending for external catalogs. |
+| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `AZURE`, `GCS` | (DEPRECATION) The list of supported storage types for a catalog. |
+| `polaris.features."CLEANUP_ON_NAMESPACE_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a namespace is dropped. |
+| `polaris.features."CLEANUP_ON_CATALOG_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a catalog is dropped. |
+| `polaris.features."DROP_WITH_PURGE_ENABLED"` | `false` | (DEPRECATION) If set to true, allows tables to be dropped with the purge parameter set to true. |
+| `polaris.features."PURGE_VIEW_METADATA_ON_DROP"` | `true` | If set to true, Polaris will attempt to delete view metadata files when a view is dropped. |
+| `polaris.features."STORAGE_CREDENTIAL_DURATION_SECONDS"` | `3600` | The duration of time that vended storage credentials are valid for. Support for longer (or shorter) durations is dependent on the storage provider. GCS current does not respect this value. |
+| `polaris.features."STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS"` | `1800` | How long to store storage credentials in the local cache. This should be less than STORAGE_CREDENTIAL_DURATION_SECONDS.key(). |
+| `polaris.features."MAX_METADATA_REFRESH_RETRIES"` | `2` | How many times to retry refreshing metadata when the previous error was retryable. |
+| `polaris.features."LIST_PAGINATION_ENABLED"` | `false` | If set to true, pagination for APIs like listTables is enabled. |
+| `polaris.features."ENABLE_GENERIC_TABLES"` | `true` | If true, the generic-tables endpoints are enabled. |
+| `polaris.features."ENTITY_CACHE_WEIGHER_TARGET"` | `104857600` | The maximum weight for the entity cache. This is a heuristic value without any particular unit of measurement. It roughly correlates with the total heap size of cached values. Fine-tuning requires experimentation in the specific deployment environment. |
+| `polaris.features."ENABLE_CATALOG_FEDERATION"` | `false` | If true, allows creating and using ExternalCatalogs containing ConnectionConfigInfos to perform federation to remote catalogs. |
+| `polaris.features."ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `false` | When enabled, allows RBAC operations to create synthetic entities for entities in federated catalogs that don't exist in the local metastore. |
+| `polaris.features."ENABLE_POLICY_STORE"` | `true` | If true, the policy-store endpoints are enabled. |
+| `polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"` | `ICEBERG_REST` | The list of supported catalog connection types for federation. |
+| `polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"` | `OAUTH`, `BEARER`, `SIGV4` | The list of supported authentication types for catalog federation. |
+| `polaris.features."ICEBERG_COMMIT_MAX_RETRIES"` | `4` | The max number of times to try committing to an Iceberg table. |
+| `polaris.features."ALLOW_SPECIFYING_FILE_IO_IMPL"` | `false` | Config key for whether to allow setting the FILE_IO_IMPL using catalog properties. Must only be enabled in dev/test environments, should not be in production systems. |
+| `polaris.features."ALLOW_INSECURE_STORAGE_TYPES"` | `false` | Allow usage of FileIO implementations that are considered insecure. Enabling this setting may expose the service to possibly severe security risks! This should only be set to 'true' for tests! |
+| `polaris.features."ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS"` | `false` | Rollback replace snapshots created by compaction which have polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback in their snapshot summary. |
+| `polaris.features."ADD_TRAILING_SLASH_TO_LOCATION"` | `true` | When set, the base location for a table or namespace will have `/` added as a suffix if not present. |
+| `polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"` | `false` | When set to true, Polaris will permit enabling the feature OPTIMIZED_SIBLING_CHECK for catalogs, this is done to prevent accidental enabling the feature in cases such as schema migrations, without backfill and hence leading to potential data integrity issues. This will be removed in 2.0.0 when polaris ships with the necessary migrations to backfill the index. |
+| `polaris.features."OPTIMIZED_SIBLING_CHECK"` | `false` | When set, an index is used to perform the sibling check between tables, views, and namespaces. New locations will be checked against previous ones based on components, so the new location /foo/bar/ will check for a sibling at /, /foo/ and /foo/bar/%. In order for this check to be correct, locations should end with a slash. See ADD_TRAILING_SLASH_TO_LOCATION for a way to enforce this when new locations are added. Only supported by the JDBC metastore. |
+| `polaris.features."DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED"` | `false` | When enabled, Iceberg tables and views created without a location specified will have a prefix applied to the location within the catalog's base location, rather than a location directly inside the parent namespace. Note that this requires ALLOW_EXTERNAL_TABLE_LOCATION to be enabled, but with OPTIMIZED_SIBLING_CHECK enabled it is still possible to enforce the uniqueness of table locations within a catalog. |
+| `polaris.features."ENABLE_CREDENTIAL_RESET"` | `true` | Flag to enable or disable the API to reset principal credentials. Defaults to enabled, but service providers may want to disable it. |
+| `polaris.features."ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `true` | If set to true (default), Polaris will allow setting or changing catalog property polaris.config.enable-sub-catalog-rbac-for-federated-catalogs.If set to false, Polaris will disallow setting or changing the above catalog property. |
+| `polaris.features."ALLOW_DROPPING_NON_EMPTY_PASSTHROUGH_FACADE_CATALOG"` | `false` | If enabled, allow dropping a passthrough-facade catalog even if it contains namespaces or tables. passthrough-facade catalogs may contain leftover entities when syncing with source catalog.In the short term these entities will be ignored, in the long term there will be method/background job to clean them up. |
+| `polaris.features."ENABLE_FINE_GRAINED_UPDATE_TABLE_PRIVILEGES"` | `true` | When true, enables finer grained update table privileges which are passed to the authorizer for update table operations. |
+| `polaris.features."ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING"` | `true` | If set to true (default), allow credential vending for external catalogs. Note this requires ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING to be true first. |
+| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
+| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
Review Comment:
`polaris.authentication.type=internal` is missing.
##########
site/content/in-dev/unreleased/configuration.md:
##########
@@ -81,51 +81,88 @@ read-only mode, as Polaris only reads the configuration file once, at startup.
## Polaris Configuration Options Reference
-| Configuration Property | Default Value | Description |
-|----------------------------------------------------------------------------------------|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
-| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
-| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
-| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
-| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
-| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
-| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
-| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
-| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
-| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | Flag to enforce check if credential rotation. |
-| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `GCS`, `AZURE` | Define the catalog supported storage. Supported values are `S3`, `GCS`, `AZURE`, `FILE`. |
-| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
-| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
-| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
-| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
-| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
-| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
-| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
-| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
-| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
-| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
-| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
-| `polaris.log.mdc.aid` | `polaris` | Define the log context (e.g. MDC) AID. |
-| `polaris.log.mdc.sid` | `polaris-service` | Define the log context (e.g. MDC) SID. |
-| `polaris.rate-limiter.filter.type` | `no-op` | Define the Polaris rate limiter. Supported values are `no-op`, `token-bucket`. |
-| `polaris.rate-limiter.token-bucket.type` | `default` | Define the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.requests-per-second` | `9999` | Define the number of requests per second for the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.window` | `PT10S` | Define the window type for the token bucket rate limiter. |
-| `polaris.metrics.tags.<tag-name>=<tag-value>` | `application=Polaris` | Define arbitrary metric tags to include in every request. |
-| `polaris.metrics.realm-id-tag.api-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in API metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in HTTP request metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-max-cardinality` | `100` | The maximum cardinality for the `realm_id` tag in HTTP request metrics. |
-| `polaris.tasks.max-concurrent-tasks` | `100` | Define the max number of concurrent tasks. |
-| `polaris.tasks.max-queued-tasks` | `1000` | Define the max number of tasks in queue. |
-| `polaris.config.rollback.compaction.on-conflicts.enabled` | `false` | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end. |
-| `polaris.event-listener.type` | `no-op` | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`. |
-| `polaris.event-listener.aws-cloudwatch.log-group` | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener. |
-| `polaris.event-listener.aws-cloudwatch.log-stream` | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required. |
-| `polaris.event-listener.aws-cloudwatch.region` | `us-east-1` | Define the AWS region for the CloudWatch event listener. |
-| `polaris.event-listener.aws-cloudwatch.synchronous-mode` | `false` | Define whether log events are sent to CloudWatch synchronously. When set to true, events are sent synchronously which may impact performance but ensures immediate delivery. When false (default), events are sent asynchronously for better performance. |
+| Configuration Property | Default Value | Description |
+|----------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
+| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
+| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
+| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
+| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
+| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
+| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
+| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
+| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
+| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | If set to true, require that principals must rotate their credentials before being used for anything else. |
+| `polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `false` | If set to true, skip credential-subscoping indirection entirely whenever trying to obtain storage credentials for instantiating a FileIO. If 'true', no attempt is made to use StorageConfigs to generate table-specific storage credentials, but instead the default fallthrough of table-level credential properties or else provider-specific APPLICATION_DEFAULT credential-loading will be used for the FileIO. Typically this setting is used in single-tenant server deployments that don't rely on "credential-vending" and can use server-default environment variables or credential config files for all storage access, or in test/dev scenarios. |
+| `polaris.features."ALLOW_SETTING_S3_ENDPOINTS"` | `true` | If set to true (default), Polaris will permit S3 storage configurations to have custom endpoints. If set to false, Polaris will not accept catalog create and update requests that contain S3 endpoint properties. |
+| `polaris.features."ALLOW_TABLE_LOCATION_OVERLAP"` | `false` | (DEPRECATION) If set to true, allow one table's location to reside within another table's location. This is only enforced within a given namespace. |
+| `polaris.features."ALLOW_NAMESPACE_LOCATION_OVERLAP"` | `false` | If set to true, allow one namespace's location to reside within another namespace's location. This is only enforced within a parent catalog or namespace. |
+| `polaris.features."ALLOW_EXTERNAL_METADATA_FILE_LOCATION"` | `false` | If set to true, allows metadata files to be located outside the default metadata directory. |
+| `polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"` | `false` | If set to true, allows catalog URLs to overlap. |
+| `polaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows unstructured table locations. |
+| `polaris.features."ALLOW_EXTERNAL_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows tables to have external locations outside the default structure. |
+| `polaris.features."ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING"` | `true` | (DEPRECATION) If set to true, allow credential vending for external catalogs. |
+| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `AZURE`, `GCS` | (DEPRECATION) The list of supported storage types for a catalog. |
+| `polaris.features."CLEANUP_ON_NAMESPACE_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a namespace is dropped. |
+| `polaris.features."CLEANUP_ON_CATALOG_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a catalog is dropped. |
+| `polaris.features."DROP_WITH_PURGE_ENABLED"` | `false` | (DEPRECATION) If set to true, allows tables to be dropped with the purge parameter set to true. |
+| `polaris.features."PURGE_VIEW_METADATA_ON_DROP"` | `true` | If set to true, Polaris will attempt to delete view metadata files when a view is dropped. |
+| `polaris.features."STORAGE_CREDENTIAL_DURATION_SECONDS"` | `3600` | The duration of time that vended storage credentials are valid for. Support for longer (or shorter) durations is dependent on the storage provider. GCS current does not respect this value. |
+| `polaris.features."STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS"` | `1800` | How long to store storage credentials in the local cache. This should be less than STORAGE_CREDENTIAL_DURATION_SECONDS.key(). |
+| `polaris.features."MAX_METADATA_REFRESH_RETRIES"` | `2` | How many times to retry refreshing metadata when the previous error was retryable. |
+| `polaris.features."LIST_PAGINATION_ENABLED"` | `false` | If set to true, pagination for APIs like listTables is enabled. |
+| `polaris.features."ENABLE_GENERIC_TABLES"` | `true` | If true, the generic-tables endpoints are enabled. |
+| `polaris.features."ENTITY_CACHE_WEIGHER_TARGET"` | `104857600` | The maximum weight for the entity cache. This is a heuristic value without any particular unit of measurement. It roughly correlates with the total heap size of cached values. Fine-tuning requires experimentation in the specific deployment environment. |
+| `polaris.features."ENABLE_CATALOG_FEDERATION"` | `false` | If true, allows creating and using ExternalCatalogs containing ConnectionConfigInfos to perform federation to remote catalogs. |
+| `polaris.features."ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `false` | When enabled, allows RBAC operations to create synthetic entities for entities in federated catalogs that don't exist in the local metastore. |
+| `polaris.features."ENABLE_POLICY_STORE"` | `true` | If true, the policy-store endpoints are enabled. |
+| `polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"` | `ICEBERG_REST` | The list of supported catalog connection types for federation. |
+| `polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"` | `OAUTH`, `BEARER`, `SIGV4` | The list of supported authentication types for catalog federation. |
+| `polaris.features."ICEBERG_COMMIT_MAX_RETRIES"` | `4` | The max number of times to try committing to an Iceberg table. |
+| `polaris.features."ALLOW_SPECIFYING_FILE_IO_IMPL"` | `false` | Config key for whether to allow setting the FILE_IO_IMPL using catalog properties. Must only be enabled in dev/test environments, should not be in production systems. |
+| `polaris.features."ALLOW_INSECURE_STORAGE_TYPES"` | `false` | Allow usage of FileIO implementations that are considered insecure. Enabling this setting may expose the service to possibly severe security risks! This should only be set to 'true' for tests! |
+| `polaris.features."ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS"` | `false` | Rollback replace snapshots created by compaction which have polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback in their snapshot summary. |
+| `polaris.features."ADD_TRAILING_SLASH_TO_LOCATION"` | `true` | When set, the base location for a table or namespace will have `/` added as a suffix if not present. |
+| `polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"` | `false` | When set to true, Polaris will permit enabling the feature OPTIMIZED_SIBLING_CHECK for catalogs, this is done to prevent accidental enabling the feature in cases such as schema migrations, without backfill and hence leading to potential data integrity issues. This will be removed in 2.0.0 when polaris ships with the necessary migrations to backfill the index. |
+| `polaris.features."OPTIMIZED_SIBLING_CHECK"` | `false` | When set, an index is used to perform the sibling check between tables, views, and namespaces. New locations will be checked against previous ones based on components, so the new location /foo/bar/ will check for a sibling at /, /foo/ and /foo/bar/%. In order for this check to be correct, locations should end with a slash. See ADD_TRAILING_SLASH_TO_LOCATION for a way to enforce this when new locations are added. Only supported by the JDBC metastore. |
+| `polaris.features."DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED"` | `false` | When enabled, Iceberg tables and views created without a location specified will have a prefix applied to the location within the catalog's base location, rather than a location directly inside the parent namespace. Note that this requires ALLOW_EXTERNAL_TABLE_LOCATION to be enabled, but with OPTIMIZED_SIBLING_CHECK enabled it is still possible to enforce the uniqueness of table locations within a catalog. |
+| `polaris.features."ENABLE_CREDENTIAL_RESET"` | `true` | Flag to enable or disable the API to reset principal credentials. Defaults to enabled, but service providers may want to disable it. |
+| `polaris.features."ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `true` | If set to true (default), Polaris will allow setting or changing catalog property polaris.config.enable-sub-catalog-rbac-for-federated-catalogs.If set to false, Polaris will disallow setting or changing the above catalog property. |
+| `polaris.features."ALLOW_DROPPING_NON_EMPTY_PASSTHROUGH_FACADE_CATALOG"` | `false` | If enabled, allow dropping a passthrough-facade catalog even if it contains namespaces or tables. passthrough-facade catalogs may contain leftover entities when syncing with source catalog.In the short term these entities will be ignored, in the long term there will be method/background job to clean them up. |
+| `polaris.features."ENABLE_FINE_GRAINED_UPDATE_TABLE_PRIVILEGES"` | `true` | When true, enables finer grained update table privileges which are passed to the authorizer for update table operations. |
+| `polaris.features."ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING"` | `true` | If set to true (default), allow credential vending for external catalogs. Note this requires ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING to be true first. |
+| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
+| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
+| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
+| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
+| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
+| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
+| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
+| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
+| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
+| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
+| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
+| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
+| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
+| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
Review Comment:
Missing `polaris.credential-manager.type=default`
##########
site/content/in-dev/unreleased/configuration.md:
##########
@@ -81,51 +81,88 @@ read-only mode, as Polaris only reads the configuration file once, at startup.
## Polaris Configuration Options Reference
-| Configuration Property | Default Value | Description |
-|----------------------------------------------------------------------------------------|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
-| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
-| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
-| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
-| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
-| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
-| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
-| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
-| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
-| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | Flag to enforce check if credential rotation. |
-| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `GCS`, `AZURE` | Define the catalog supported storage. Supported values are `S3`, `GCS`, `AZURE`, `FILE`. |
-| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
-| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
-| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
-| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
-| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
-| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
-| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
-| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
-| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
-| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
-| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
-| `polaris.log.mdc.aid` | `polaris` | Define the log context (e.g. MDC) AID. |
-| `polaris.log.mdc.sid` | `polaris-service` | Define the log context (e.g. MDC) SID. |
-| `polaris.rate-limiter.filter.type` | `no-op` | Define the Polaris rate limiter. Supported values are `no-op`, `token-bucket`. |
-| `polaris.rate-limiter.token-bucket.type` | `default` | Define the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.requests-per-second` | `9999` | Define the number of requests per second for the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.window` | `PT10S` | Define the window type for the token bucket rate limiter. |
-| `polaris.metrics.tags.<tag-name>=<tag-value>` | `application=Polaris` | Define arbitrary metric tags to include in every request. |
-| `polaris.metrics.realm-id-tag.api-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in API metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in HTTP request metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-max-cardinality` | `100` | The maximum cardinality for the `realm_id` tag in HTTP request metrics. |
-| `polaris.tasks.max-concurrent-tasks` | `100` | Define the max number of concurrent tasks. |
-| `polaris.tasks.max-queued-tasks` | `1000` | Define the max number of tasks in queue. |
-| `polaris.config.rollback.compaction.on-conflicts.enabled` | `false` | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end. |
-| `polaris.event-listener.type` | `no-op` | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`. |
-| `polaris.event-listener.aws-cloudwatch.log-group` | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener. |
-| `polaris.event-listener.aws-cloudwatch.log-stream` | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required. |
-| `polaris.event-listener.aws-cloudwatch.region` | `us-east-1` | Define the AWS region for the CloudWatch event listener. |
-| `polaris.event-listener.aws-cloudwatch.synchronous-mode` | `false` | Define whether log events are sent to CloudWatch synchronously. When set to true, events are sent synchronously which may impact performance but ensures immediate delivery. When false (default), events are sent asynchronously for better performance. |
+| Configuration Property | Default Value | Description |
+|----------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
+| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
+| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
+| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
+| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
+| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
+| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
+| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
+| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
+| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | If set to true, require that principals must rotate their credentials before being used for anything else. |
+| `polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `false` | If set to true, skip credential-subscoping indirection entirely whenever trying to obtain storage credentials for instantiating a FileIO. If 'true', no attempt is made to use StorageConfigs to generate table-specific storage credentials, but instead the default fallthrough of table-level credential properties or else provider-specific APPLICATION_DEFAULT credential-loading will be used for the FileIO. Typically this setting is used in single-tenant server deployments that don't rely on "credential-vending" and can use server-default environment variables or credential config files for all storage access, or in test/dev scenarios. |
+| `polaris.features."ALLOW_SETTING_S3_ENDPOINTS"` | `true` | If set to true (default), Polaris will permit S3 storage configurations to have custom endpoints. If set to false, Polaris will not accept catalog create and update requests that contain S3 endpoint properties. |
+| `polaris.features."ALLOW_TABLE_LOCATION_OVERLAP"` | `false` | (DEPRECATION) If set to true, allow one table's location to reside within another table's location. This is only enforced within a given namespace. |
+| `polaris.features."ALLOW_NAMESPACE_LOCATION_OVERLAP"` | `false` | If set to true, allow one namespace's location to reside within another namespace's location. This is only enforced within a parent catalog or namespace. |
+| `polaris.features."ALLOW_EXTERNAL_METADATA_FILE_LOCATION"` | `false` | If set to true, allows metadata files to be located outside the default metadata directory. |
+| `polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"` | `false` | If set to true, allows catalog URLs to overlap. |
+| `polaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows unstructured table locations. |
+| `polaris.features."ALLOW_EXTERNAL_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows tables to have external locations outside the default structure. |
+| `polaris.features."ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING"` | `true` | (DEPRECATION) If set to true, allow credential vending for external catalogs. |
+| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `AZURE`, `GCS` | (DEPRECATION) The list of supported storage types for a catalog. |
+| `polaris.features."CLEANUP_ON_NAMESPACE_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a namespace is dropped. |
+| `polaris.features."CLEANUP_ON_CATALOG_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a catalog is dropped. |
+| `polaris.features."DROP_WITH_PURGE_ENABLED"` | `false` | (DEPRECATION) If set to true, allows tables to be dropped with the purge parameter set to true. |
+| `polaris.features."PURGE_VIEW_METADATA_ON_DROP"` | `true` | If set to true, Polaris will attempt to delete view metadata files when a view is dropped. |
+| `polaris.features."STORAGE_CREDENTIAL_DURATION_SECONDS"` | `3600` | The duration of time that vended storage credentials are valid for. Support for longer (or shorter) durations is dependent on the storage provider. GCS current does not respect this value. |
+| `polaris.features."STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS"` | `1800` | How long to store storage credentials in the local cache. This should be less than STORAGE_CREDENTIAL_DURATION_SECONDS.key(). |
+| `polaris.features."MAX_METADATA_REFRESH_RETRIES"` | `2` | How many times to retry refreshing metadata when the previous error was retryable. |
+| `polaris.features."LIST_PAGINATION_ENABLED"` | `false` | If set to true, pagination for APIs like listTables is enabled. |
+| `polaris.features."ENABLE_GENERIC_TABLES"` | `true` | If true, the generic-tables endpoints are enabled. |
+| `polaris.features."ENTITY_CACHE_WEIGHER_TARGET"` | `104857600` | The maximum weight for the entity cache. This is a heuristic value without any particular unit of measurement. It roughly correlates with the total heap size of cached values. Fine-tuning requires experimentation in the specific deployment environment. |
+| `polaris.features."ENABLE_CATALOG_FEDERATION"` | `false` | If true, allows creating and using ExternalCatalogs containing ConnectionConfigInfos to perform federation to remote catalogs. |
+| `polaris.features."ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `false` | When enabled, allows RBAC operations to create synthetic entities for entities in federated catalogs that don't exist in the local metastore. |
+| `polaris.features."ENABLE_POLICY_STORE"` | `true` | If true, the policy-store endpoints are enabled. |
+| `polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"` | `ICEBERG_REST` | The list of supported catalog connection types for federation. |
+| `polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"` | `OAUTH`, `BEARER`, `SIGV4` | The list of supported authentication types for catalog federation. |
+| `polaris.features."ICEBERG_COMMIT_MAX_RETRIES"` | `4` | The max number of times to try committing to an Iceberg table. |
+| `polaris.features."ALLOW_SPECIFYING_FILE_IO_IMPL"` | `false` | Config key for whether to allow setting the FILE_IO_IMPL using catalog properties. Must only be enabled in dev/test environments, should not be in production systems. |
+| `polaris.features."ALLOW_INSECURE_STORAGE_TYPES"` | `false` | Allow usage of FileIO implementations that are considered insecure. Enabling this setting may expose the service to possibly severe security risks! This should only be set to 'true' for tests! |
+| `polaris.features."ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS"` | `false` | Rollback replace snapshots created by compaction which have polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback in their snapshot summary. |
+| `polaris.features."ADD_TRAILING_SLASH_TO_LOCATION"` | `true` | When set, the base location for a table or namespace will have `/` added as a suffix if not present. |
+| `polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"` | `false` | When set to true, Polaris will permit enabling the feature OPTIMIZED_SIBLING_CHECK for catalogs, this is done to prevent accidental enabling the feature in cases such as schema migrations, without backfill and hence leading to potential data integrity issues. This will be removed in 2.0.0 when polaris ships with the necessary migrations to backfill the index. |
+| `polaris.features."OPTIMIZED_SIBLING_CHECK"` | `false` | When set, an index is used to perform the sibling check between tables, views, and namespaces. New locations will be checked against previous ones based on components, so the new location /foo/bar/ will check for a sibling at /, /foo/ and /foo/bar/%. In order for this check to be correct, locations should end with a slash. See ADD_TRAILING_SLASH_TO_LOCATION for a way to enforce this when new locations are added. Only supported by the JDBC metastore. |
+| `polaris.features."DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED"` | `false` | When enabled, Iceberg tables and views created without a location specified will have a prefix applied to the location within the catalog's base location, rather than a location directly inside the parent namespace. Note that this requires ALLOW_EXTERNAL_TABLE_LOCATION to be enabled, but with OPTIMIZED_SIBLING_CHECK enabled it is still possible to enforce the uniqueness of table locations within a catalog. |
+| `polaris.features."ENABLE_CREDENTIAL_RESET"` | `true` | Flag to enable or disable the API to reset principal credentials. Defaults to enabled, but service providers may want to disable it. |
+| `polaris.features."ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `true` | If set to true (default), Polaris will allow setting or changing catalog property polaris.config.enable-sub-catalog-rbac-for-federated-catalogs.If set to false, Polaris will disallow setting or changing the above catalog property. |
+| `polaris.features."ALLOW_DROPPING_NON_EMPTY_PASSTHROUGH_FACADE_CATALOG"` | `false` | If enabled, allow dropping a passthrough-facade catalog even if it contains namespaces or tables. passthrough-facade catalogs may contain leftover entities when syncing with source catalog.In the short term these entities will be ignored, in the long term there will be method/background job to clean them up. |
+| `polaris.features."ENABLE_FINE_GRAINED_UPDATE_TABLE_PRIVILEGES"` | `true` | When true, enables finer grained update table privileges which are passed to the authorizer for update table operations. |
+| `polaris.features."ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING"` | `true` | If set to true (default), allow credential vending for external catalogs. Note this requires ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING to be true first. |
+| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
+| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
+| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
+| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
+| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
+| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
+| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
+| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
+| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
+| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
Review Comment:
All the `polaris.oidc.*` properties are missing:
```properties
# OIDC Principals mapping
polaris.oidc.principal-mapper.type=default
# polaris.oidc.principal-mapper.id-claim-path=sub
# polaris.oidc.principal-mapper.name-claim-path=preferred_username
# Per-tenant overrides:
# polaris.oidc.idp1.principal-mapper.id-claim-path=polaris/principal_id
# polaris.oidc.idp1.principal-mapper.name-claim-path=polaris/principal_name
# OIDC Principal roles mapping
polaris.oidc.principal-roles-mapper.type=default
# Principal role mapping is done through quarkus.oidc.roles.role-claim-path
# The properties below define how the roles mapped by Quarkus are converted to Polaris roles
# polaris.oidc.principal-roles-mapper.filter=PRINCIPAL_ROLE:.*
# polaris.oidc.principal-roles-mapper.mappings[0].regex=PRINCIPAL_ROLE:(.*)
# polaris.oidc.principal-roles-mapper.mappings[0].replacement=PRINCIPAL_ROLE:$1
# Per-tenant overrides:
# polaris.oidc.idp1.principal-roles-mapper.type=custom
# polaris.oidc.idp1.principal-roles-mapper.filter=POLARIS_ROLE:.*
# polaris.oidc.idp1.principal-roles-mapper.mappings[0].regex=POLARIS_ROLE:(.*)
# polaris.oidc.idp1.principal-roles-mapper.mappings[0].replacement=POLARIS_ROLE:$1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
MonkeyCanCode commented on code in PR #2898:
URL: https://github.com/apache/polaris/pull/2898#discussion_r2467728301
##########
site/content/in-dev/unreleased/configuration.md:
##########
@@ -81,51 +81,88 @@ read-only mode, as Polaris only reads the configuration file once, at startup.
## Polaris Configuration Options Reference
-| Configuration Property | Default Value | Description |
-|----------------------------------------------------------------------------------------|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
-| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
-| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
-| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
-| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
-| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
-| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
-| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
-| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
-| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | Flag to enforce check if credential rotation. |
-| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `GCS`, `AZURE` | Define the catalog supported storage. Supported values are `S3`, `GCS`, `AZURE`, `FILE`. |
-| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
-| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
-| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
-| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
-| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
-| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
-| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
-| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
-| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
-| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
-| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
-| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
-| `polaris.log.mdc.aid` | `polaris` | Define the log context (e.g. MDC) AID. |
-| `polaris.log.mdc.sid` | `polaris-service` | Define the log context (e.g. MDC) SID. |
-| `polaris.rate-limiter.filter.type` | `no-op` | Define the Polaris rate limiter. Supported values are `no-op`, `token-bucket`. |
-| `polaris.rate-limiter.token-bucket.type` | `default` | Define the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.requests-per-second` | `9999` | Define the number of requests per second for the token bucket rate limiter. |
-| `polaris.rate-limiter.token-bucket.window` | `PT10S` | Define the window type for the token bucket rate limiter. |
-| `polaris.metrics.tags.<tag-name>=<tag-value>` | `application=Polaris` | Define arbitrary metric tags to include in every request. |
-| `polaris.metrics.realm-id-tag.api-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in API metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-enabled` | `false` | Whether to enable the `realm_id` metric tag in HTTP request metrics. |
-| `polaris.metrics.realm-id-tag.http-metrics-max-cardinality` | `100` | The maximum cardinality for the `realm_id` tag in HTTP request metrics. |
-| `polaris.tasks.max-concurrent-tasks` | `100` | Define the max number of concurrent tasks. |
-| `polaris.tasks.max-queued-tasks` | `1000` | Define the max number of tasks in queue. |
-| `polaris.config.rollback.compaction.on-conflicts.enabled` | `false` | When set to true Polaris will apply the deconfliction by rollbacking those REPLACE operations snapshots which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary set to `rollback`, to resolve conflicts at the server end. |
-| `polaris.event-listener.type` | `no-op` | Define the Polaris event listener type. Supported values are `no-op`, `aws-cloudwatch`. |
-| `polaris.event-listener.aws-cloudwatch.log-group` | `polaris-cloudwatch-default-group` | Define the AWS CloudWatch log group name for the event listener. |
-| `polaris.event-listener.aws-cloudwatch.log-stream` | `polaris-cloudwatch-default-stream`| Define the AWS CloudWatch log stream name for the event listener. Ensure that Polaris' IAM credentials have the following actions: "PutLogEvents", "DescribeLogStreams", and "DescribeLogGroups" on the specified log stream/group. If the specified log stream/group does not exist, then "CreateLogStream" and "CreateLogGroup" will also be required. |
-| `polaris.event-listener.aws-cloudwatch.region` | `us-east-1` | Define the AWS region for the CloudWatch event listener. |
-| `polaris.event-listener.aws-cloudwatch.synchronous-mode` | `false` | Define whether log events are sent to CloudWatch synchronously. When set to true, events are sent synchronously which may impact performance but ensures immediate delivery. When false (default), events are sent asynchronously for better performance. |
+| Configuration Property | Default Value | Description |
+|----------------------------------------------------------------------------------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `polaris.persistence.type` | `relational-jdbc` | Define the persistence backend used by Polaris (`in-memory`, `relational-jdbc`, `eclipse-link` (deprecated)). See [Configuring Apache Polaris for Production)[{{% ref "configuring-polaris-for-production.md" %}}) |
+| `polaris.persistence.relational.jdbc.max-retries` | `1` | Total number of retries JDBC persistence will attempt on connection resets or serialization failures before giving up. |
+| `polaris.persistence.relational.jdbc.max_duaration_in_ms` | `5000 ms` | Max time interval (ms) since the start of a transaction when retries can be attempted. |
+| `polaris.persistence.relational.jdbc.initial_delay_in_ms` | `100 ms` | Initial delay before retrying. The delay is doubled after each retry. |
+| `polaris.persistence.eclipselink.configurationFile` | | Define the location of the `persistence.xml`. By default, it's the built-in `persistence.xml` in use. |
+| `polaris.persistence.eclipselink.persistenceUnit` | `polaris` | Define the name of the persistence unit to use, as defined in the `persistence.xml`. |
+| `polaris.realm-context.type` | `default` | Define the type of the Polaris realm to use. |
+| `polaris.realm-context.realms` | `POLARIS` | Define the list of realms to use. |
+| `polaris.realm-context.header-name` | `Polaris-Realm` | Define the header name defining the realm context. |
+| `polaris.features."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"` | `false` | If set to true, require that principals must rotate their credentials before being used for anything else. |
+| `polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `false` | If set to true, skip credential-subscoping indirection entirely whenever trying to obtain storage credentials for instantiating a FileIO. If 'true', no attempt is made to use StorageConfigs to generate table-specific storage credentials, but instead the default fallthrough of table-level credential properties or else provider-specific APPLICATION_DEFAULT credential-loading will be used for the FileIO. Typically this setting is used in single-tenant server deployments that don't rely on "credential-vending" and can use server-default environment variables or credential config files for all storage access, or in test/dev scenarios. |
+| `polaris.features."ALLOW_SETTING_S3_ENDPOINTS"` | `true` | If set to true (default), Polaris will permit S3 storage configurations to have custom endpoints. If set to false, Polaris will not accept catalog create and update requests that contain S3 endpoint properties. |
+| `polaris.features."ALLOW_TABLE_LOCATION_OVERLAP"` | `false` | (DEPRECATION) If set to true, allow one table's location to reside within another table's location. This is only enforced within a given namespace. |
+| `polaris.features."ALLOW_NAMESPACE_LOCATION_OVERLAP"` | `false` | If set to true, allow one namespace's location to reside within another namespace's location. This is only enforced within a parent catalog or namespace. |
+| `polaris.features."ALLOW_EXTERNAL_METADATA_FILE_LOCATION"` | `false` | If set to true, allows metadata files to be located outside the default metadata directory. |
+| `polaris.features."ALLOW_OVERLAPPING_CATALOG_URLS"` | `false` | If set to true, allows catalog URLs to overlap. |
+| `polaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows unstructured table locations. |
+| `polaris.features."ALLOW_EXTERNAL_TABLE_LOCATION"` | `false` | (DEPRECATION) If set to true, allows tables to have external locations outside the default structure. |
+| `polaris.features."ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING"` | `true` | (DEPRECATION) If set to true, allow credential vending for external catalogs. |
+| `polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"` | `S3`, `AZURE`, `GCS` | (DEPRECATION) The list of supported storage types for a catalog. |
+| `polaris.features."CLEANUP_ON_NAMESPACE_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a namespace is dropped. |
+| `polaris.features."CLEANUP_ON_CATALOG_DROP"` | `false` | (DEPRECATION) If set to true, clean up data when a catalog is dropped. |
+| `polaris.features."DROP_WITH_PURGE_ENABLED"` | `false` | (DEPRECATION) If set to true, allows tables to be dropped with the purge parameter set to true. |
+| `polaris.features."PURGE_VIEW_METADATA_ON_DROP"` | `true` | If set to true, Polaris will attempt to delete view metadata files when a view is dropped. |
+| `polaris.features."STORAGE_CREDENTIAL_DURATION_SECONDS"` | `3600` | The duration of time that vended storage credentials are valid for. Support for longer (or shorter) durations is dependent on the storage provider. GCS current does not respect this value. |
+| `polaris.features."STORAGE_CREDENTIAL_CACHE_DURATION_SECONDS"` | `1800` | How long to store storage credentials in the local cache. This should be less than STORAGE_CREDENTIAL_DURATION_SECONDS.key(). |
+| `polaris.features."MAX_METADATA_REFRESH_RETRIES"` | `2` | How many times to retry refreshing metadata when the previous error was retryable. |
+| `polaris.features."LIST_PAGINATION_ENABLED"` | `false` | If set to true, pagination for APIs like listTables is enabled. |
+| `polaris.features."ENABLE_GENERIC_TABLES"` | `true` | If true, the generic-tables endpoints are enabled. |
+| `polaris.features."ENTITY_CACHE_WEIGHER_TARGET"` | `104857600` | The maximum weight for the entity cache. This is a heuristic value without any particular unit of measurement. It roughly correlates with the total heap size of cached values. Fine-tuning requires experimentation in the specific deployment environment. |
+| `polaris.features."ENABLE_CATALOG_FEDERATION"` | `false` | If true, allows creating and using ExternalCatalogs containing ConnectionConfigInfos to perform federation to remote catalogs. |
+| `polaris.features."ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `false` | When enabled, allows RBAC operations to create synthetic entities for entities in federated catalogs that don't exist in the local metastore. |
+| `polaris.features."ENABLE_POLICY_STORE"` | `true` | If true, the policy-store endpoints are enabled. |
+| `polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"` | `ICEBERG_REST` | The list of supported catalog connection types for federation. |
+| `polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"` | `OAUTH`, `BEARER`, `SIGV4` | The list of supported authentication types for catalog federation. |
+| `polaris.features."ICEBERG_COMMIT_MAX_RETRIES"` | `4` | The max number of times to try committing to an Iceberg table. |
+| `polaris.features."ALLOW_SPECIFYING_FILE_IO_IMPL"` | `false` | Config key for whether to allow setting the FILE_IO_IMPL using catalog properties. Must only be enabled in dev/test environments, should not be in production systems. |
+| `polaris.features."ALLOW_INSECURE_STORAGE_TYPES"` | `false` | Allow usage of FileIO implementations that are considered insecure. Enabling this setting may expose the service to possibly severe security risks! This should only be set to 'true' for tests! |
+| `polaris.features."ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS"` | `false` | Rollback replace snapshots created by compaction which have polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback in their snapshot summary. |
+| `polaris.features."ADD_TRAILING_SLASH_TO_LOCATION"` | `true` | When set, the base location for a table or namespace will have `/` added as a suffix if not present. |
+| `polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"` | `false` | When set to true, Polaris will permit enabling the feature OPTIMIZED_SIBLING_CHECK for catalogs, this is done to prevent accidental enabling the feature in cases such as schema migrations, without backfill and hence leading to potential data integrity issues. This will be removed in 2.0.0 when polaris ships with the necessary migrations to backfill the index. |
+| `polaris.features."OPTIMIZED_SIBLING_CHECK"` | `false` | When set, an index is used to perform the sibling check between tables, views, and namespaces. New locations will be checked against previous ones based on components, so the new location /foo/bar/ will check for a sibling at /, /foo/ and /foo/bar/%. In order for this check to be correct, locations should end with a slash. See ADD_TRAILING_SLASH_TO_LOCATION for a way to enforce this when new locations are added. Only supported by the JDBC metastore. |
+| `polaris.features."DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED"` | `false` | When enabled, Iceberg tables and views created without a location specified will have a prefix applied to the location within the catalog's base location, rather than a location directly inside the parent namespace. Note that this requires ALLOW_EXTERNAL_TABLE_LOCATION to be enabled, but with OPTIMIZED_SIBLING_CHECK enabled it is still possible to enforce the uniqueness of table locations within a catalog. |
+| `polaris.features."ENABLE_CREDENTIAL_RESET"` | `true` | Flag to enable or disable the API to reset principal credentials. Defaults to enabled, but service providers may want to disable it. |
+| `polaris.features."ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS"` | `true` | If set to true (default), Polaris will allow setting or changing catalog property polaris.config.enable-sub-catalog-rbac-for-federated-catalogs.If set to false, Polaris will disallow setting or changing the above catalog property. |
+| `polaris.features."ALLOW_DROPPING_NON_EMPTY_PASSTHROUGH_FACADE_CATALOG"` | `false` | If enabled, allow dropping a passthrough-facade catalog even if it contains namespaces or tables. passthrough-facade catalogs may contain leftover entities when syncing with source catalog.In the short term these entities will be ignored, in the long term there will be method/background job to clean them up. |
+| `polaris.features."ENABLE_FINE_GRAINED_UPDATE_TABLE_PRIVILEGES"` | `true` | When true, enables finer grained update table privileges which are passed to the authorizer for update table operations. |
+| `polaris.features."ALLOW_FEDERATED_CATALOGS_CREDENTIAL_VENDING"` | `true` | If set to true (default), allow credential vending for external catalogs. Note this requires ALLOW_EXTERNAL_CATALOG_CREDENTIAL_VENDING to be true first. |
+| `polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"` | `true` | "Override" realm features, here the skip credential subscoping indirection flag. |
+| `polaris.authentication.authenticator.type` | `default` | Define the Polaris authenticator type. |
+| `polaris.authentication.token-service.type` | `default` | Define the Polaris token service type. |
+| `polaris.authentication.token-broker.type` | `rsa-key-pair` | Define the Polaris token broker type. Also configure the location of the key files. For RSA: if the locations of the key files are not configured, an ephemeral key-pair will be created on each Polaris server instance startup, which breaks existing tokens after server restarts and is also incompatible with running multiple Polaris server instances. |
+| `polaris.authentication.token-broker.max-token-generation` | `PT1H` | Define the max token generation policy on the token broker. |
+| `polaris.authentication.token-broker.rsa-key-pair.private-key-file` | | Define the location of the RSA-256 private key file, if present the `public-key` file must be specified, too. |
+| `polaris.authentication.token-broker.rsa-key-pair.public-key-file` | | Define the location of the RSA-256 public key file, if present the `private-key` file must be specified, too. |
+| `polaris.authentication.token-broker.symmetric-key.secret` | `secret` | Define the secret of the symmetric key. |
+| `polaris.authentication.token-broker.symmetric-key.file` | `/tmp/symmetric.key` | Define the location of the symmetric key file. |
+| `polaris.storage.aws.access-key` | `accessKey` | Define the AWS S3 access key. If unset, the default credential provider chain will be used. |
+| `polaris.storage.aws.secret-key` | `secretKey` | Define the AWS S3 secret key. If unset, the default credential provider chain will be used. |
+| `polaris.storage.gcp.token` | `token` | Define the Google Cloud Storage token. If unset, the default credential provider chain will be used. |
+| `polaris.storage.gcp.lifespan` | `PT1H` | Define the Google Cloud Storage lifespan type. If unset, the default credential provider chain will be used. |
+| `polaris.log.request-id-header-name` | `Polaris-Request-Id` | Define the header name to match request ID in the log. |
Review Comment:
Yes. There are couple more missing ones which are also added now such as `polaris.event-listener.persistence-in-memory-buffer.buffer-time` and `polaris.event-listener.persistence-in-memory-buffer.max-buffer-size`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org