You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by yu...@apache.org on 2024/09/17 23:26:56 UTC

(polaris) branch main updated: Add the doc on how to configure Postgres as the backend database (#296)

This is an automated email from the ASF dual-hosted git repository.

yufei pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/polaris.git


The following commit(s) were added to refs/heads/main by this push:
     new f374ab44 Add the doc on how to configure Postgres as the backend database (#296)
f374ab44 is described below

commit f374ab4442fbe0ab405905cd64b0b083f393d0ce
Author: Aihua Xu <ai...@gmail.com>
AuthorDate: Tue Sep 17 16:26:52 2024 -0700

    Add the doc on how to configure Postgres as the backend database (#296)
---
 docs/configuring-polaris-for-production.md         |   4 +-
 docs/metastores.md                                 | 101 +++++++++++++++++++++
 .../PolarisEclipseLinkMetaStoreSessionImpl.java    |   2 +-
 polaris-service/build.gradle.kts                   |   1 -
 4 files changed, 104 insertions(+), 4 deletions(-)

diff --git a/docs/configuring-polaris-for-production.md b/docs/configuring-polaris-for-production.md
index 64dc3904..37f68281 100644
--- a/docs/configuring-polaris-for-production.md
+++ b/docs/configuring-polaris-for-production.md
@@ -20,7 +20,7 @@
 
 # Configuring Apache Polaris (Incubating) for Production
 
-The default `polaris-server.yml` configuration is intended for develoment and testing. When deploying Polaris in production, there are several best practices to keep in mind.
+The default `polaris-server.yml` configuration is intended for development and testing. When deploying Polaris in production, there are several best practices to keep in mind.
 
 ## Security
 
@@ -56,7 +56,7 @@ Be sure to secure your metastore backend since it will be storing credentials an
 
 ### Configuring EclipseLink
 
-To use [EclipseLink](https://eclipse.dev/eclipselink/) for metastore management, specify the configuration `metaStoreManager.conf-file` to point to an [EclipseLink `persistence.xml` file](https://eclipse.dev/eclipselink/documentation/2.5/solutions/testingjpa002.htm). This file, local to the Polaris service, will contain information on what database to use for metastore management and how to connect to it.
+To use EclipseLink for metastore management, specify the configuration `metaStoreManager.conf-file` to point to an EclipseLink `persistence.xml` file. This file, local to the Polaris service, contains details of the database used for metastore management and the connection settings. For more information, refer to [metastore documentation](./metastores.md) for details.
 
 ### Bootstrapping
 
diff --git a/docs/metastores.md b/docs/metastores.md
new file mode 100644
index 00000000..26e8c14d
--- /dev/null
+++ b/docs/metastores.md
@@ -0,0 +1,101 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+This page documents the general configurations to connect to production database through [EclipseLink](https://eclipse.dev/eclipselink/).
+
+## Polaris Server Configuration
+Configure the `metaStoreManager` section in server configuration `polaris-server.yml` as follows:
+```
+metaStoreManager:
+  type: eclipse-link
+  conf-file: META-INF/persistence.xml
+  persistence-unit: polaris
+```
+`conf-file` points to the EclipseLink configuration file and `persistence-unit` tells which unit from the configuration file to use for database connection.
+
+E.g., `polaris-dev` and `polaris` persistence units are configured in `persistence.xml` to connect to development database and production database respectively. Updating `persistence-unit` from `polaris-dev` to `polaris` switch from the development to the production.
+
+`conf-file` by default points to the embedded resource file `META-INF/persistence.xml` in `polaris-eclipselink` module.
+To specify an external configuration file,
+1) Place `persistence.xml` into a jar file: `jar cvf /tmp/conf.jar persistence.xml`.
+2) Use `conf-file: /tmp/conf.jar!/persistence.xml`.
+
+## EclipseLink Configuration - persistence.xml
+The configuration file `persistence.xml` is used to set up the database connection properties, which can differ depending on the type of database and its configuration.
+
+Check out [default persistence.xml](https://github.com/apache/polaris/blob/main/extension/persistence/eclipselink/src/main/resources/META-INF/persistence.xml) for the complete sample and the following shows the database connection properties against file-based H2 database. Polaris creates and connects to a separate database for each realm. Specifically, `{realm}` placeholder in `jakarta.persistence.jdbc.url` is substituted with the actual realm name, allowing the Polaris server to connec [...]
+
+> Note: some database systems such as Postgres don't create databases automatically. Database admins need to create them manually before running Polaris server.
+```xml
+<persistence-unit name="polaris" transaction-type="RESOURCE_LOCAL">
+    <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
+    <class>org.apache.polaris.core.persistence.models.ModelEntity</class>
+    <class>org.apache.polaris.core.persistence.models.ModelEntityActive</class>
+    <class>org.apache.polaris.core.persistence.models.ModelEntityChangeTracking</class>
+    <class>org.apache.polaris.core.persistence.models.ModelEntityDropped</class>
+    <class>org.apache.polaris.core.persistence.models.ModelGrantRecord</class>
+    <class>org.apache.polaris.core.persistence.models.ModelPrincipalSecrets</class>
+    <class>org.apache.polaris.core.persistence.models.ModelSequenceId</class>
+    <shared-cache-mode>NONE</shared-cache-mode>
+    <properties>
+      <property name="jakarta.persistence.jdbc.url"
+        value="jdbc:h2:file:tmp/polaris_test/filedb_{realm}"/>
+      <property name="jakarta.persistence.jdbc.user" value="sa"/>
+      <property name="jakarta.persistence.jdbc.password" value=""/>
+      <property name="jakarta.persistence.schema-generation.database.action" value="create"/>
+    </properties>
+</persistence-unit>
+```
+
+Build with H2 dependency and run Polaris service:
+```bash
+polaris> ./gradlew --no-daemon --info -PeclipseLink=true -PeclipseLinkDeps=com.h2database:h2:2.3.232 clean shadowJar
+polaris> java -jar  polaris-service/build/libs/polaris-service-*.jar server ./polaris-server.yml
+```
+
+### Postgres
+
+The following shows a sample configuration for Postgres database.
+
+```xml
+<persistence-unit name="polaris" transaction-type="RESOURCE_LOCAL">
+  <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
+  <class>org.apache.polaris.core.persistence.models.ModelEntity</class>
+  <class>org.apache.polaris.core.persistence.models.ModelEntityActive</class>
+  <class>org.apache.polaris.core.persistence.models.ModelEntityChangeTracking</class>
+  <class>org.apache.polaris.core.persistence.models.ModelEntityDropped</class>
+  <class>org.apache.polaris.core.persistence.models.ModelGrantRecord</class>
+  <class>org.apache.polaris.core.persistence.models.ModelPrincipalSecrets</class>
+  <class>org.apache.polaris.core.persistence.models.ModelSequenceId</class>
+  <shared-cache-mode>NONE</shared-cache-mode>
+  <properties>
+    <property name="jakarta.persistence.jdbc.url"
+              value="jdbc:postgresql://localhost:5432/{realm}"/>
+    <property name="jakarta.persistence.jdbc.user" value="postgres"/>
+    <property name="jakarta.persistence.jdbc.password" value="postgres"/>
+    <property name="jakarta.persistence.schema-generation.database.action" value="create"/>
+    <property name="eclipselink.persistence-context.flush-mode" value="auto"/>
+  </properties>
+</persistence-unit>
+```
+Build with Postgres dependency and run Polaris service:
+```bash
+polaris> ./gradlew --no-daemon --info -PeclipseLink=true -PeclipseLinkDeps=org.postgresql:postgresql:42.7.4 clean shadowJar
+polaris> java -jar  polaris-service/build/libs/polaris-service-*.jar server ./polaris-server.yml
+```
\ No newline at end of file
diff --git a/extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java b/extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java
index e7c813a6..75d4530d 100644
--- a/extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java
+++ b/extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java
@@ -147,7 +147,7 @@ public class PolarisEclipseLinkMetaStoreSessionImpl implements PolarisMetaStoreS
       // Currently eclipseLink can only support configuration as a resource inside a jar. To support
       // external configuration, persistence.xml needs be placed inside a jar and here is to add the
       // jar to the classpath.
-      // Supported configuration file: META-INFO/persistence.xml, /tmp/conf.jar!/persistence.xml
+      // Supported configuration file: META-INF/persistence.xml, /tmp/conf.jar!/persistence.xml
       int splitPosition = confFile.indexOf("!/");
       if (splitPosition != -1) {
         String jarPrefixPath = confFile.substring(0, splitPosition);
diff --git a/polaris-service/build.gradle.kts b/polaris-service/build.gradle.kts
index 31fd5707..4b0028aa 100644
--- a/polaris-service/build.gradle.kts
+++ b/polaris-service/build.gradle.kts
@@ -111,7 +111,6 @@ dependencies {
   testRuntimeOnly("org.junit.platform:junit-platform-launcher")
 
   testRuntimeOnly(project(":polaris-eclipselink"))
-  testRuntimeOnly(libs.h2)
 }
 
 if (project.properties.get("eclipseLink") == "true") {