You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "tsreaper (via GitHub)" <gi...@apache.org> on 2024/07/25 09:11:12 UTC

[PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

tsreaper opened a new pull request, #3816:
URL: https://github.com/apache/paimon/pull/3816

   ### Purpose
   
   In this PR we have supported reading partitions from fallback branch.
   
   You can set the table option `scan.fallback-branch` so that when a batch job reads from the current branch, if a partition does not exist, the reader will try to read this partition from the fallback branch. For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.
   
   What's the use case of this feature? Say you have created a Paimon table partitioned by date. You have a long-running streaming job which inserts records into Paimon, so that today's data can be queried in time. You also have a batch job which runs at every night to insert corrected records of yesterday into Paimon, so that the preciseness of the data can be promised.
   
   When you query from this Paimon table, you would like to first read from the results of batch job. But if a partition (for example, today's partition) does not exist in its result, then you would like to read from the results of streaming job. In this case, you can create a branch for streaming job, and set `scan.fallback-branch` to this streaming branch.
   
   Let's look at an example.
   
   ```sql
   -- create Paimon table
   CREATE TABLE T (
       dt STRING NOT NULL,
       name STRING NOT NULL,
       amount BIGINT
   ) PARTITIONED BY (dt);
   
   -- create a branch for streaming job
   -- you can even specify primary keys and bucket number, even if the original branch has no primary key
   -- in this example, we create a new branch `test`, uses `dt` and `name` as primary keys, has a bucket number of 2, and will copy the table options from the original branch
   CALL sys.create_branch('default.T', 'test', 'dt, name', 2, true);
   
   -- set fallback branch
   ALTER TABLE T SET (
       'scan.fallback-branch' = 'test'
   );
   
   -- set changelog producer for the streaming branch, in case a streaming job would like to read from it in the future
   ALTER TABLE `T$branch_test` SET (
       'changelog-producer' = 'lookup'
   );
   
   -- write records into the streaming branch
   INSERT INTO `T$branch_test` VALUES ('20240725', 'apple', 4), ('20240726', 'cherry', 3), ('20240726', 'pear', 6);
   
   -- write records into the default branch
   INSERT INTO T VALUES ('20240725', 'apple', 5), ('20240725', 'banana', 7);
   
   SELECT * FROM T;
   /*
   +------------------+------------------+--------+
   |               dt |             name | amount |
   +------------------+------------------+--------+
   |         20240725 |            apple |      5 |
   |         20240725 |           banana |      7 |
   |         20240726 |           cherry |      3 |
   |         20240726 |             pear |      6 |
   +------------------+------------------+--------+
   */
   ```
   
   ### Tests
   
   IT cases.
   
   ### API and Format
   
   No changes in format.
   
   ### Documentation
   
   Yes. Document is also updated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3816:
URL: https://github.com/apache/paimon/pull/3816#discussion_r1696189991


##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,86 @@ Run the following command:
 {{< /tab >}}
 
 {{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.
+
+What's the use case of this feature? Say you have created a Paimon table partitioned by date.
+You have a long-running streaming job which inserts records into Paimon, so that today's data can be queried in time.
+You also have a batch job which runs at every night to insert corrected records of yesterday into Paimon,
+so that the preciseness of the data can be promised.
+
+When you query from this Paimon table, you would like to first read from the results of batch job.
+But if a partition (for example, today's partition) does not exist in its result,
+then you would like to read from the results of streaming job.
+In this case, you can create a branch for streaming job, and set `scan.fallback-branch` to this streaming branch.
+
+Let's look at an example.
+
+{{< tabs "read-fallback-branch" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+-- create Paimon table
+CREATE TABLE T (
+    dt STRING NOT NULL,
+    name STRING NOT NULL,
+    amount BIGINT
+) PARTITIONED BY (dt);
+
+-- create a branch for streaming job
+-- you can even specify primary keys and bucket number, even if the original branch has no primary key
+-- in this example, we create a new branch `test`, uses `dt` and `name` as primary keys, has a bucket number of 2, and will copy the table options from the original branch
+CALL sys.create_branch('default.T', 'test', 'dt, name', 2, true);

Review Comment:
   With the subsequent development, different branches may have significant differences and require a lot of options modifications. Can we consider directly supporting the modification of all options of an empty table?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper commented on code in PR #3816:
URL: https://github.com/apache/paimon/pull/3816#discussion_r1694464606


##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,72 @@ Run the following command:
 {{< /tab >}}
 
 {{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.

Review Comment:
   We haven't decided yet. We need to talk with the users about it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #3816:
URL: https://github.com/apache/paimon/pull/3816


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3816:
URL: https://github.com/apache/paimon/pull/3816#discussion_r1697842340


##########
paimon-core/src/main/java/org/apache/paimon/table/FileStoreTableFactory.java:
##########
@@ -89,6 +90,24 @@ public static FileStoreTable create(
                                 fileIO, tablePath, tableSchema, catalogEnvironment)
                         : new PrimaryKeyFileStoreTable(
                                 fileIO, tablePath, tableSchema, catalogEnvironment);
-        return table.copy(dynamicOptions.toMap());
+        table = table.copy(dynamicOptions.toMap());
+
+        Options options = new Options(table.options());
+        String fallbackBranch = options.get(CoreOptions.SCAN_FALLBACK_BRANCH);
+        if (!StringUtils.isNullOrWhitespaceOnly(fallbackBranch)) {
+            Options branchOptions = new Options();
+            branchOptions.set(CoreOptions.BRANCH, fallbackBranch);
+            branchOptions.set(CoreOptions.SCAN_FALLBACK_BRANCH, "");

Review Comment:
   You can introduce private method to `createNoWrappedTable`.



##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,90 @@ Run the following command:
 {{< /tab >}}
 
 {{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.
+
+What's the use case of this feature? Say you have created a Paimon table partitioned by date.
+You have a long-running streaming job which inserts records into Paimon, so that today's data can be queried in time.
+You also have a batch job which runs at every night to insert corrected records of yesterday into Paimon,
+so that the preciseness of the data can be promised.
+
+When you query from this Paimon table, you would like to first read from the results of batch job.
+But if a partition (for example, today's partition) does not exist in its result,
+then you would like to read from the results of streaming job.
+In this case, you can create a branch for streaming job, and set `scan.fallback-branch` to this streaming branch.
+
+Let's look at an example.
+
+{{< tabs "read-fallback-branch" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+-- create Paimon table
+CREATE TABLE T (
+    dt STRING NOT NULL,
+    name STRING NOT NULL,
+    amount BIGINT
+) PARTITIONED BY (dt);
+
+-- create a branch for streaming job
+CALL sys.create_branch('default.T', 'test');
+
+-- set primary key and bucket number for the branch
+ALTER TABLE `T$branch_test` SET (
+    'primary-key' = 'dt,name',
+    'bucket' = '2'
+);
+
+-- set fallback branch
+ALTER TABLE T SET (
+    'scan.fallback-branch' = 'test'
+);
+
+-- set changelog producer for the streaming branch, in case a streaming job would like to read from it in the future
+ALTER TABLE `T$branch_test` SET (

Review Comment:
   put two alter together.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on code in PR #3816:
URL: https://github.com/apache/paimon/pull/3816#discussion_r1693678021


##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,72 @@ Run the following command:
 {{< /tab >}}
 
 {{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.
+
+What's the use case of this feature? Say you have created a Paimon table partitioned by date.
+You have a long-running streaming job which inserts records into Paimon, so that today's data can be queried in time.
+You also have a batch job which runs at every night to insert corrected records of yesterday into Paimon,
+so that the preciseness of the data can be promised.
+
+When you query from this Paimon table, you would like to first read from the results of batch job.
+But if a partition (for example, today's partition) does not exist in its result,
+then you would like to read from the results of streaming job.
+In this case, you can create a branch for streaming job, and set `scan.fallback-branch` to this streaming branch.
+
+Let's look at an example.
+
+{{< tabs "read-fallback-branch" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+-- create Paimon table
+CREATE TABLE T (
+    dt STRING NOT NULL,
+    name STRING NOT NULL,
+    amount BIGINT
+) PARTITIONED BY (dt);
+
+-- create a branch for streaming job
+-- you can even specify primary keys and bucket number, even if the original branch has no primary key
+-- in this example, we create a new branch `test`, uses `dt` and `name` as primary keys, has a bucket number of 2, and will copy the table options from the original branch
+CALL sys.create_branch('default.T', 'test', 'dt, name', 2, true);
+
+-- set fallback branch
+ALTER TABLE T SET (
+    'scan.fallback-branch' = 'test'
+);
+
+-- set changelog producer for the streaming branch, in case a streaming job would like to read from it in the future
+ALTER TABLE `T$branch_test` SET (
+    'changelog-producer' = 'lookup'
+);
+
+-- write records into the streaming branch
+INSERT INTO `T$branch_test` VALUES ('20240725', 'apple', 4), ('20240725', 'peach', 10), ('20240726', 'cherry', 3), ('20240726', 'pear', 6);
+
+-- write records into the default branch
+INSERT INTO T VALUES ('20240725', 'apple', 5), ('20240725', 'banana', 7);
+
+SELECT * FROM T;
+/*
++------------------+------------------+--------+
+|               dt |             name | amount |
++------------------+------------------+--------+
+|         20240725 |            apple |      5 |

Review Comment:
   better first run the query without setting `scan.fallback-branch` and return the two rows of 20240725, then set `'scan.fallback-branch' = 'test'` and run the query again to return the result of 4 rows with the data from the fallback branch. WDYT?



##########
paimon-core/src/main/java/org/apache/paimon/table/FileStoreTableFactory.java:
##########
@@ -89,6 +91,27 @@ public static FileStoreTable create(
                                 fileIO, tablePath, tableSchema, catalogEnvironment)
                         : new PrimaryKeyFileStoreTable(
                                 fileIO, tablePath, tableSchema, catalogEnvironment);
-        return table.copy(dynamicOptions.toMap());
+        table = table.copy(dynamicOptions.toMap());
+
+        Options options = new Options(table.options());
+        String fallbackBranch = options.get(CoreOptions.SCAN_FALLBACK_BRANCH);
+        if (!StringUtils.isNullOrWhitespaceOnly(fallbackBranch)) {
+            Options branchOptions = new Options();
+            branchOptions.set(CoreOptions.BRANCH, fallbackBranch);
+            branchOptions.set(CoreOptions.SCAN_FALLBACK_BRANCH, "");
+            FileStoreTable fallbackTable =
+                    FileStoreTableFactory.create(
+                            fileIO,
+                            tablePath,
+                            new SchemaManager(fileIO, tablePath, fallbackBranch).latest().get(),
+                            branchOptions,
+                            catalogEnvironment);
+
+            Preconditions.checkArgument(!(table instanceof FallbackReadFileStoreTable));

Review Comment:
   Does it make sense to move the two Preconditions checks to the constructor of `FallbackReadFileStoreTable` ?



##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,72 @@ Run the following command:
 {{< /tab >}}
 
 {{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.
+
+What's the use case of this feature? Say you have created a Paimon table partitioned by date.
+You have a long-running streaming job which inserts records into Paimon, so that today's data can be queried in time.
+You also have a batch job which runs at every night to insert corrected records of yesterday into Paimon,
+so that the preciseness of the data can be promised.
+
+When you query from this Paimon table, you would like to first read from the results of batch job.
+But if a partition (for example, today's partition) does not exist in its result,
+then you would like to read from the results of streaming job.
+In this case, you can create a branch for streaming job, and set `scan.fallback-branch` to this streaming branch.
+
+Let's look at an example.
+
+{{< tabs "read-fallback-branch" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+-- create Paimon table
+CREATE TABLE T (
+    dt STRING NOT NULL,
+    name STRING NOT NULL,
+    amount BIGINT
+) PARTITIONED BY (dt);
+
+-- create a branch for streaming job
+-- you can even specify primary keys and bucket number, even if the original branch has no primary key
+-- in this example, we create a new branch `test`, uses `dt` and `name` as primary keys, has a bucket number of 2, and will copy the table options from the original branch
+CALL sys.create_branch('default.T', 'test', 'dt, name', 2, true);
+
+-- set fallback branch
+ALTER TABLE T SET (
+    'scan.fallback-branch' = 'test'

Review Comment:
   How to reset it? 'scan.fallback-branch' = null ?



##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,72 @@ Run the following command:
 {{< /tab >}}
 
 {{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.

Review Comment:
   What would be the expected behaviour if the feature for the streaming read had correctly implemented?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

Posted by "JingGe (via GitHub)" <gi...@apache.org>.
JingGe commented on code in PR #3816:
URL: https://github.com/apache/paimon/pull/3816#discussion_r1693693991


##########
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/BranchSqlITCase.java:
##########
@@ -75,12 +76,90 @@ public void testAlterTable() throws Exception {
                         "+I[2, 40, wolf, 400]");
     }
 
-    private List<String> collectResult(TableEnvironment tEnv, String sql) throws Exception {
+    @Test
+    public void testFallbackBranchBatchRead() throws Exception {
+        TableEnvironment tEnv = tableEnvironmentBuilder().batchMode().build();
+        tEnv.executeSql(
+                "CREATE CATALOG mycat WITH ( 'type' = 'paimon', 'warehouse' = '" + tempDir + "' )");
+        tEnv.executeSql("USE CATALOG mycat");
+        tEnv.executeSql(
+                "CREATE TABLE t ( pt INT NOT NULL, k INT NOT NULL, v STRING ) PARTITIONED BY (pt) WITH ( 'bucket' = '-1' )");
+        tEnv.executeSql("CALL sys.create_branch('default.t', 'pk', 'pt, k', 2, true)").await();
+        tEnv.executeSql("ALTER TABLE t SET ( 'scan.fallback-branch' = 'pk' )");
+
+        tEnv.executeSql("INSERT INTO t VALUES (1, 10, 'apple'), (1, 20, 'banana')").await();
+        tEnv.executeSql("INSERT INTO `t$branch_pk` VALUES (1, 10, 'cat'), (1, 20, 'dog')").await();
+        assertThat(collectResult(tEnv, "SELECT v, k FROM t"))
+                .containsExactlyInAnyOrder("+I[apple, 10]", "+I[banana, 20]");
+
+        tEnv.executeSql("INSERT INTO `t$branch_pk` VALUES (2, 10, 'tiger'), (2, 20, 'wolf')")
+                .await();
+        assertThat(collectResult(tEnv, "SELECT v, k FROM t"))
+                .containsExactlyInAnyOrder(
+                        "+I[apple, 10]", "+I[banana, 20]", "+I[tiger, 10]", "+I[wolf, 20]");
+        assertThat(collectResult(tEnv, "SELECT v, k FROM t WHERE pt = 1"))
+                .containsExactlyInAnyOrder("+I[apple, 10]", "+I[banana, 20]");
+        assertThat(collectResult(tEnv, "SELECT v, k FROM t WHERE pt = 2"))
+                .containsExactlyInAnyOrder("+I[tiger, 10]", "+I[wolf, 20]");
+
+        tEnv.executeSql("INSERT INTO `t$branch_pk` VALUES (2, 10, 'lion')").await();
+        assertThat(collectResult(tEnv, "SELECT v, k FROM t"))
+                .containsExactlyInAnyOrder(
+                        "+I[apple, 10]", "+I[banana, 20]", "+I[lion, 10]", "+I[wolf, 20]");

Review Comment:
   Nit: additional test when `scan.fallback-branch` is not set, i.e. only return data from main branch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Support reading partition from fallback branch when not found in current branch [paimon]

Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper commented on code in PR #3816:
URL: https://github.com/apache/paimon/pull/3816#discussion_r1694465925


##########
docs/content/maintenance/manage-branches.md:
##########
@@ -159,3 +159,72 @@ Run the following command:
 {{< /tab >}}
 
 {{< /tabs >}}
+
+### Batch Reading from Fallback Branch
+
+You can set the table option `scan.fallback-branch`
+so that when a batch job reads from the current branch, if a partition does not exist,
+the reader will try to read this partition from the fallback branch.
+For streaming read jobs, this feature is currently not supported, and will only produce results from the current branch.
+
+What's the use case of this feature? Say you have created a Paimon table partitioned by date.
+You have a long-running streaming job which inserts records into Paimon, so that today's data can be queried in time.
+You also have a batch job which runs at every night to insert corrected records of yesterday into Paimon,
+so that the preciseness of the data can be promised.
+
+When you query from this Paimon table, you would like to first read from the results of batch job.
+But if a partition (for example, today's partition) does not exist in its result,
+then you would like to read from the results of streaming job.
+In this case, you can create a branch for streaming job, and set `scan.fallback-branch` to this streaming branch.
+
+Let's look at an example.
+
+{{< tabs "read-fallback-branch" >}}
+
+{{< tab "Flink" >}}
+
+```sql
+-- create Paimon table
+CREATE TABLE T (
+    dt STRING NOT NULL,
+    name STRING NOT NULL,
+    amount BIGINT
+) PARTITIONED BY (dt);
+
+-- create a branch for streaming job
+-- you can even specify primary keys and bucket number, even if the original branch has no primary key
+-- in this example, we create a new branch `test`, uses `dt` and `name` as primary keys, has a bucket number of 2, and will copy the table options from the original branch
+CALL sys.create_branch('default.T', 'test', 'dt, name', 2, true);
+
+-- set fallback branch
+ALTER TABLE T SET (
+    'scan.fallback-branch' = 'test'

Review Comment:
   `ALTER TABLE T RESET ('scan.fallback-branch')`, see [Paimon docs](https://paimon.apache.org/docs/master/flink/sql-alter/#removing-table-properties).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org