You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "MehulBatra (via GitHub)" <gi...@apache.org> on 2026/02/03 16:07:45 UTC
[PR] [docs] binlog virtual table [fluss]
MehulBatra opened a new pull request, #2565:
URL: https://github.com/apache/fluss/pull/2565
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- What is the purpose of the change -->
### Brief change log
Gives info to user about what is binlog table and how it can be used & accessed.
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2762776906
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -55,23 +49,21 @@ For Primary Key Tables, the following change types are supported:
| Change Type | Description |
|-------------|-------------|
-| `+I` | **Insert** - A new row was inserted |
-| `-U` | **Update Before** - The previous value of an updated row (retraction) |
-| `+U` | **Update After** - The new value of an updated row |
-| `-D` | **Delete** - A row was deleted |
+| `insert` | A new row was inserted |
+| `update_before` | The previous value of an updated row (retraction) |
+| `update_after` | The new value of an updated row |
+| `delete` | A row was deleted |
#### Log Tables
For Log Tables (append-only), only one change type is used:
| Change Type | Description |
|-------------|-------------|
-| `+A` | **Append** - A new row was appended to the log |
+| `insert` | A new row was inserted into the log |
Review Comment:
ditto
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
++----+---------------+-------------+---------------------+----------+---------------+---------+
```
:::note
The `op` column is Flink's row kind indicator. For changelog virtual tables, all rows are emitted as `+I` (insert) to the downstream, while the actual change type is captured in the `_change_type` column.
:::
-#### Log Table Changelog
-Consider a Log Table storing click events:
+## Binlog Table
+
+The `$binlog` virtual table provides access to change data where each record contains both the before and after images of the row. This is useful for:
+
+:::note
+The `$binlog` virtual table is only available for **Primary Key Tables**.
+:::
+
+### Accessing the Binlog
+
+To access the binlog of a Primary Key Table, append `$binlog` to the table name:
+
+```sql title="Flink SQL"
+SELECT * FROM my_pk_table$binlog;
+```
+
+### Schema
+
+The binlog virtual table includes three metadata columns followed by nested `before` and `after` row structures:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `_change_type` | STRING NOT NULL | The type of change operation: `insert`, `update`, or `delete` |
+| `_log_offset` | BIGINT NOT NULL | The offset position in the log |
+| `_commit_timestamp` | TIMESTAMP_LTZ(3) NOT NULL | The timestamp when the change was committed |
+| `before` | ROW<...> | The row values before the change (NULL for inserts) |
+| `after` | ROW<...> | The row values after the change (NULL for deletes) |
+
+The `before` and `after` columns are nested ROW types containing all columns from the base table.
+
+### Change Types
+
+| Change Type | Description | `before` | `after` |
+|-------------|-------------|----------|---------|
+| `insert` | A new row was inserted | NULL | Contains new row values |
+| `update` | A row was updated | Contains old row values | Contains new row values |
+| `delete` | A row was deleted | Contains deleted row values | NULL |
+
+### Examples
```sql title="Flink SQL"
--- Create a log table (no primary key)
-CREATE TABLE click_events (
- event_id INT,
- user_id INT,
- event_type STRING
+-- Create a primary key table
+CREATE TABLE users (
+ user_id INT NOT NULL,
+ name STRING,
+ email STRING,
+ PRIMARY KEY (user_id) NOT ENFORCED
) WITH ('bucket.num' = '1');
--- Append events
-INSERT INTO click_events VALUES (1, 101, 'click'), (2, 102, 'view');
+-- Insert, update, then delete a record
+INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
+INSERT INTO users VALUES (1, 'Alice Smith', 'alice.smith@example.com');
+DELETE FROM users WHERE user_id = 1;
--- Query the changelog
-SELECT * FROM click_events$changelog;
+-- Query the binlog
+SELECT * FROM users$binlog;
```
Output:
```
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| op | _change_type | _log_offset | _commit_timestamp | event_id | user_id | event_type |
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| +I | +A | 0 | 2024-01-15 11:00:00 | 1 | 101 | click |
-| +I | +A | 1 | 2024-01-15 11:00:00 | 2 | 102 | view |
-+----+--------------+-------------+---------------------+----------+---------+------------+
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| op | _change_type | _log_offset | _commit_timestamp | before | after |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | NULL | (1, Alice, alice@example.com) |
+| +I | update | 2 | 2024-01-15 10:35:00 | (1, Alice, alice@example.com) | (1, Alice Smith, alice.smith@example.com) |
+| +I | delete | 3 | 2024-01-15 10:40:00 | (1, Alice Smith, alice.smith@example.com) | NULL |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+```
+
+#### Accessing Nested Fields
+
+You can access individual fields from the `before` and `after` structures:
+
+```sql title="Flink SQL"
+SELECT
+ _change_type,
+ _commit_timestamp,
+ `before`.name AS old_name,
+ `after`.name AS new_name
+FROM users$binlog
+WHERE _change_type = 'update';
```
### Startup Modes
-The changelog virtual table supports different startup modes to control where reading begins:
+
+| Mode | Description |
+|------|-------------|
+| `earliest` | Start reading from the beginning of the log |
+| `latest` | Start reading from the current end of the log (only new changes) |
+| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
+
+
+The virtual table supports different startup modes to control where reading begins:
```sql title="Flink SQL"
-- Read from the beginning (default)
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'earliest') */;
-- Read only new changes from now
-SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
+SELECT * FROM orders$binlog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
-- Read from a specific timestamp
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'timestamp', 'scan.startup.timestamp' = '1705312200000') */;
```
-| Mode | Description |
-|------|-------------|
-| `earliest` | Start reading from the beginning of the log |
-| `latest` | Start reading from the current end of the log (only new changes) |
-| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
### Limitations
-- Projection & partition & predicate pushdowns are not supported yet. This will be addressed in future releases.
+- Projection & partition & predicate pushdowns are not supported yet.
Review Comment:
Projection, partition pruning, and predicate pushdown are not supported yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2764255698
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
++----+---------------+-------------+---------------------+----------+---------------+---------+
```
:::note
The `op` column is Flink's row kind indicator. For changelog virtual tables, all rows are emitted as `+I` (insert) to the downstream, while the actual change type is captured in the `_change_type` column.
:::
-#### Log Table Changelog
-Consider a Log Table storing click events:
+## Binlog Table
+
+The `$binlog` virtual table provides access to change data where each record contains both the before and after images of the row. This is useful for:
+
+:::note
+The `$binlog` virtual table is only available for **Primary Key Tables**.
+:::
+
+### Accessing the Binlog
+
+To access the binlog of a Primary Key Table, append `$binlog` to the table name:
+
+```sql title="Flink SQL"
+SELECT * FROM my_pk_table$binlog;
+```
+
+### Schema
+
+The binlog virtual table includes three metadata columns followed by nested `before` and `after` row structures:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `_change_type` | STRING NOT NULL | The type of change operation: `insert`, `update`, or `delete` |
+| `_log_offset` | BIGINT NOT NULL | The offset position in the log |
+| `_commit_timestamp` | TIMESTAMP_LTZ(3) NOT NULL | The timestamp when the change was committed |
+| `before` | ROW<...> | The row values before the change (NULL for inserts) |
+| `after` | ROW<...> | The row values after the change (NULL for deletes) |
+
+The `before` and `after` columns are nested ROW types containing all columns from the base table.
+
+### Change Types
+
+| Change Type | Description | `before` | `after` |
+|-------------|-------------|----------|---------|
+| `insert` | A new row was inserted | NULL | Contains new row values |
+| `update` | A row was updated | Contains old row values | Contains new row values |
+| `delete` | A row was deleted | Contains deleted row values | NULL |
+
+### Examples
```sql title="Flink SQL"
--- Create a log table (no primary key)
-CREATE TABLE click_events (
- event_id INT,
- user_id INT,
- event_type STRING
+-- Create a primary key table
+CREATE TABLE users (
+ user_id INT NOT NULL,
+ name STRING,
+ email STRING,
+ PRIMARY KEY (user_id) NOT ENFORCED
) WITH ('bucket.num' = '1');
--- Append events
-INSERT INTO click_events VALUES (1, 101, 'click'), (2, 102, 'view');
+-- Insert, update, then delete a record
+INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
+INSERT INTO users VALUES (1, 'Alice Smith', 'alice.smith@example.com');
+DELETE FROM users WHERE user_id = 1;
--- Query the changelog
-SELECT * FROM click_events$changelog;
+-- Query the binlog
+SELECT * FROM users$binlog;
```
Output:
```
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| op | _change_type | _log_offset | _commit_timestamp | event_id | user_id | event_type |
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| +I | +A | 0 | 2024-01-15 11:00:00 | 1 | 101 | click |
-| +I | +A | 1 | 2024-01-15 11:00:00 | 2 | 102 | view |
-+----+--------------+-------------+---------------------+----------+---------+------------+
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| op | _change_type | _log_offset | _commit_timestamp | before | after |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | NULL | (1, Alice, alice@example.com) |
+| +I | update | 2 | 2024-01-15 10:35:00 | (1, Alice, alice@example.com) | (1, Alice Smith, alice.smith@example.com) |
+| +I | delete | 3 | 2024-01-15 10:40:00 | (1, Alice Smith, alice.smith@example.com) | NULL |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+```
+
+#### Accessing Nested Fields
+
+You can access individual fields from the `before` and `after` structures:
+
+```sql title="Flink SQL"
+SELECT
+ _change_type,
+ _commit_timestamp,
+ `before`.name AS old_name,
+ `after`.name AS new_name
+FROM users$binlog
+WHERE _change_type = 'update';
```
### Startup Modes
-The changelog virtual table supports different startup modes to control where reading begins:
+
+| Mode | Description |
+|------|-------------|
+| `earliest` | Start reading from the beginning of the log |
+| `latest` | Start reading from the current end of the log (only new changes) |
+| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
+
+
+The virtual table supports different startup modes to control where reading begins:
```sql title="Flink SQL"
-- Read from the beginning (default)
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'earliest') */;
-- Read only new changes from now
-SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
+SELECT * FROM orders$binlog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
-- Read from a specific timestamp
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'timestamp', 'scan.startup.timestamp' = '1705312200000') */;
```
-| Mode | Description |
-|------|-------------|
-| `earliest` | Start reading from the beginning of the log |
-| `latest` | Start reading from the current end of the log (only new changes) |
-| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
### Limitations
-- Projection & partition & predicate pushdowns are not supported yet. This will be addressed in future releases.
+- Projection & partition & predicate pushdowns are not supported yet.
Review Comment:
Yes, they aren't. We will work on this in 0.10, currently you can select specific columns via flink sql level but that will not reflect in storage level, hence no projection pushdown aswell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong merged PR #2565:
URL: https://github.com/apache/fluss/pull/2565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2764259214
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
Review Comment:
Agreed, removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2762749250
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -55,23 +49,21 @@ For Primary Key Tables, the following change types are supported:
| Change Type | Description |
|-------------|-------------|
-| `+I` | **Insert** - A new row was inserted |
-| `-U` | **Update Before** - The previous value of an updated row (retraction) |
-| `+U` | **Update After** - The new value of an updated row |
-| `-D` | **Delete** - A row was deleted |
+| `insert` | A new row was inserted |
Review Comment:
maybe use `op` and `change column` columns? and make them uppercase.
This also helps with consistency throughout the doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2762787853
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
++----+---------------+-------------+---------------------+----------+---------------+---------+
```
:::note
The `op` column is Flink's row kind indicator. For changelog virtual tables, all rows are emitted as `+I` (insert) to the downstream, while the actual change type is captured in the `_change_type` column.
:::
-#### Log Table Changelog
-Consider a Log Table storing click events:
+## Binlog Table
+
+The `$binlog` virtual table provides access to change data where each record contains both the before and after images of the row. This is useful for:
+
+:::note
+The `$binlog` virtual table is only available for **Primary Key Tables**.
+:::
+
+### Accessing the Binlog
+
+To access the binlog of a Primary Key Table, append `$binlog` to the table name:
+
+```sql title="Flink SQL"
+SELECT * FROM my_pk_table$binlog;
+```
+
+### Schema
+
+The binlog virtual table includes three metadata columns followed by nested `before` and `after` row structures:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `_change_type` | STRING NOT NULL | The type of change operation: `insert`, `update`, or `delete` |
+| `_log_offset` | BIGINT NOT NULL | The offset position in the log |
+| `_commit_timestamp` | TIMESTAMP_LTZ(3) NOT NULL | The timestamp when the change was committed |
+| `before` | ROW<...> | The row values before the change (NULL for inserts) |
+| `after` | ROW<...> | The row values after the change (NULL for deletes) |
+
+The `before` and `after` columns are nested ROW types containing all columns from the base table.
+
+### Change Types
+
+| Change Type | Description | `before` | `after` |
+|-------------|-------------|----------|---------|
+| `insert` | A new row was inserted | NULL | Contains new row values |
+| `update` | A row was updated | Contains old row values | Contains new row values |
+| `delete` | A row was deleted | Contains deleted row values | NULL |
+
+### Examples
```sql title="Flink SQL"
--- Create a log table (no primary key)
-CREATE TABLE click_events (
- event_id INT,
- user_id INT,
- event_type STRING
+-- Create a primary key table
+CREATE TABLE users (
+ user_id INT NOT NULL,
+ name STRING,
+ email STRING,
+ PRIMARY KEY (user_id) NOT ENFORCED
) WITH ('bucket.num' = '1');
--- Append events
-INSERT INTO click_events VALUES (1, 101, 'click'), (2, 102, 'view');
+-- Insert, update, then delete a record
+INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
+INSERT INTO users VALUES (1, 'Alice Smith', 'alice.smith@example.com');
+DELETE FROM users WHERE user_id = 1;
--- Query the changelog
-SELECT * FROM click_events$changelog;
+-- Query the binlog
+SELECT * FROM users$binlog;
```
Output:
```
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| op | _change_type | _log_offset | _commit_timestamp | event_id | user_id | event_type |
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| +I | +A | 0 | 2024-01-15 11:00:00 | 1 | 101 | click |
-| +I | +A | 1 | 2024-01-15 11:00:00 | 2 | 102 | view |
-+----+--------------+-------------+---------------------+----------+---------+------------+
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| op | _change_type | _log_offset | _commit_timestamp | before | after |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | NULL | (1, Alice, alice@example.com) |
+| +I | update | 2 | 2024-01-15 10:35:00 | (1, Alice, alice@example.com) | (1, Alice Smith, alice.smith@example.com) |
+| +I | delete | 3 | 2024-01-15 10:40:00 | (1, Alice Smith, alice.smith@example.com) | NULL |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+```
+
+#### Accessing Nested Fields
+
+You can access individual fields from the `before` and `after` structures:
+
+```sql title="Flink SQL"
+SELECT
+ _change_type,
+ _commit_timestamp,
+ `before`.name AS old_name,
+ `after`.name AS new_name
+FROM users$binlog
+WHERE _change_type = 'update';
```
### Startup Modes
-The changelog virtual table supports different startup modes to control where reading begins:
+
+| Mode | Description |
+|------|-------------|
+| `earliest` | Start reading from the beginning of the log |
+| `latest` | Start reading from the current end of the log (only new changes) |
+| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
+
+
+The virtual table supports different startup modes to control where reading begins:
```sql title="Flink SQL"
-- Read from the beginning (default)
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'earliest') */;
-- Read only new changes from now
-SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
+SELECT * FROM orders$binlog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
-- Read from a specific timestamp
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'timestamp', 'scan.startup.timestamp' = '1705312200000') */;
```
-| Mode | Description |
-|------|-------------|
-| `earliest` | Start reading from the beginning of the log |
-| `latest` | Start reading from the current end of the log (only new changes) |
-| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
### Limitations
-- Projection & partition & predicate pushdowns are not supported yet. This will be addressed in future releases.
+- Projection & partition & predicate pushdowns are not supported yet.
Review Comment:
Projections, partition pruning, and predicate pushdowns are not supported yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2762749250
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -55,23 +49,21 @@ For Primary Key Tables, the following change types are supported:
| Change Type | Description |
|-------------|-------------|
-| `+I` | **Insert** - A new row was inserted |
-| `-U` | **Update Before** - The previous value of an updated row (retraction) |
-| `+U` | **Update After** - The new value of an updated row |
-| `-D` | **Delete** - A row was deleted |
+| `insert` | A new row was inserted |
Review Comment:
maybe use `op` and `change column` columns? This also helps with consistency throughout the doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2763743253
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
++----+---------------+-------------+---------------------+----------+---------------+---------+
```
:::note
The `op` column is Flink's row kind indicator. For changelog virtual tables, all rows are emitted as `+I` (insert) to the downstream, while the actual change type is captured in the `_change_type` column.
:::
-#### Log Table Changelog
-Consider a Log Table storing click events:
+## Binlog Table
+
+The `$binlog` virtual table provides access to change data where each record contains both the before and after images of the row. This is useful for:
+
+:::note
+The `$binlog` virtual table is only available for **Primary Key Tables**.
+:::
+
+### Accessing the Binlog
+
+To access the binlog of a Primary Key Table, append `$binlog` to the table name:
+
+```sql title="Flink SQL"
+SELECT * FROM my_pk_table$binlog;
+```
+
+### Schema
+
+The binlog virtual table includes three metadata columns followed by nested `before` and `after` row structures:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `_change_type` | STRING NOT NULL | The type of change operation: `insert`, `update`, or `delete` |
+| `_log_offset` | BIGINT NOT NULL | The offset position in the log |
+| `_commit_timestamp` | TIMESTAMP_LTZ(3) NOT NULL | The timestamp when the change was committed |
+| `before` | ROW<...> | The row values before the change (NULL for inserts) |
+| `after` | ROW<...> | The row values after the change (NULL for deletes) |
+
+The `before` and `after` columns are nested ROW types containing all columns from the base table.
+
+### Change Types
+
+| Change Type | Description | `before` | `after` |
+|-------------|-------------|----------|---------|
+| `insert` | A new row was inserted | NULL | Contains new row values |
+| `update` | A row was updated | Contains old row values | Contains new row values |
+| `delete` | A row was deleted | Contains deleted row values | NULL |
+
+### Examples
```sql title="Flink SQL"
--- Create a log table (no primary key)
-CREATE TABLE click_events (
- event_id INT,
- user_id INT,
- event_type STRING
+-- Create a primary key table
+CREATE TABLE users (
+ user_id INT NOT NULL,
+ name STRING,
+ email STRING,
+ PRIMARY KEY (user_id) NOT ENFORCED
) WITH ('bucket.num' = '1');
--- Append events
-INSERT INTO click_events VALUES (1, 101, 'click'), (2, 102, 'view');
+-- Insert, update, then delete a record
+INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
+INSERT INTO users VALUES (1, 'Alice Smith', 'alice.smith@example.com');
+DELETE FROM users WHERE user_id = 1;
--- Query the changelog
-SELECT * FROM click_events$changelog;
+-- Query the binlog
+SELECT * FROM users$binlog;
```
Output:
```
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| op | _change_type | _log_offset | _commit_timestamp | event_id | user_id | event_type |
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| +I | +A | 0 | 2024-01-15 11:00:00 | 1 | 101 | click |
-| +I | +A | 1 | 2024-01-15 11:00:00 | 2 | 102 | view |
-+----+--------------+-------------+---------------------+----------+---------+------------+
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| op | _change_type | _log_offset | _commit_timestamp | before | after |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | NULL | (1, Alice, alice@example.com) |
+| +I | update | 2 | 2024-01-15 10:35:00 | (1, Alice, alice@example.com) | (1, Alice Smith, alice.smith@example.com) |
+| +I | delete | 3 | 2024-01-15 10:40:00 | (1, Alice Smith, alice.smith@example.com) | NULL |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+```
+
+#### Accessing Nested Fields
+
+You can access individual fields from the `before` and `after` structures:
+
+```sql title="Flink SQL"
+SELECT
+ _change_type,
+ _commit_timestamp,
+ `before`.name AS old_name,
+ `after`.name AS new_name
+FROM users$binlog
+WHERE _change_type = 'update';
```
### Startup Modes
Review Comment:
We should add "Startup Modes` section for `$changelog` table as well. Currently, this is under the section of `$binlog` table.
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
++----+---------------+-------------+---------------------+----------+---------------+---------+
```
:::note
The `op` column is Flink's row kind indicator. For changelog virtual tables, all rows are emitted as `+I` (insert) to the downstream, while the actual change type is captured in the `_change_type` column.
:::
-#### Log Table Changelog
-Consider a Log Table storing click events:
+## Binlog Table
+
+The `$binlog` virtual table provides access to change data where each record contains both the before and after images of the row. This is useful for:
+
+:::note
+The `$binlog` virtual table is only available for **Primary Key Tables**.
+:::
+
+### Accessing the Binlog
+
+To access the binlog of a Primary Key Table, append `$binlog` to the table name:
+
+```sql title="Flink SQL"
+SELECT * FROM my_pk_table$binlog;
+```
+
+### Schema
+
+The binlog virtual table includes three metadata columns followed by nested `before` and `after` row structures:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `_change_type` | STRING NOT NULL | The type of change operation: `insert`, `update`, or `delete` |
+| `_log_offset` | BIGINT NOT NULL | The offset position in the log |
+| `_commit_timestamp` | TIMESTAMP_LTZ(3) NOT NULL | The timestamp when the change was committed |
+| `before` | ROW<...> | The row values before the change (NULL for inserts) |
+| `after` | ROW<...> | The row values after the change (NULL for deletes) |
+
+The `before` and `after` columns are nested ROW types containing all columns from the base table.
+
+### Change Types
+
+| Change Type | Description | `before` | `after` |
+|-------------|-------------|----------|---------|
+| `insert` | A new row was inserted | NULL | Contains new row values |
+| `update` | A row was updated | Contains old row values | Contains new row values |
+| `delete` | A row was deleted | Contains deleted row values | NULL |
+
+### Examples
```sql title="Flink SQL"
--- Create a log table (no primary key)
-CREATE TABLE click_events (
- event_id INT,
- user_id INT,
- event_type STRING
+-- Create a primary key table
+CREATE TABLE users (
+ user_id INT NOT NULL,
+ name STRING,
+ email STRING,
+ PRIMARY KEY (user_id) NOT ENFORCED
) WITH ('bucket.num' = '1');
--- Append events
-INSERT INTO click_events VALUES (1, 101, 'click'), (2, 102, 'view');
+-- Insert, update, then delete a record
+INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
+INSERT INTO users VALUES (1, 'Alice Smith', 'alice.smith@example.com');
+DELETE FROM users WHERE user_id = 1;
--- Query the changelog
-SELECT * FROM click_events$changelog;
+-- Query the binlog
+SELECT * FROM users$binlog;
```
Output:
```
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| op | _change_type | _log_offset | _commit_timestamp | event_id | user_id | event_type |
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| +I | +A | 0 | 2024-01-15 11:00:00 | 1 | 101 | click |
-| +I | +A | 1 | 2024-01-15 11:00:00 | 2 | 102 | view |
-+----+--------------+-------------+---------------------+----------+---------+------------+
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| op | _change_type | _log_offset | _commit_timestamp | before | after |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | NULL | (1, Alice, alice@example.com) |
+| +I | update | 2 | 2024-01-15 10:35:00 | (1, Alice, alice@example.com) | (1, Alice Smith, alice.smith@example.com) |
+| +I | delete | 3 | 2024-01-15 10:40:00 | (1, Alice Smith, alice.smith@example.com) | NULL |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+```
+
+#### Accessing Nested Fields
+
+You can access individual fields from the `before` and `after` structures:
+
+```sql title="Flink SQL"
+SELECT
+ _change_type,
+ _commit_timestamp,
+ `before`.name AS old_name,
+ `after`.name AS new_name
+FROM users$binlog
+WHERE _change_type = 'update';
```
### Startup Modes
-The changelog virtual table supports different startup modes to control where reading begins:
+
+| Mode | Description |
+|------|-------------|
+| `earliest` | Start reading from the beginning of the log |
+| `latest` | Start reading from the current end of the log (only new changes) |
+| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
+
+
+The virtual table supports different startup modes to control where reading begins:
```sql title="Flink SQL"
-- Read from the beginning (default)
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'earliest') */;
-- Read only new changes from now
-SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
+SELECT * FROM orders$binlog /*+ OPTIONS('scan.startup.mode' = 'latest') */;
-- Read from a specific timestamp
SELECT * FROM orders$changelog /*+ OPTIONS('scan.startup.mode' = 'timestamp', 'scan.startup.timestamp' = '1705312200000') */;
```
-| Mode | Description |
-|------|-------------|
-| `earliest` | Start reading from the beginning of the log |
-| `latest` | Start reading from the current end of the log (only new changes) |
-| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
### Limitations
-- Projection & partition & predicate pushdowns are not supported yet. This will be addressed in future releases.
+- Projection & partition & predicate pushdowns are not supported yet.
Review Comment:
We should add the limitation for `$changelog` table as well. Currently, this is under the section of `$binlog` table.
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
Review Comment:
Can we remove the `op` column from the output? This is very confusing we carry an additional column `op` in the schema of `$binlog` or `$changelog` table.
Flink SQL CLI supports options to omit the `op` column
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#sql-client-result-modes
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -99,70 +91,128 @@ SELECT * FROM orders$changelog;
Output:
```
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
-| +I | +I | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
-| +I | -U | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
-| +I | +U | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
-| +I | -D | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
-+----+--------------+-------------+---------------------+----------+---------------+---------+
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| op | _change_type | _log_offset | _commit_timestamp | order_id | customer_name | amount |
++----+---------------+-------------+---------------------+----------+---------------+---------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | 1 | Rhea | 100.00 |
+| +I | update_before | 1 | 2024-01-15 10:35:00 | 1 | Rhea | 100.00 |
+| +I | update_after | 2 | 2024-01-15 10:35:00 | 1 | Rhea | 150.00 |
+| +I | delete | 3 | 2024-01-15 10:40:00 | 1 | Rhea | 150.00 |
++----+---------------+-------------+---------------------+----------+---------------+---------+
```
:::note
The `op` column is Flink's row kind indicator. For changelog virtual tables, all rows are emitted as `+I` (insert) to the downstream, while the actual change type is captured in the `_change_type` column.
:::
-#### Log Table Changelog
-Consider a Log Table storing click events:
+## Binlog Table
+
+The `$binlog` virtual table provides access to change data where each record contains both the before and after images of the row. This is useful for:
+
+:::note
+The `$binlog` virtual table is only available for **Primary Key Tables**.
+:::
+
+### Accessing the Binlog
+
+To access the binlog of a Primary Key Table, append `$binlog` to the table name:
+
+```sql title="Flink SQL"
+SELECT * FROM my_pk_table$binlog;
+```
+
+### Schema
+
+The binlog virtual table includes three metadata columns followed by nested `before` and `after` row structures:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `_change_type` | STRING NOT NULL | The type of change operation: `insert`, `update`, or `delete` |
+| `_log_offset` | BIGINT NOT NULL | The offset position in the log |
+| `_commit_timestamp` | TIMESTAMP_LTZ(3) NOT NULL | The timestamp when the change was committed |
+| `before` | ROW<...> | The row values before the change (NULL for inserts) |
+| `after` | ROW<...> | The row values after the change (NULL for deletes) |
+
+The `before` and `after` columns are nested ROW types containing all columns from the base table.
+
+### Change Types
+
+| Change Type | Description | `before` | `after` |
+|-------------|-------------|----------|---------|
+| `insert` | A new row was inserted | NULL | Contains new row values |
+| `update` | A row was updated | Contains old row values | Contains new row values |
+| `delete` | A row was deleted | Contains deleted row values | NULL |
+
+### Examples
```sql title="Flink SQL"
--- Create a log table (no primary key)
-CREATE TABLE click_events (
- event_id INT,
- user_id INT,
- event_type STRING
+-- Create a primary key table
+CREATE TABLE users (
+ user_id INT NOT NULL,
+ name STRING,
+ email STRING,
+ PRIMARY KEY (user_id) NOT ENFORCED
) WITH ('bucket.num' = '1');
--- Append events
-INSERT INTO click_events VALUES (1, 101, 'click'), (2, 102, 'view');
+-- Insert, update, then delete a record
+INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
+INSERT INTO users VALUES (1, 'Alice Smith', 'alice.smith@example.com');
+DELETE FROM users WHERE user_id = 1;
--- Query the changelog
-SELECT * FROM click_events$changelog;
+-- Query the binlog
+SELECT * FROM users$binlog;
```
Output:
```
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| op | _change_type | _log_offset | _commit_timestamp | event_id | user_id | event_type |
-+----+--------------+-------------+---------------------+----------+---------+------------+
-| +I | +A | 0 | 2024-01-15 11:00:00 | 1 | 101 | click |
-| +I | +A | 1 | 2024-01-15 11:00:00 | 2 | 102 | view |
-+----+--------------+-------------+---------------------+----------+---------+------------+
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| op | _change_type | _log_offset | _commit_timestamp | before | after |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+| +I | insert | 0 | 2024-01-15 10:30:00 | NULL | (1, Alice, alice@example.com) |
+| +I | update | 2 | 2024-01-15 10:35:00 | (1, Alice, alice@example.com) | (1, Alice Smith, alice.smith@example.com) |
+| +I | delete | 3 | 2024-01-15 10:40:00 | (1, Alice Smith, alice.smith@example.com) | NULL |
++----+--------------+-------------+---------------------+----------------------------------+--------------------------------------+
+```
+
+#### Accessing Nested Fields
+
+You can access individual fields from the `before` and `after` structures:
+
+```sql title="Flink SQL"
+SELECT
+ _change_type,
+ _commit_timestamp,
+ `before`.name AS old_name,
+ `after`.name AS new_name
+FROM users$binlog
+WHERE _change_type = 'update';
```
### Startup Modes
-The changelog virtual table supports different startup modes to control where reading begins:
+
+| Mode | Description |
+|------|-------------|
+| `earliest` | Start reading from the beginning of the log |
+| `latest` | Start reading from the current end of the log (only new changes) |
+| `timestamp` | Start reading from a specific timestamp (milliseconds since epoch) |
+
+
+The virtual table supports different startup modes to control where reading begins:
Review Comment:
Not all virtual table will support startup modes. We will also include `$lake` (and more in the future) into this page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] binlog virtual table [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #2565:
URL: https://github.com/apache/fluss/pull/2565#discussion_r2764242705
##########
website/docs/table-design/virtual-tables.md:
##########
@@ -55,23 +49,21 @@ For Primary Key Tables, the following change types are supported:
| Change Type | Description |
|-------------|-------------|
-| `+I` | **Insert** - A new row was inserted |
-| `-U` | **Update Before** - The previous value of an updated row (retraction) |
-| `+U` | **Update After** - The new value of an updated row |
-| `-D` | **Delete** - A row was deleted |
+| `insert` | A new row was inserted |
Review Comment:
Removing the op column and keeping the change_type column values small because it will originally be generated in smallcase, so to make it real and relatble not keeping it uppercase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org