You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/12/03 03:26:48 UTC
[GitHub] [flink] sjwiesman opened a new pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
sjwiesman opened a new pull request #14292:
URL: https://github.com/apache/flink/pull/14292
## What is the purpose of the change
The streaming concepts section of the documentation is very negative and focuses on the difficulties of SQL. While we don't want to lie to users about edge cases, or things that may be tricky, I think we can make the experience of getting started more approachable.
## Brief change log
See commit messages.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
## Does this pull request potentially affect one of the following parts:
No
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 69b091b21c93f13cf8e999367791f9a2f8ac2692 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] alpinegizmo commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
alpinegizmo commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-738136341
This needs a much more careful review than I've done so far, but I'll share a few initial impressions:
I prefer the original explanation of Relational Queries on Data Streams in the Dynamic Tables page. I find it easier to understand.
On the Time Attributes page I like putting event time before processing time. However, I am wondering about the deletion of the TableSource sub-sections. I don't like having those sections here -- these are low-level details that get in the way of understanding -- but is this valuable information that is being lost?
I mostly like what's going on in the page on joins, but at first I was left confused by the relationship between versioned tables and temporal joins. I had to read pretty closely to figure out what's going on; I think there are too many similar concepts with related names for this to be easy to grasp. I think we need to think carefully about how to organize the docs so that the relationships are clear.
I suggest we take the section on Temporal Table Functions out of the page on versioned tables, along with the section on joins with temporal table functions in the page on Joins in Continuous Queries, and put those two sections into their own "THIS PAGE DESCRIBES A LEGACY FEATURE" page by themselves.
We also need to do more than just move things around. Consider this note concerning the phony temporal table function example `SELECT * FROM Rates('11:00:00');`:
> Note: Currently, Flink doesn’t support directly querying the temporal table functions with a constant time attribute parameter. The above example was used to provide an intuition about what the function Rates(timeAttribute) returns.
This old note is nonsense, since temporal table functions are a legacy feature we intend to phase out. But the tone of this note suggests that there might be further development.
FWIW, these pages have typos and grammar issues, most of which pre-date this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bd3e853fd1bd205632bd8c22b82f621fb031a0d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506)
* 7d1723792f216e5c11cad7cfa8d8f1d58e5c842a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #14292:
URL: https://github.com/apache/flink/pull/14292#discussion_r539658939
##########
File path: docs/redirects/temporal_table.md
##########
@@ -0,0 +1,24 @@
+---
+title: Temporal Tables
+layout: redirect
+redirect: /dev/table/streaming/versioned_table.html
Review comment:
damn, thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bd3e853fd1bd205632bd8c22b82f621fb031a0d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506)
* 7d1723792f216e5c11cad7cfa8d8f1d58e5c842a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
}, {
"hash" : "023089909048230b211d7cb9e5f9cdee432ed411",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "023089909048230b211d7cb9e5f9cdee432ed411",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 69b091b21c93f13cf8e999367791f9a2f8ac2692 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661)
* 023089909048230b211d7cb9e5f9cdee432ed411 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737642096
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit dd2905f71cb565e5fd513217a056fa2d757f84e7 (Thu Dec 03 03:28:45 UTC 2020)
✅no warnings
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] alpinegizmo commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
alpinegizmo commented on a change in pull request #14292:
URL: https://github.com/apache/flink/pull/14292#discussion_r539643544
##########
File path: docs/redirects/temporal_table.md
##########
@@ -0,0 +1,24 @@
+---
+title: Temporal Tables
+layout: redirect
+redirect: /dev/table/streaming/versioned_table.html
Review comment:
The new file is versioned_tables (plural, with an 's'). But is this where the redirect should go, or should it go to legacy.md? Maybe it would make sense to leave it this way, redirecting to the page describing the new feature, but with a brief note on versioned_tables.md that mentions and links to the content in legacy.md.
And in general, should we also create redirects for the zh versions as well?
```suggestion
redirect: /dev/table/streaming/versioned_tables.html
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-741969426
I also renamed the file and added a redirect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #14292:
URL: https://github.com/apache/flink/pull/14292#discussion_r538482958
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In above example, the specific version of the table is tracked by `update_time` and `product_id`, the `product_id` would be a primary key for `product_changelog` table and `update_time` would be the event time.
-
-In Flink, this is represented by a [*versioned table*](#defining-versioned-table).
-
-### Correlate with a regular table
-On the other hand, some use cases require to join a regular table which is an external database table.
+While at `13:00:00,` we found find another.
-Let's assume that `LatestRates` is a table (e.g. stored in HBase) which is materialized with the latest rates. The `LatestRates` always represents the latest content of hbase table `rates`.
-
-Then the content of `LatestRates` table when we query at time `10:15:00` is:
{% highlight sql %}
-10:15:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 114
-Yen 1
-{% endhighlight %}
-
-Then the content of `LatestRates` table when we query at time `11:00:00` is:
-{% highlight sql %}
-11:00:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 116
-Yen 1
+update_time product_id product_name price
+=========== ========== ============ =====
+12:00:00 p_001 scooter 12.99
+12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In Flink, this is represented by a [*regular Table*](#defining-regular-table).
-Temporal Table
---------------
-<span class="label label-danger">Attention</span> This is only supported in Blink planner.
+## Versioned Table Sources
-Flink uses primary key constraint and event time to define both versioned table and versioned view.
+Versioned tables are defined implicity for any table whose underlying source or format directly define changelogs. Examples include the [upsert Kafka]({% link dev/table/connectors/upsert-kafka.md %}) source as well as database changelog formats such as [debezium]({% link dev/table/connectors/formats/debezium.md %}) and [canal]({% link dev/table/connectors/formats/canal.md %}). As discussed above, the only additional requirement is the `CREATE` table statement must contain a primary key and event-time attribute.
-### Defining Versioned Table
-The table is a versioned table in Flink only is the table contains primary key constraint and event time.
{% highlight sql %}
--- Define a versioned table
-CREATE TABLE product_changelog (
- product_id STRING,
- product_name STRING,
- product_price DECIMAL(10, 4),
- update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
- PRIMARY KEY(product_id) NOT ENFORCED, -- (1) defines the primary key constraint
- WATERMARK FOR update_time AS update_time -- (2) defines the event time by watermark
-) WITH (
- 'connector' = 'kafka',
- 'topic' = 'products',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'value.format' = 'debezium-json'
-);
+CREATE TABLE products (
+ product_id STRING,
+ product_name STRING,
+ price DECIMAL(32, 2),
+ update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
+ PRIMARY KEY (product_id) NOT ENFORCED
+ WATERMARK FOR update_time AS update_time
+) WITH (...);
{% endhighlight %}
-Line `(1)` defines the primary key constraint for table `product_changelog`, Line `(2)` defines the `update_time` as event time for table `product_changelog`,
-thus table `product_changelog` is a versioned table.
+## Versioned Table Views
-**Note**: The grammar `METADATA FROM 'value.source.timestamp' VIRTUAL` means extract the database
-operation execution time for every changelog, it's strongly recommended defines the database operation execution time
-as event time rather than ingestion-time or time in the record, otherwise the version extract from the changelog may
-mismatch with the version in database.
-
-### Defining Versioned View
+Flink also supports defining versioned views if the underlying query contains a unique key constraint and event-time attribute. Image an append-only table of currency rates.
-Flink also supports defining versioned view only if the view contains unique key constraint and event time.
-
-Let’s assume that we have the following table `RatesHistory`:
{% highlight sql %}
--- Define an append-only table
-CREATE TABLE RatesHistory (
- currency_time TIMESTAMP(3),
- currency STRING,
- rate DECIMAL(38, 10),
- WATERMARK FOR currency_time AS currency_time -- defines the event time
+CREATE TABLE currency_rates (
+ currency STRING,
+ rate DECIMAL(32, 10)
+ update_time TIMESTAMP(3),
+ WATERMARK FOR update_time AS update_time
) WITH (
- 'connector' = 'kafka',
- 'topic' = 'rates',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'format' = 'json' -- this is an append only source
-)
+ 'connector' = 'kafka',
+ 'topic' = 'rates',
+ 'properties.bootstrap.servers' = 'localhost:9092',
+ 'format' = 'json'
+);
{% endhighlight %}
-Table `RatesHistory` represents an ever growing append-only table of currency exchange rates with respect to
-Yen (which has a rate of 1). For example, the exchange rate for the period from 09:00 to 10:45 of Euro to Yen was 114.
-From 10:45 to 11:15 it was 116.
-
-{% highlight sql %}
-SELECT * FROM RatesHistory;
-
-currency_time currency rate
-============= ========= ====
-09:00:00 US Dollar 102
-09:00:00 Euro 114
-09:00:00 Yen 1
-10:45:00 Euro 116
-11:15:00 Euro 119
-11:49:00 Pounds 108
-{% endhighlight %}
+The table `currency_rates` contains a row for each currency - with respect to USD - and receives a new row each time the rate changes.
+The `JSON` format does not support native changelog semantics, and so Flink can only read this table as append-only.
+However, it is clear to us (the query developer) that this table has all the necessary information to define a versioned table.
-To define a versioned table on `RatesHistory`, Flink supports defining a versioned view
-by [deduplication query]({% link dev/table/sql/queries.md %}#deduplication) which produces an ordered changelog
-stream with an inferred primary key(`currency`) and event time(`currency_time`).
+Flink can reinterpret this table as a versioned table by defining a [deduplication query]({% link dev/table/sql/queries.md %}#deduplication) which produces an ordered changelog stream with an inferred primary key (currency) and event time (update_time).
Review comment:
The real answer is flink has a special optimization step that turns any `OVER WINDOW` with exactly this format into a versioned table keyed by the fields in the `PARTITION BY` clause.
```sql
SELECT *,
ROW_NUMBER() OVER (PARTITION BY [key,] ORDER BY [time attribute] DESC) AS row_num
WHERE
row_num = 1
```
In a batch query, this would return the most recent value for each currency. The streaming semantic is it provides the new value for each partition each time a more recent value is made available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
}, {
"hash" : "023089909048230b211d7cb9e5f9cdee432ed411",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670",
"triggerID" : "023089909048230b211d7cb9e5f9cdee432ed411",
"triggerType" : "PUSH"
}, {
"hash" : "c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10725",
"triggerID" : "c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10725)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
}, {
"hash" : "023089909048230b211d7cb9e5f9cdee432ed411",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670",
"triggerID" : "023089909048230b211d7cb9e5f9cdee432ed411",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 69b091b21c93f13cf8e999367791f9a2f8ac2692 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661)
* 023089909048230b211d7cb9e5f9cdee432ed411 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-738448892
Thanks for taking a look @alpinegizmo
I went back and made a grammar pass on each page
> I prefer the original explanation of Relational Queries on Data Streams in the Dynamic Tables page. I find it easier to understand.
I put the table back and just slightly tweaked the language to try and be more approachable
> On the Time Attributes page I like putting event time before processing time. However, I am wondering about the deletion of the TableSource sub-sections. I don't like having those sections here -- these are low-level details that get in the way of understanding -- but is this valuable information that is being lost?
The `TableSource` interface is deprecated, that's why I dropped it.
I also completely rewrote the versioned table description. I think it's much better now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] alpinegizmo commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
alpinegizmo commented on a change in pull request #14292:
URL: https://github.com/apache/flink/pull/14292#discussion_r539417802
##########
File path: docs/dev/table/streaming/legacy.md
##########
@@ -0,0 +1,135 @@
+---
+title: "Legacy Features"
+nav-parent_id: streaming_tableapi
+nav-pos: 1001
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+As Flink SQL has matured there are some features that have been replaced with more modern and better functioning substitutes.
+These legacy features remain documented here for those users that have not yet or are unable to, upgrade to the more modern variant.
+
+* This will be replaced by the TOC
+{:toc}
+
+# Temporal Table Function
+
+The temporal table function is the legacy way of defining something akin to a [versioned table]({% link dev/table/streaming/temporal_tables.md %})
+that can be used in a temporal table join.
+Please define temporal joins using [versioned table]({% link dev/table/streaming/temporal_tables.md %}) as described above in new queries.
Review comment:
Temporal tables are no longer "above".
```suggestion
Please define temporal joins using [versioned tables]({% link dev/table/streaming/temporal_tables.md %}) in new queries.
```
##########
File path: docs/dev/table/streaming/legacy.md
##########
@@ -0,0 +1,135 @@
+---
+title: "Legacy Features"
+nav-parent_id: streaming_tableapi
+nav-pos: 1001
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+As Flink SQL has matured there are some features that have been replaced with more modern and better functioning substitutes.
+These legacy features remain documented here for those users that have not yet or are unable to, upgrade to the more modern variant.
+
+* This will be replaced by the TOC
+{:toc}
+
+# Temporal Table Function
+
+The temporal table function is the legacy way of defining something akin to a [versioned table]({% link dev/table/streaming/temporal_tables.md %})
+that can be used in a temporal table join.
+Please define temporal joins using [versioned table]({% link dev/table/streaming/temporal_tables.md %}) as described above in new queries.
+
+Unlike a versioned table, temporal table functions can only be defined on top of append-only streams
+— it does not support changelog inputs.
+Additionally, a temporal table function cannot be defined in pure SQL DDL.
+
+#### Defining a Temporal Table Function
+
+Temporal table function can be defined on top of append-only streams using the [Table API]({% link dev/table/tableApi.md %}).
+The table is registered with one or more key columns, and a time attribute used for versioning.
+
+Suppose we have an append-only table of currency rates that we would like to
+register as a temporal table function.
+
+{% highlight sql %}
+SELECT * FROM currency_rates;
+
+update_time currency rate
+============= ========= ====
+09:00:00 Yen 102
+09:00:00 Euro 114
+09:00:00 USD 1
+11:15:00 Euro 119
+11:49:00 Pounds 108
+{% endhighlight %}
+
+Using the Table API, we can register this stream using `currency` for the key and `update_time` as
+the versioning time attribute.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+TemporalTableFunction rates = tEnv
+ .from("currency_rates").
+ .createTemporalTableFunction("update_time", "currency");
+
+tEnv.registerFunction("rates", rates);
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+rates = tEnv
+ .from("currency_rates").
+ .createTemporalTableFunction("update_time", "currency")
+
+tEnv.registerFunction("rates", rates)
+{% endhighlight %}
+</div>
+</div>
+
+#### Temporal Table Function Join
+
+Once defined, a temporal table function is used as a standard [table function]({% link dev/table/functions/udfs.md %}#table-functions).
+Append-only tables (left input/probe side) can join with a temporal table (right input/build side),
+i.e., a table that changes over time and tracks its changes, to retrieve the value for a key as it was at a particular point in time.
+
+Consider an append-only table `orders` that tracks customers orders in different currencies.
Review comment:
```suggestion
Consider an append-only table `orders` that tracks customers' orders in different currencies.
```
##########
File path: docs/dev/table/streaming/legacy.md
##########
@@ -0,0 +1,135 @@
+---
+title: "Legacy Features"
+nav-parent_id: streaming_tableapi
+nav-pos: 1001
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+As Flink SQL has matured there are some features that have been replaced with more modern and better functioning substitutes.
+These legacy features remain documented here for those users that have not yet or are unable to, upgrade to the more modern variant.
+
+* This will be replaced by the TOC
+{:toc}
+
+# Temporal Table Function
+
+The temporal table function is the legacy way of defining something akin to a [versioned table]({% link dev/table/streaming/temporal_tables.md %})
+that can be used in a temporal table join.
+Please define temporal joins using [versioned table]({% link dev/table/streaming/temporal_tables.md %}) as described above in new queries.
+
+Unlike a versioned table, temporal table functions can only be defined on top of append-only streams
+— it does not support changelog inputs.
+Additionally, a temporal table function cannot be defined in pure SQL DDL.
+
+#### Defining a Temporal Table Function
+
+Temporal table function can be defined on top of append-only streams using the [Table API]({% link dev/table/tableApi.md %}).
Review comment:
```suggestion
Temporal table functions can be defined on top of append-only streams using the [Table API]({% link dev/table/tableApi.md %}).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d1723792f216e5c11cad7cfa8d8f1d58e5c842a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508)
* 6ca7d5619e4d14b3da3cca362da015785ba199eb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-741923224
> Also, I think it might be a preferable to rename temporal_tables.md to versioned_tables.md. It's unfortunate that the page title no longer matches the filename. But doing this would mean checking and updating any links to this page (and maybe even adding a redirect).
Yeah, that's why I avoided it 🙈 I'll make the change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6ca7d5619e4d14b3da3cca362da015785ba199eb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd2905f71cb565e5fd513217a056fa2d757f84e7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-742074292
thanks for the review, will merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd2905f71cb565e5fd513217a056fa2d757f84e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
}, {
"hash" : "023089909048230b211d7cb9e5f9cdee432ed411",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670",
"triggerID" : "023089909048230b211d7cb9e5f9cdee432ed411",
"triggerType" : "PUSH"
}, {
"hash" : "c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 023089909048230b211d7cb9e5f9cdee432ed411 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670)
* c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d1723792f216e5c11cad7cfa8d8f1d58e5c842a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
}, {
"hash" : "023089909048230b211d7cb9e5f9cdee432ed411",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670",
"triggerID" : "023089909048230b211d7cb9e5f9cdee432ed411",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 023089909048230b211d7cb9e5f9cdee432ed411 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman closed pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman closed pull request #14292:
URL: https://github.com/apache/flink/pull/14292
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd2905f71cb565e5fd513217a056fa2d757f84e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] alpinegizmo commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
alpinegizmo commented on a change in pull request #14292:
URL: https://github.com/apache/flink/pull/14292#discussion_r535987359
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -22,213 +22,42 @@ specific language governing permissions and limitations
under the License.
-->
-Flink is able to process streaming data based on different notions of *time*.
+Flink can process data based on different notions of time.
-- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
-- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
-- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
+*Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
Review comment:
```suggestion
- *Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -22,213 +22,42 @@ specific language governing permissions and limitations
under the License.
-->
-Flink is able to process streaming data based on different notions of *time*.
+Flink can process data based on different notions of time.
-- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
-- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
-- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
+*Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
+- *Event time* refers to the processing of streaming data based on timestamps that are attached to each row. The timestamps can encode when an event happened.
-For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({% link dev/event_time.md %}).
+For more information about time handling in Flink, see the introduction about [event time and watermarks]({% link dev/event_time.md %}).
-This page explains how time attributes can be defined for time-based operations in Flink's Table API & SQL.
* This will be replaced by the TOC
{:toc}
Introduction to Time Attributes
-------------------------------
-Time-based operations such as windows in both the [Table API]({% link dev/table/tableApi.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.md %}#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+Time attributes can be part of every table schema.
+They are defined when creating a table from a `CREATE TABLE DDL` or a `DataStream`.
+Once a time attribute is defined, it can be referenced as a field and used in time-based operations.
+As long as a time attribute is not modified and forwarded from one part of the query to another, it remains valid.
Review comment:
```suggestion
As long as a time attribute is not modified, and is simply forwarded from one part of a query to another, it remains a valid time attribute.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -22,213 +22,42 @@ specific language governing permissions and limitations
under the License.
-->
-Flink is able to process streaming data based on different notions of *time*.
+Flink can process data based on different notions of time.
-- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
-- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
-- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
+*Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
+- *Event time* refers to the processing of streaming data based on timestamps that are attached to each row. The timestamps can encode when an event happened.
-For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({% link dev/event_time.md %}).
+For more information about time handling in Flink, see the introduction about [event time and watermarks]({% link dev/event_time.md %}).
-This page explains how time attributes can be defined for time-based operations in Flink's Table API & SQL.
* This will be replaced by the TOC
{:toc}
Introduction to Time Attributes
-------------------------------
-Time-based operations such as windows in both the [Table API]({% link dev/table/tableApi.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.md %}#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+Time attributes can be part of every table schema.
+They are defined when creating a table from a `CREATE TABLE DDL` or a `DataStream`.
+Once a time attribute is defined, it can be referenced as a field and used in time-based operations.
+As long as a time attribute is not modified and forwarded from one part of the query to another, it remains valid.
+Time attributes behave like regular timestamps and accessible for calculations.
Review comment:
```suggestion
Time attributes behave like regular timestamps, and are accessible for calculations.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -22,213 +22,42 @@ specific language governing permissions and limitations
under the License.
-->
-Flink is able to process streaming data based on different notions of *time*.
+Flink can process data based on different notions of time.
-- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
-- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
-- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
+*Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
+- *Event time* refers to the processing of streaming data based on timestamps that are attached to each row. The timestamps can encode when an event happened.
-For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({% link dev/event_time.md %}).
+For more information about time handling in Flink, see the introduction about [event time and watermarks]({% link dev/event_time.md %}).
-This page explains how time attributes can be defined for time-based operations in Flink's Table API & SQL.
* This will be replaced by the TOC
{:toc}
Introduction to Time Attributes
-------------------------------
-Time-based operations such as windows in both the [Table API]({% link dev/table/tableApi.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.md %}#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+Time attributes can be part of every table schema.
+They are defined when creating a table from a `CREATE TABLE DDL` or a `DataStream`.
+Once a time attribute is defined, it can be referenced as a field and used in time-based operations.
+As long as a time attribute is not modified and forwarded from one part of the query to another, it remains valid.
+Time attributes behave like regular timestamps and accessible for calculations.
+When used in computation, time attributes are materialized and act as standard timestamps.
+However, the opposite is not true; ordinary timestamp columns cannot be arbitrarily converted to time attributes in the middle of a query.
Review comment:
```suggestion
However, ordinary timestamps cannot be used in place of, or be converted to, time attributes.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -318,98 +145,61 @@ val windowedTable = table.window(Tumble over 10.minutes on $"user_action_time" a
</div>
</div>
-### Using a TableSource
-The event time attribute is defined by a `TableSource` that implements the `DefinedRowtimeAttributes` interface. The `getRowtimeAttributeDescriptors()` method returns a list of `RowtimeAttributeDescriptor` for describing the final name of a time attribute, a timestamp extractor to derive the values of the attribute, and the watermark strategy associated with the attribute.
+Processing Time
+---------------
+
+Processing time allows a table program to produce results based on the time of the local machine. It is the simplest notion of time but does not provide determinism. It neither requires timestamp extraction nor watermark generation.
Review comment:
```suggestion
Processing time allows a table program to produce results based on the time of the local machine. It is the simplest notion of time, but it will generate non-deterministic results. Processing time does not require timestamp extraction or watermark generation.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -22,213 +22,42 @@ specific language governing permissions and limitations
under the License.
-->
-Flink is able to process streaming data based on different notions of *time*.
+Flink can process data based on different notions of time.
-- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
-- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
-- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
+*Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
+- *Event time* refers to the processing of streaming data based on timestamps that are attached to each row. The timestamps can encode when an event happened.
-For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({% link dev/event_time.md %}).
+For more information about time handling in Flink, see the introduction about [event time and watermarks]({% link dev/event_time.md %}).
-This page explains how time attributes can be defined for time-based operations in Flink's Table API & SQL.
* This will be replaced by the TOC
{:toc}
Introduction to Time Attributes
-------------------------------
-Time-based operations such as windows in both the [Table API]({% link dev/table/tableApi.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.md %}#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+Time attributes can be part of every table schema.
+They are defined when creating a table from a `CREATE TABLE DDL` or a `DataStream`.
+Once a time attribute is defined, it can be referenced as a field and used in time-based operations.
+As long as a time attribute is not modified and forwarded from one part of the query to another, it remains valid.
+Time attributes behave like regular timestamps and accessible for calculations.
+When used in computation, time attributes are materialized and act as standard timestamps.
+However, the opposite is not true; ordinary timestamp columns cannot be arbitrarily converted to time attributes in the middle of a query.
-Time attributes can be part of every table schema. They are defined when creating a table from a CREATE TABLE DDL or a `DataStream` or are pre-defined when using a `TableSource`. Once a time attribute has been defined at the beginning, it can be referenced as a field and can be used in time-based operations.
-
-As long as a time attribute is not modified and is simply forwarded from one part of the query to another, it remains a valid time attribute. Time attributes behave like regular timestamps and can be accessed for calculations. If a time attribute is used in a calculation, it will be materialized and becomes a regular timestamp. Regular timestamps do not cooperate with Flink's time and watermarking system and thus can not be used for time-based operations anymore.
-
-Table programs require that the corresponding time characteristic has been specified for the streaming environment:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-
-env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime); // default
-
-// alternatively:
-// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
-// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-
-env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime) // default
-
-// alternatively:
-// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime)
-// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
-{% endhighlight %}
-</div>
-<div data-lang="python" markdown="1">
-{% highlight python %}
-env = StreamExecutionEnvironment.get_execution_environment()
-
-env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime) # default
-
-# alternatively:
-# env.set_stream_time_characteristic(TimeCharacteristic.IngestionTime)
-# env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
-{% endhighlight %}
-</div>
-</div>
-
-Processing time
----------------
-
-Processing time allows a table program to produce results based on the time of the local machine. It is the simplest notion of time but does not provide determinism. It neither requires timestamp extraction nor watermark generation.
-
-There are three ways to define a processing time attribute.
-
-### Defining in create table DDL
-
-The processing time attribute is defined as a computed column in create table DDL using the system `PROCTIME()` function. Please see [CREATE TABLE DDL]({% link dev/table/sql/create.md %}#create-table) for more information about computed column.
-
-{% highlight sql %}
-
-CREATE TABLE user_actions (
- user_name STRING,
- data STRING,
- user_action_time AS PROCTIME() -- declare an additional field as a processing time attribute
-) WITH (
- ...
-);
-
-SELECT TUMBLE_START(user_action_time, INTERVAL '10' MINUTE), COUNT(DISTINCT user_name)
-FROM user_actions
-GROUP BY TUMBLE(user_action_time, INTERVAL '10' MINUTE);
-
-{% endhighlight %}
-
-
-### During DataStream-to-Table Conversion
-
-The processing time attribute is defined with the `.proctime` property during schema definition. The time attribute must only extend the physical schema by an additional logical field. Thus, it can only be defined at the end of the schema definition.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-DataStream<Tuple2<String, String>> stream = ...;
-
-// declare an additional logical field as a processing time attribute
-Table table = tEnv.fromDataStream(stream, $("user_name"), $("data"), $("user_action_time").proctime());
-
-WindowedTable windowedTable = table.window(
- Tumble.over(lit(10).minutes())
- .on($("user_action_time"))
- .as("userActionWindow"));
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val stream: DataStream[(String, String)] = ...
-
-// declare an additional logical field as a processing time attribute
-val table = tEnv.fromDataStream(stream, $"UserActionTimestamp", $"user_name", $"data", $"user_action_time".proctime)
-
-val windowedTable = table.window(Tumble over 10.minutes on $"user_action_time" as "userActionWindow")
-{% endhighlight %}
-</div>
-</div>
-
-### Using a TableSource
-
-The processing time attribute is defined by a `TableSource` that implements the `DefinedProctimeAttribute` interface. The logical time attribute is appended to the physical schema defined by the return type of the `TableSource`.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// define a table source with a processing attribute
-public class UserActionSource implements StreamTableSource<Row>, DefinedProctimeAttribute {
-
- @Override
- public TypeInformation<Row> getReturnType() {
- String[] names = new String[] {"user_name" , "data"};
- TypeInformation[] types = new TypeInformation[] {Types.STRING(), Types.STRING()};
- return Types.ROW(names, types);
- }
-
- @Override
- public DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv) {
- // create stream
- DataStream<Row> stream = ...;
- return stream;
- }
-
- @Override
- public String getProctimeAttribute() {
- // field with this name will be appended as a third field
- return "user_action_time";
- }
-}
-
-// register table source
-tEnv.registerTableSource("user_actions", new UserActionSource());
-
-WindowedTable windowedTable = tEnv
- .from("user_actions")
- .window(Tumble
- .over(lit(10).minutes())
- .on($("user_action_time"))
- .as("userActionWindow"));
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// define a table source with a processing attribute
-class UserActionSource extends StreamTableSource[Row] with DefinedProctimeAttribute {
-
- override def getReturnType = {
- val names = Array[String]("user_name" , "data")
- val types = Array[TypeInformation[_]](Types.STRING, Types.STRING)
- Types.ROW(names, types)
- }
-
- override def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[Row] = {
- // create stream
- val stream = ...
- stream
- }
-
- override def getProctimeAttribute = {
- // field with this name will be appended as a third field
- "user_action_time"
- }
-}
-
-// register table source
-tEnv.registerTableSource("user_actions", new UserActionSource)
-
-val windowedTable = tEnv
- .from("user_actions")
- .window(Tumble over 10.minutes on $"user_action_time" as "userActionWindow")
-{% endhighlight %}
-</div>
-</div>
-
-Event time
+Event Time
----------
-Event time allows a table program to produce results based on the time that is contained in every record. This allows for consistent results even in case of out-of-order events or late events. It also ensures replayable results of the table program when reading records from persistent storage.
+Event time allows a table program to produce results based on timestamps in every record, allowing for consistent results even in case of out-of-order events or late events. It also ensures the replayability of the results of the table program when reading records from persistent storage.
-Additionally, event time allows for unified syntax for table programs in both batch and streaming environments. A time attribute in a streaming environment can be a regular field of a record in a batch environment.
+Additionally, event time allows for unified syntax for table programs in both batch and streaming environments. A time attribute in a streaming environment can be a regular column of a row in a batch environment.
-In order to handle out-of-order events and distinguish between on-time and late events in streaming, Flink needs to extract timestamps from events and make some kind of progress in time (so-called [watermarks]({% link dev/event_time.md %})).
+To handle out-of-order events and distinguish between on-time and late events in streaming, Flink needs to extract timestamps from events and make some progress in time (so-called [watermarks]({% link dev/event_time.md %})).
Review comment:
```suggestion
To handle out-of-order events and to distinguish between on-time and late events in streaming, Flink needs to know the timestamp for each row, and it also needs regular indications of how far along in event time the processing has progressed so far (via so-called [watermarks]({% link dev/event_time.md %})).
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -22,213 +22,42 @@ specific language governing permissions and limitations
under the License.
-->
-Flink is able to process streaming data based on different notions of *time*.
+Flink can process data based on different notions of time.
-- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
-- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
-- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
+*Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
+- *Event time* refers to the processing of streaming data based on timestamps that are attached to each row. The timestamps can encode when an event happened.
-For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({% link dev/event_time.md %}).
+For more information about time handling in Flink, see the introduction about [event time and watermarks]({% link dev/event_time.md %}).
-This page explains how time attributes can be defined for time-based operations in Flink's Table API & SQL.
* This will be replaced by the TOC
{:toc}
Introduction to Time Attributes
-------------------------------
-Time-based operations such as windows in both the [Table API]({% link dev/table/tableApi.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.md %}#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+Time attributes can be part of every table schema.
+They are defined when creating a table from a `CREATE TABLE DDL` or a `DataStream`.
+Once a time attribute is defined, it can be referenced as a field and used in time-based operations.
+As long as a time attribute is not modified and forwarded from one part of the query to another, it remains valid.
+Time attributes behave like regular timestamps and accessible for calculations.
+When used in computation, time attributes are materialized and act as standard timestamps.
+However, the opposite is not true; ordinary timestamp columns cannot be arbitrarily converted to time attributes in the middle of a query.
-Time attributes can be part of every table schema. They are defined when creating a table from a CREATE TABLE DDL or a `DataStream` or are pre-defined when using a `TableSource`. Once a time attribute has been defined at the beginning, it can be referenced as a field and can be used in time-based operations.
-
-As long as a time attribute is not modified and is simply forwarded from one part of the query to another, it remains a valid time attribute. Time attributes behave like regular timestamps and can be accessed for calculations. If a time attribute is used in a calculation, it will be materialized and becomes a regular timestamp. Regular timestamps do not cooperate with Flink's time and watermarking system and thus can not be used for time-based operations anymore.
-
-Table programs require that the corresponding time characteristic has been specified for the streaming environment:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-
-env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime); // default
-
-// alternatively:
-// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
-// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-
-env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime) // default
-
-// alternatively:
-// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime)
-// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
-{% endhighlight %}
-</div>
-<div data-lang="python" markdown="1">
-{% highlight python %}
-env = StreamExecutionEnvironment.get_execution_environment()
-
-env.set_stream_time_characteristic(TimeCharacteristic.ProcessingTime) # default
-
-# alternatively:
-# env.set_stream_time_characteristic(TimeCharacteristic.IngestionTime)
-# env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
-{% endhighlight %}
-</div>
-</div>
-
-Processing time
----------------
-
-Processing time allows a table program to produce results based on the time of the local machine. It is the simplest notion of time but does not provide determinism. It neither requires timestamp extraction nor watermark generation.
-
-There are three ways to define a processing time attribute.
-
-### Defining in create table DDL
-
-The processing time attribute is defined as a computed column in create table DDL using the system `PROCTIME()` function. Please see [CREATE TABLE DDL]({% link dev/table/sql/create.md %}#create-table) for more information about computed column.
-
-{% highlight sql %}
-
-CREATE TABLE user_actions (
- user_name STRING,
- data STRING,
- user_action_time AS PROCTIME() -- declare an additional field as a processing time attribute
-) WITH (
- ...
-);
-
-SELECT TUMBLE_START(user_action_time, INTERVAL '10' MINUTE), COUNT(DISTINCT user_name)
-FROM user_actions
-GROUP BY TUMBLE(user_action_time, INTERVAL '10' MINUTE);
-
-{% endhighlight %}
-
-
-### During DataStream-to-Table Conversion
-
-The processing time attribute is defined with the `.proctime` property during schema definition. The time attribute must only extend the physical schema by an additional logical field. Thus, it can only be defined at the end of the schema definition.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-DataStream<Tuple2<String, String>> stream = ...;
-
-// declare an additional logical field as a processing time attribute
-Table table = tEnv.fromDataStream(stream, $("user_name"), $("data"), $("user_action_time").proctime());
-
-WindowedTable windowedTable = table.window(
- Tumble.over(lit(10).minutes())
- .on($("user_action_time"))
- .as("userActionWindow"));
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val stream: DataStream[(String, String)] = ...
-
-// declare an additional logical field as a processing time attribute
-val table = tEnv.fromDataStream(stream, $"UserActionTimestamp", $"user_name", $"data", $"user_action_time".proctime)
-
-val windowedTable = table.window(Tumble over 10.minutes on $"user_action_time" as "userActionWindow")
-{% endhighlight %}
-</div>
-</div>
-
-### Using a TableSource
-
-The processing time attribute is defined by a `TableSource` that implements the `DefinedProctimeAttribute` interface. The logical time attribute is appended to the physical schema defined by the return type of the `TableSource`.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// define a table source with a processing attribute
-public class UserActionSource implements StreamTableSource<Row>, DefinedProctimeAttribute {
-
- @Override
- public TypeInformation<Row> getReturnType() {
- String[] names = new String[] {"user_name" , "data"};
- TypeInformation[] types = new TypeInformation[] {Types.STRING(), Types.STRING()};
- return Types.ROW(names, types);
- }
-
- @Override
- public DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv) {
- // create stream
- DataStream<Row> stream = ...;
- return stream;
- }
-
- @Override
- public String getProctimeAttribute() {
- // field with this name will be appended as a third field
- return "user_action_time";
- }
-}
-
-// register table source
-tEnv.registerTableSource("user_actions", new UserActionSource());
-
-WindowedTable windowedTable = tEnv
- .from("user_actions")
- .window(Tumble
- .over(lit(10).minutes())
- .on($("user_action_time"))
- .as("userActionWindow"));
-{% endhighlight %}
-</div>
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// define a table source with a processing attribute
-class UserActionSource extends StreamTableSource[Row] with DefinedProctimeAttribute {
-
- override def getReturnType = {
- val names = Array[String]("user_name" , "data")
- val types = Array[TypeInformation[_]](Types.STRING, Types.STRING)
- Types.ROW(names, types)
- }
-
- override def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[Row] = {
- // create stream
- val stream = ...
- stream
- }
-
- override def getProctimeAttribute = {
- // field with this name will be appended as a third field
- "user_action_time"
- }
-}
-
-// register table source
-tEnv.registerTableSource("user_actions", new UserActionSource)
-
-val windowedTable = tEnv
- .from("user_actions")
- .window(Tumble over 10.minutes on $"user_action_time" as "userActionWindow")
-{% endhighlight %}
-</div>
-</div>
-
-Event time
+Event Time
----------
-Event time allows a table program to produce results based on the time that is contained in every record. This allows for consistent results even in case of out-of-order events or late events. It also ensures replayable results of the table program when reading records from persistent storage.
+Event time allows a table program to produce results based on timestamps in every record, allowing for consistent results even in case of out-of-order events or late events. It also ensures the replayability of the results of the table program when reading records from persistent storage.
Review comment:
Can we really argue that results are consistent with late events?
```suggestion
Event time allows a table program to produce results based on timestamps in every record, allowing for consistent results despite out-of-order or late events. It also ensures the replayability of the results of the table program when reading records from persistent storage.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -22,213 +22,42 @@ specific language governing permissions and limitations
under the License.
-->
-Flink is able to process streaming data based on different notions of *time*.
+Flink can process data based on different notions of time.
-- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
-- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
-- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
+*Processing time* refers to the machine's system time (also known as "wall-clock time") that is executing the respective operation.
+- *Event time* refers to the processing of streaming data based on timestamps that are attached to each row. The timestamps can encode when an event happened.
-For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({% link dev/event_time.md %}).
+For more information about time handling in Flink, see the introduction about [event time and watermarks]({% link dev/event_time.md %}).
-This page explains how time attributes can be defined for time-based operations in Flink's Table API & SQL.
* This will be replaced by the TOC
{:toc}
Introduction to Time Attributes
-------------------------------
-Time-based operations such as windows in both the [Table API]({% link dev/table/tableApi.md %}#group-windows) and [SQL]({% link dev/table/sql/queries.md %}#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+Time attributes can be part of every table schema.
+They are defined when creating a table from a `CREATE TABLE DDL` or a `DataStream`.
+Once a time attribute is defined, it can be referenced as a field and used in time-based operations.
+As long as a time attribute is not modified and forwarded from one part of the query to another, it remains valid.
+Time attributes behave like regular timestamps and accessible for calculations.
+When used in computation, time attributes are materialized and act as standard timestamps.
Review comment:
```suggestion
When used in calculations, time attributes are materialized and act as standard timestamps.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -251,14 +80,12 @@ GROUP BY TUMBLE(user_action_time, INTERVAL '10' MINUTE);
### During DataStream-to-Table Conversion
-The event time attribute is defined with the `.rowtime` property during schema definition. [Timestamps and watermarks]({% link dev/event_time.md %}) must have been assigned in the `DataStream` that is converted.
+When converting a `DataStream` to a table, an event time attribute can be defined with the `.rowtime` property during schema definition. [Timestamps and watermarks]({% link dev/event_time.md %}) must have been already assigned in the `DataStream` being converted.
-There are two ways of defining the time attribute when converting a `DataStream` into a `Table`. Depending on whether the specified `.rowtime` field name exists in the schema of the `DataStream` or not, the timestamp field is either
+There are two ways of defining the time attribute when converting a `DataStream` into a `Table`. Depending on whether the specified `.rowtime` field name exists in the schema of the `DataStream`, the timestamp is either (1) appended as a new column to the schema or
+(2) replaces an existing column.
Review comment:
```suggestion
There are two ways of defining the time attribute when converting a `DataStream` into a `Table`. Depending on whether the specified `.rowtime` field name exists in the schema of the `DataStream`, the timestamp is either (1) appended as a new column, or it
(2) replaces an existing column.
```
##########
File path: docs/dev/table/streaming/time_attributes.md
##########
@@ -251,14 +80,12 @@ GROUP BY TUMBLE(user_action_time, INTERVAL '10' MINUTE);
### During DataStream-to-Table Conversion
-The event time attribute is defined with the `.rowtime` property during schema definition. [Timestamps and watermarks]({% link dev/event_time.md %}) must have been assigned in the `DataStream` that is converted.
+When converting a `DataStream` to a table, an event time attribute can be defined with the `.rowtime` property during schema definition. [Timestamps and watermarks]({% link dev/event_time.md %}) must have been already assigned in the `DataStream` being converted.
Review comment:
```suggestion
When converting a `DataStream` to a table, an event time attribute can be defined with the `.rowtime` property during schema definition. [Timestamps and watermarks]({% link dev/event_time.md %}) must have already been assigned in the `DataStream` being converted.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
}, {
"hash" : "023089909048230b211d7cb9e5f9cdee432ed411",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670",
"triggerID" : "023089909048230b211d7cb9e5f9cdee432ed411",
"triggerType" : "PUSH"
}, {
"hash" : "c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10725",
"triggerID" : "c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 023089909048230b211d7cb9e5f9cdee432ed411 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10670)
* c6ce81f794da8c8249aa4d0a9f52aa8730b6c45c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10725)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-738448892
Thanks for taking a look @alpinegizmo
I went back and made a grammar pass on each page (except joins I haven't gotten that far yet).
> I prefer the original explanation of Relational Queries on Data Streams in the Dynamic Tables page. I find it easier to understand.
I put the table back and just slightly tweaked the language to try and be more approachable
> On the Time Attributes page I like putting event time before processing time. However, I am wondering about the deletion of the TableSource sub-sections. I don't like having those sections here -- these are low-level details that get in the way of understanding -- but is this valuable information that is being lost?
The `TableSource` interface is deprecated, that's why I dropped it.
I also completely rewrote the versioned table description. I think it's much better now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-741969240
@alpinegizmo I've applied your changes, rebased, and squashed the fixup. Let me know if you have any other concerns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-738830651
thank you @alpinegizmo I've applied your suggestions inline.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-740035289
If there aren't any other comments, I'll merge this tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] alpinegizmo commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
alpinegizmo commented on a change in pull request #14292:
URL: https://github.com/apache/flink/pull/14292#discussion_r538223361
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -22,30 +22,29 @@ specific language governing permissions and limitations
under the License.
-->
-A Temporal table is a table that evolves over time - otherwise known in Flink as a [dynamic table]({% link dev/table/streaming/dynamic_tables.md %}). Rows in a temporal table are associated with one or more temporal periods and all Flink tables are temporal(dynamic).
+Flink SQL operates over dynamic tables that evolve, which may either be append-only or updating.
+Versioned tables represent a special type of updating table that remembers the past values for each key.
-A temporal table contains one or more versioned table snapshots, it can be a changing history table which tracks the changes(e.g. database changelog, contains all snapshots) or a changing dimensioned table which materializes the changes(e.g. database table, contains the latest snapshot).
+<span class="label label-danger">Attention</span> The Legacy planner does not support versioned tables.
-**Version**: A temporal table can split into a set of versioned table snapshots, the version in table snapshots represents the valid life circle of rows, the start time and the end time of the valid period can be assigned by users.
-Temporal table can split to `versioned table` and `regular table` according to the table can tracks its history version or not.
+* This will be replaced by the TOC
+{:toc}
-**Versioned table**: If the rows a in temporal table can track its history changes and visit its history versions, we call this a versioned table. Tables that comes from a database changelog can be defined as a versioned table.
+## Concept
-**Regular table**: If the row in temporal table can only track and visit its latest version，we call this kind of temporal table as regular table. Tables that comes from a database or HBase can be defined as a regular table.
+Dynamic tables define relations over time.
+Often, particularly when working with metadata, a key's old value does not become irrelevant when it changes.
-* This will be replaced by the TOC
-{:toc}
+Flink SQL can define versioned tables over any dynamic table with a `PRIMARY KEY` constraint and time attribute.
-Motivation
-----------
+A primary key constraint in Flink means that a column or set of columns of a table or view are unique and non-null.
+The primary key semantic on a upserting table means the materialized changes for a particular key (`INSERT`/`UPDATE`/`DELETE`) represent the changes to a single row over time. The time attribute on a upserting table defines when each change occurred.
-### Correlate with a versioned table
-Given a scenario the order stream correlates the dimension table product, the table `orders` comes from kafka which contains the real time orders, the table `product_changelog` comes from the changelog of the database table `products`,
- the product price in table `products` is changing over time.
+Taken together, Flink can track the changes to a row over time and maintain the period for which each value was valid for that key.
-{% highlight sql %}
-SELECT * FROM product_changelog;
+Suppose a table that tracks the price of different products in a store.
Review comment:
```suggestion
Suppose a table tracks the prices of different products in a store.
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
Review comment:
```suggestion
00:01:00 p_001 scooter 11.11
00:02:00 p_002 basketball 23.11
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In above example, the specific version of the table is tracked by `update_time` and `product_id`, the `product_id` would be a primary key for `product_changelog` table and `update_time` would be the event time.
-
-In Flink, this is represented by a [*versioned table*](#defining-versioned-table).
-
-### Correlate with a regular table
-On the other hand, some use cases require to join a regular table which is an external database table.
+While at `13:00:00,` we found find another.
-Let's assume that `LatestRates` is a table (e.g. stored in HBase) which is materialized with the latest rates. The `LatestRates` always represents the latest content of hbase table `rates`.
-
-Then the content of `LatestRates` table when we query at time `10:15:00` is:
{% highlight sql %}
-10:15:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 114
-Yen 1
-{% endhighlight %}
-
-Then the content of `LatestRates` table when we query at time `11:00:00` is:
-{% highlight sql %}
-11:00:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 116
-Yen 1
+update_time product_id product_name price
+=========== ========== ============ =====
+12:00:00 p_001 scooter 12.99
+12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In Flink, this is represented by a [*regular Table*](#defining-regular-table).
-Temporal Table
---------------
-<span class="label label-danger">Attention</span> This is only supported in Blink planner.
+## Versioned Table Sources
-Flink uses primary key constraint and event time to define both versioned table and versioned view.
+Versioned tables are defined implicity for any table whose underlying source or format directly define changelogs. Examples include the [upsert Kafka]({% link dev/table/connectors/upsert-kafka.md %}) source as well as database changelog formats such as [debezium]({% link dev/table/connectors/formats/debezium.md %}) and [canal]({% link dev/table/connectors/formats/canal.md %}). As discussed above, the only additional requirement is the `CREATE` table statement must contain a primary key and event-time attribute.
-### Defining Versioned Table
-The table is a versioned table in Flink only is the table contains primary key constraint and event time.
{% highlight sql %}
--- Define a versioned table
-CREATE TABLE product_changelog (
- product_id STRING,
- product_name STRING,
- product_price DECIMAL(10, 4),
- update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
- PRIMARY KEY(product_id) NOT ENFORCED, -- (1) defines the primary key constraint
- WATERMARK FOR update_time AS update_time -- (2) defines the event time by watermark
-) WITH (
- 'connector' = 'kafka',
- 'topic' = 'products',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'value.format' = 'debezium-json'
-);
+CREATE TABLE products (
+ product_id STRING,
+ product_name STRING,
+ price DECIMAL(32, 2),
+ update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
+ PRIMARY KEY (product_id) NOT ENFORCED
+ WATERMARK FOR update_time AS update_time
+) WITH (...);
{% endhighlight %}
-Line `(1)` defines the primary key constraint for table `product_changelog`, Line `(2)` defines the `update_time` as event time for table `product_changelog`,
-thus table `product_changelog` is a versioned table.
+## Versioned Table Views
-**Note**: The grammar `METADATA FROM 'value.source.timestamp' VIRTUAL` means extract the database
-operation execution time for every changelog, it's strongly recommended defines the database operation execution time
-as event time rather than ingestion-time or time in the record, otherwise the version extract from the changelog may
-mismatch with the version in database.
-
-### Defining Versioned View
+Flink also supports defining versioned views if the underlying query contains a unique key constraint and event-time attribute. Image an append-only table of currency rates.
Review comment:
```suggestion
Flink also supports defining versioned views if the underlying query contains a unique key constraint and event-time attribute. Imagine an append-only table of currency rates.
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In above example, the specific version of the table is tracked by `update_time` and `product_id`, the `product_id` would be a primary key for `product_changelog` table and `update_time` would be the event time.
-
-In Flink, this is represented by a [*versioned table*](#defining-versioned-table).
-
-### Correlate with a regular table
-On the other hand, some use cases require to join a regular table which is an external database table.
+While at `13:00:00,` we found find another.
-Let's assume that `LatestRates` is a table (e.g. stored in HBase) which is materialized with the latest rates. The `LatestRates` always represents the latest content of hbase table `rates`.
-
-Then the content of `LatestRates` table when we query at time `10:15:00` is:
{% highlight sql %}
-10:15:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 114
-Yen 1
-{% endhighlight %}
-
-Then the content of `LatestRates` table when we query at time `11:00:00` is:
-{% highlight sql %}
-11:00:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 116
-Yen 1
+update_time product_id product_name price
+=========== ========== ============ =====
+12:00:00 p_001 scooter 12.99
+12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In Flink, this is represented by a [*regular Table*](#defining-regular-table).
-Temporal Table
---------------
-<span class="label label-danger">Attention</span> This is only supported in Blink planner.
+## Versioned Table Sources
-Flink uses primary key constraint and event time to define both versioned table and versioned view.
+Versioned tables are defined implicity for any table whose underlying source or format directly define changelogs. Examples include the [upsert Kafka]({% link dev/table/connectors/upsert-kafka.md %}) source as well as database changelog formats such as [debezium]({% link dev/table/connectors/formats/debezium.md %}) and [canal]({% link dev/table/connectors/formats/canal.md %}). As discussed above, the only additional requirement is the `CREATE` table statement must contain a primary key and event-time attribute.
Review comment:
```suggestion
Versioned tables are defined implicitly for any tables whose underlying sources or formats directly define changelogs. Examples include the [upsert Kafka]({% link dev/table/connectors/upsert-kafka.md %}) source as well as database changelog formats such as [debezium]({% link dev/table/connectors/formats/debezium.md %}) and [canal]({% link dev/table/connectors/formats/canal.md %}). As discussed above, the only additional requirement is the `CREATE` table statement must contain a primary key and an event-time attribute.
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In above example, the specific version of the table is tracked by `update_time` and `product_id`, the `product_id` would be a primary key for `product_changelog` table and `update_time` would be the event time.
-
-In Flink, this is represented by a [*versioned table*](#defining-versioned-table).
-
-### Correlate with a regular table
-On the other hand, some use cases require to join a regular table which is an external database table.
+While at `13:00:00,` we found find another.
-Let's assume that `LatestRates` is a table (e.g. stored in HBase) which is materialized with the latest rates. The `LatestRates` always represents the latest content of hbase table `rates`.
-
-Then the content of `LatestRates` table when we query at time `10:15:00` is:
{% highlight sql %}
-10:15:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 114
-Yen 1
-{% endhighlight %}
-
-Then the content of `LatestRates` table when we query at time `11:00:00` is:
-{% highlight sql %}
-11:00:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 116
-Yen 1
+update_time product_id product_name price
+=========== ========== ============ =====
+12:00:00 p_001 scooter 12.99
+12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In Flink, this is represented by a [*regular Table*](#defining-regular-table).
-Temporal Table
---------------
-<span class="label label-danger">Attention</span> This is only supported in Blink planner.
+## Versioned Table Sources
-Flink uses primary key constraint and event time to define both versioned table and versioned view.
+Versioned tables are defined implicity for any table whose underlying source or format directly define changelogs. Examples include the [upsert Kafka]({% link dev/table/connectors/upsert-kafka.md %}) source as well as database changelog formats such as [debezium]({% link dev/table/connectors/formats/debezium.md %}) and [canal]({% link dev/table/connectors/formats/canal.md %}). As discussed above, the only additional requirement is the `CREATE` table statement must contain a primary key and event-time attribute.
-### Defining Versioned Table
-The table is a versioned table in Flink only is the table contains primary key constraint and event time.
{% highlight sql %}
--- Define a versioned table
-CREATE TABLE product_changelog (
- product_id STRING,
- product_name STRING,
- product_price DECIMAL(10, 4),
- update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
- PRIMARY KEY(product_id) NOT ENFORCED, -- (1) defines the primary key constraint
- WATERMARK FOR update_time AS update_time -- (2) defines the event time by watermark
-) WITH (
- 'connector' = 'kafka',
- 'topic' = 'products',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'value.format' = 'debezium-json'
-);
+CREATE TABLE products (
+ product_id STRING,
+ product_name STRING,
+ price DECIMAL(32, 2),
+ update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
+ PRIMARY KEY (product_id) NOT ENFORCED
+ WATERMARK FOR update_time AS update_time
+) WITH (...);
{% endhighlight %}
-Line `(1)` defines the primary key constraint for table `product_changelog`, Line `(2)` defines the `update_time` as event time for table `product_changelog`,
-thus table `product_changelog` is a versioned table.
+## Versioned Table Views
-**Note**: The grammar `METADATA FROM 'value.source.timestamp' VIRTUAL` means extract the database
-operation execution time for every changelog, it's strongly recommended defines the database operation execution time
-as event time rather than ingestion-time or time in the record, otherwise the version extract from the changelog may
-mismatch with the version in database.
-
-### Defining Versioned View
+Flink also supports defining versioned views if the underlying query contains a unique key constraint and event-time attribute. Image an append-only table of currency rates.
-Flink also supports defining versioned view only if the view contains unique key constraint and event time.
-
-Let’s assume that we have the following table `RatesHistory`:
{% highlight sql %}
--- Define an append-only table
-CREATE TABLE RatesHistory (
- currency_time TIMESTAMP(3),
- currency STRING,
- rate DECIMAL(38, 10),
- WATERMARK FOR currency_time AS currency_time -- defines the event time
+CREATE TABLE currency_rates (
+ currency STRING,
+ rate DECIMAL(32, 10)
+ update_time TIMESTAMP(3),
+ WATERMARK FOR update_time AS update_time
) WITH (
- 'connector' = 'kafka',
- 'topic' = 'rates',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'format' = 'json' -- this is an append only source
-)
+ 'connector' = 'kafka',
+ 'topic' = 'rates',
+ 'properties.bootstrap.servers' = 'localhost:9092',
+ 'format' = 'json'
+);
{% endhighlight %}
-Table `RatesHistory` represents an ever growing append-only table of currency exchange rates with respect to
-Yen (which has a rate of 1). For example, the exchange rate for the period from 09:00 to 10:45 of Euro to Yen was 114.
-From 10:45 to 11:15 it was 116.
-
-{% highlight sql %}
-SELECT * FROM RatesHistory;
-
-currency_time currency rate
-============= ========= ====
-09:00:00 US Dollar 102
-09:00:00 Euro 114
-09:00:00 Yen 1
-10:45:00 Euro 116
-11:15:00 Euro 119
-11:49:00 Pounds 108
-{% endhighlight %}
+The table `currency_rates` contains a row for each currency - with respect to USD - and receives a new row each time the rate changes.
Review comment:
```suggestion
The table `currency_rates` contains a row for each currency — with respect to USD — and receives a new row each time the rate changes.
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -194,10 +127,10 @@ SELECT currency, rate, currency_time -- (1) `currency_time` keeps the
SELECT *,
ROW_NUMBER() OVER (PARTITION BY currency -- (2) the inferred unique key `currency` can be a primary key
ORDER BY currency_time DESC) AS rowNum
Review comment:
As mentioned above, `currency_time` should become `update time` throughout this view definition.
```suggestion
ORDER BY update_time DESC) AS rowNum
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In above example, the specific version of the table is tracked by `update_time` and `product_id`, the `product_id` would be a primary key for `product_changelog` table and `update_time` would be the event time.
-
-In Flink, this is represented by a [*versioned table*](#defining-versioned-table).
-
-### Correlate with a regular table
-On the other hand, some use cases require to join a regular table which is an external database table.
+While at `13:00:00,` we found find another.
Review comment:
```suggestion
While at `13:00:00,` we would find another set of prices.
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -194,10 +127,10 @@ SELECT currency, rate, currency_time -- (1) `currency_time` keeps the
SELECT *,
ROW_NUMBER() OVER (PARTITION BY currency -- (2) the inferred unique key `currency` can be a primary key
ORDER BY currency_time DESC) AS rowNum
- FROM RatesHistory)
+ FROM currency_rates)
WHERE rowNum = 1;
--- the view `versioned_rates` will produce changelog as the following.
+-- the view `versioned_rates` will produce a changelog as the following.
(changelog kind) currency_time currency rate
Review comment:
```suggestion
(changelog kind) update_time currency rate
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In above example, the specific version of the table is tracked by `update_time` and `product_id`, the `product_id` would be a primary key for `product_changelog` table and `update_time` would be the event time.
-
-In Flink, this is represented by a [*versioned table*](#defining-versioned-table).
-
-### Correlate with a regular table
-On the other hand, some use cases require to join a regular table which is an external database table.
+While at `13:00:00,` we found find another.
-Let's assume that `LatestRates` is a table (e.g. stored in HBase) which is materialized with the latest rates. The `LatestRates` always represents the latest content of hbase table `rates`.
-
-Then the content of `LatestRates` table when we query at time `10:15:00` is:
{% highlight sql %}
-10:15:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 114
-Yen 1
-{% endhighlight %}
-
-Then the content of `LatestRates` table when we query at time `11:00:00` is:
-{% highlight sql %}
-11:00:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 116
-Yen 1
+update_time product_id product_name price
+=========== ========== ============ =====
+12:00:00 p_001 scooter 12.99
+12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In Flink, this is represented by a [*regular Table*](#defining-regular-table).
-Temporal Table
---------------
-<span class="label label-danger">Attention</span> This is only supported in Blink planner.
+## Versioned Table Sources
-Flink uses primary key constraint and event time to define both versioned table and versioned view.
+Versioned tables are defined implicity for any table whose underlying source or format directly define changelogs. Examples include the [upsert Kafka]({% link dev/table/connectors/upsert-kafka.md %}) source as well as database changelog formats such as [debezium]({% link dev/table/connectors/formats/debezium.md %}) and [canal]({% link dev/table/connectors/formats/canal.md %}). As discussed above, the only additional requirement is the `CREATE` table statement must contain a primary key and event-time attribute.
-### Defining Versioned Table
-The table is a versioned table in Flink only is the table contains primary key constraint and event time.
{% highlight sql %}
--- Define a versioned table
-CREATE TABLE product_changelog (
- product_id STRING,
- product_name STRING,
- product_price DECIMAL(10, 4),
- update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
- PRIMARY KEY(product_id) NOT ENFORCED, -- (1) defines the primary key constraint
- WATERMARK FOR update_time AS update_time -- (2) defines the event time by watermark
-) WITH (
- 'connector' = 'kafka',
- 'topic' = 'products',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'value.format' = 'debezium-json'
-);
+CREATE TABLE products (
+ product_id STRING,
+ product_name STRING,
+ price DECIMAL(32, 2),
+ update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
+ PRIMARY KEY (product_id) NOT ENFORCED
+ WATERMARK FOR update_time AS update_time
+) WITH (...);
{% endhighlight %}
-Line `(1)` defines the primary key constraint for table `product_changelog`, Line `(2)` defines the `update_time` as event time for table `product_changelog`,
-thus table `product_changelog` is a versioned table.
+## Versioned Table Views
-**Note**: The grammar `METADATA FROM 'value.source.timestamp' VIRTUAL` means extract the database
-operation execution time for every changelog, it's strongly recommended defines the database operation execution time
-as event time rather than ingestion-time or time in the record, otherwise the version extract from the changelog may
-mismatch with the version in database.
-
-### Defining Versioned View
+Flink also supports defining versioned views if the underlying query contains a unique key constraint and event-time attribute. Image an append-only table of currency rates.
-Flink also supports defining versioned view only if the view contains unique key constraint and event time.
-
-Let’s assume that we have the following table `RatesHistory`:
{% highlight sql %}
--- Define an append-only table
-CREATE TABLE RatesHistory (
- currency_time TIMESTAMP(3),
- currency STRING,
- rate DECIMAL(38, 10),
- WATERMARK FOR currency_time AS currency_time -- defines the event time
+CREATE TABLE currency_rates (
+ currency STRING,
+ rate DECIMAL(32, 10)
+ update_time TIMESTAMP(3),
+ WATERMARK FOR update_time AS update_time
) WITH (
- 'connector' = 'kafka',
- 'topic' = 'rates',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'format' = 'json' -- this is an append only source
-)
+ 'connector' = 'kafka',
+ 'topic' = 'rates',
+ 'properties.bootstrap.servers' = 'localhost:9092',
+ 'format' = 'json'
+);
{% endhighlight %}
-Table `RatesHistory` represents an ever growing append-only table of currency exchange rates with respect to
-Yen (which has a rate of 1). For example, the exchange rate for the period from 09:00 to 10:45 of Euro to Yen was 114.
-From 10:45 to 11:15 it was 116.
-
-{% highlight sql %}
-SELECT * FROM RatesHistory;
-
-currency_time currency rate
-============= ========= ====
-09:00:00 US Dollar 102
-09:00:00 Euro 114
-09:00:00 Yen 1
-10:45:00 Euro 116
-11:15:00 Euro 119
-11:49:00 Pounds 108
-{% endhighlight %}
+The table `currency_rates` contains a row for each currency - with respect to USD - and receives a new row each time the rate changes.
+The `JSON` format does not support native changelog semantics, and so Flink can only read this table as append-only.
Review comment:
```suggestion
The `JSON` format does not support native changelog semantics, so Flink can only read this table as append-only.
```
##########
File path: docs/dev/table/streaming/temporal_tables.md
##########
@@ -57,134 +56,68 @@ SELECT * FROM product_changelog;
-(DELETE) 18:00:00 p_001 scooter 12.99
{% endhighlight %}
-The table `product_changelog` represents an ever growing changelog of database table `products`, for example, the initial price of product `scooter` is `11.11` at `00:01:00`, and the price increases to `12.99` at `12:00:00`,
- the product item is deleted from the table `products` at `18:00:00`.
+Given this set of changes, we track how the price of a scooter changes over time.
+It is initially $11.11 at `00:01:00` when added to the catalog.
+The price then rises to $12.99 at `12:00:00` before being deleted from the catalog at `18:00:00`.
-Given that we would like to output the version of `product_changelog` table of the time `10:00:00`, the following table shows the result.
-{% highlight sql %}
-update_time product_id product_name price
-=========== ========== ============ =====
-00:01:00 p_001 scooter 11.11
-00:02:00 p_002 basketball 23.11
-{% endhighlight %}
+If we queried the table for various products' prices at different times, we would retrieve different results. At `10:00:00` the table would show one set of prices.
-Given that we would like to output the version of `product_changelog` table of the time `13:00:00`, the following table shows the result.
{% highlight sql %}
update_time product_id product_name price
=========== ========== ============ =====
12:00:00 p_001 scooter 12.99
12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In above example, the specific version of the table is tracked by `update_time` and `product_id`, the `product_id` would be a primary key for `product_changelog` table and `update_time` would be the event time.
-
-In Flink, this is represented by a [*versioned table*](#defining-versioned-table).
-
-### Correlate with a regular table
-On the other hand, some use cases require to join a regular table which is an external database table.
+While at `13:00:00,` we found find another.
-Let's assume that `LatestRates` is a table (e.g. stored in HBase) which is materialized with the latest rates. The `LatestRates` always represents the latest content of hbase table `rates`.
-
-Then the content of `LatestRates` table when we query at time `10:15:00` is:
{% highlight sql %}
-10:15:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 114
-Yen 1
-{% endhighlight %}
-
-Then the content of `LatestRates` table when we query at time `11:00:00` is:
-{% highlight sql %}
-11:00:00 > SELECT * FROM LatestRates;
-
-currency rate
-========= ====
-US Dollar 102
-Euro 116
-Yen 1
+update_time product_id product_name price
+=========== ========== ============ =====
+12:00:00 p_001 scooter 12.99
+12:00:00 p_002 basketball 19.99
{% endhighlight %}
-In Flink, this is represented by a [*regular Table*](#defining-regular-table).
-Temporal Table
---------------
-<span class="label label-danger">Attention</span> This is only supported in Blink planner.
+## Versioned Table Sources
-Flink uses primary key constraint and event time to define both versioned table and versioned view.
+Versioned tables are defined implicity for any table whose underlying source or format directly define changelogs. Examples include the [upsert Kafka]({% link dev/table/connectors/upsert-kafka.md %}) source as well as database changelog formats such as [debezium]({% link dev/table/connectors/formats/debezium.md %}) and [canal]({% link dev/table/connectors/formats/canal.md %}). As discussed above, the only additional requirement is the `CREATE` table statement must contain a primary key and event-time attribute.
-### Defining Versioned Table
-The table is a versioned table in Flink only is the table contains primary key constraint and event time.
{% highlight sql %}
--- Define a versioned table
-CREATE TABLE product_changelog (
- product_id STRING,
- product_name STRING,
- product_price DECIMAL(10, 4),
- update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
- PRIMARY KEY(product_id) NOT ENFORCED, -- (1) defines the primary key constraint
- WATERMARK FOR update_time AS update_time -- (2) defines the event time by watermark
-) WITH (
- 'connector' = 'kafka',
- 'topic' = 'products',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'value.format' = 'debezium-json'
-);
+CREATE TABLE products (
+ product_id STRING,
+ product_name STRING,
+ price DECIMAL(32, 2),
+ update_time TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,
+ PRIMARY KEY (product_id) NOT ENFORCED
+ WATERMARK FOR update_time AS update_time
+) WITH (...);
{% endhighlight %}
-Line `(1)` defines the primary key constraint for table `product_changelog`, Line `(2)` defines the `update_time` as event time for table `product_changelog`,
-thus table `product_changelog` is a versioned table.
+## Versioned Table Views
-**Note**: The grammar `METADATA FROM 'value.source.timestamp' VIRTUAL` means extract the database
-operation execution time for every changelog, it's strongly recommended defines the database operation execution time
-as event time rather than ingestion-time or time in the record, otherwise the version extract from the changelog may
-mismatch with the version in database.
-
-### Defining Versioned View
+Flink also supports defining versioned views if the underlying query contains a unique key constraint and event-time attribute. Image an append-only table of currency rates.
-Flink also supports defining versioned view only if the view contains unique key constraint and event time.
-
-Let’s assume that we have the following table `RatesHistory`:
{% highlight sql %}
--- Define an append-only table
-CREATE TABLE RatesHistory (
- currency_time TIMESTAMP(3),
- currency STRING,
- rate DECIMAL(38, 10),
- WATERMARK FOR currency_time AS currency_time -- defines the event time
+CREATE TABLE currency_rates (
+ currency STRING,
+ rate DECIMAL(32, 10)
+ update_time TIMESTAMP(3),
+ WATERMARK FOR update_time AS update_time
) WITH (
- 'connector' = 'kafka',
- 'topic' = 'rates',
- 'scan.startup.mode' = 'earliest-offset',
- 'properties.bootstrap.servers' = 'localhost:9092',
- 'format' = 'json' -- this is an append only source
-)
+ 'connector' = 'kafka',
+ 'topic' = 'rates',
+ 'properties.bootstrap.servers' = 'localhost:9092',
+ 'format' = 'json'
+);
{% endhighlight %}
-Table `RatesHistory` represents an ever growing append-only table of currency exchange rates with respect to
-Yen (which has a rate of 1). For example, the exchange rate for the period from 09:00 to 10:45 of Euro to Yen was 114.
-From 10:45 to 11:15 it was 116.
-
-{% highlight sql %}
-SELECT * FROM RatesHistory;
-
-currency_time currency rate
-============= ========= ====
-09:00:00 US Dollar 102
-09:00:00 Euro 114
-09:00:00 Yen 1
-10:45:00 Euro 116
-11:15:00 Euro 119
-11:49:00 Pounds 108
-{% endhighlight %}
+The table `currency_rates` contains a row for each currency - with respect to USD - and receives a new row each time the rate changes.
+The `JSON` format does not support native changelog semantics, and so Flink can only read this table as append-only.
+However, it is clear to us (the query developer) that this table has all the necessary information to define a versioned table.
-To define a versioned table on `RatesHistory`, Flink supports defining a versioned view
-by [deduplication query]({% link dev/table/sql/queries.md %}#deduplication) which produces an ordered changelog
-stream with an inferred primary key(`currency`) and event time(`currency_time`).
+Flink can reinterpret this table as a versioned table by defining a [deduplication query]({% link dev/table/sql/queries.md %}#deduplication) which produces an ordered changelog stream with an inferred primary key (currency) and event time (update_time).
Review comment:
In the view defined below, `currency_time` must become `update_time` in both the SELECT and in the comment. github won't allow me to suggest changes here, unfortunately.
Also, I'm confused by one thing. I see how `currency` can be the primary key that a versioned table needs, but how does this actually work without our explicitly declaring that `currency` is the PRIMARY KEY for this view? Why do we specify a primary key when using CREATE TABLE, but not when using CREATE VIEW?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bd3e853fd1bd205632bd8c22b82f621fb031a0d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd2905f71cb565e5fd513217a056fa2d757f84e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461)
* 6bd3e853fd1bd205632bd8c22b82f621fb031a0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d1723792f216e5c11cad7cfa8d8f1d58e5c842a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508)
* 6ca7d5619e4d14b3da3cca362da015785ba199eb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6ca7d5619e4d14b3da3cca362da015785ba199eb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532)
* 69b091b21c93f13cf8e999367791f9a2f8ac2692 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10661)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #14292:
URL: https://github.com/apache/flink/pull/14292#discussion_r539660475
##########
File path: docs/redirects/temporal_table.md
##########
@@ -0,0 +1,24 @@
+---
+title: Temporal Tables
+layout: redirect
+redirect: /dev/table/streaming/versioned_table.html
Review comment:
- Now that you say it, I think legacy makes more sense. That's the content people are expecting to find. And that page already cross-links to versioned_tables.
- The `zh` redirect happens automatically
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dd2905f71cb565e5fd513217a056fa2d757f84e7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461)
* 6bd3e853fd1bd205632bd8c22b82f621fb031a0d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737644985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10461",
"triggerID" : "dd2905f71cb565e5fd513217a056fa2d757f84e7",
"triggerType" : "PUSH"
}, {
"hash" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10506",
"triggerID" : "6bd3e853fd1bd205632bd8c22b82f621fb031a0d",
"triggerType" : "PUSH"
}, {
"hash" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10508",
"triggerID" : "7d1723792f216e5c11cad7cfa8d8f1d58e5c842a",
"triggerType" : "PUSH"
}, {
"hash" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532",
"triggerID" : "6ca7d5619e4d14b3da3cca362da015785ba199eb",
"triggerType" : "PUSH"
}, {
"hash" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "69b091b21c93f13cf8e999367791f9a2f8ac2692",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6ca7d5619e4d14b3da3cca362da015785ba199eb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10532)
* 69b091b21c93f13cf8e999367791f9a2f8ac2692 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-737641638
@twalthr @wuchong @morsapaes @alpinegizmo @knaufk
I think all of you would have good opinions on these changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #14292: [FLINK-20456][docs] Make streaming SQL concepts more approachable
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on pull request #14292:
URL: https://github.com/apache/flink/pull/14292#issuecomment-740793531
@alpinegizmo thank you for your patience
I've updated the versioned view section.
I also moved temporal table functions to a new "legacy" page and rewrote the example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org