You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by al...@apache.org on 2016/06/07 08:45:13 UTC
flink git commit: [FLINK-4026] Fix code, grammar,
and link issues in the Streaming documentation
Repository: flink
Updated Branches:
refs/heads/master 61d69a229 -> 9c561fb48
[FLINK-4026] Fix code, grammar, and link issues in the Streaming documentation
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/9c561fb4
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/9c561fb4
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/9c561fb4
Branch: refs/heads/master
Commit: 9c561fb4825d202cefbe382df36abce7a6d825b3
Parents: 61d69a2
Author: Dyana Rose <dy...@gmail.com>
Authored: Wed May 18 22:14:15 2016 +0100
Committer: Dyana Rose <dy...@salecycle.com>
Committed: Mon Jun 6 16:21:46 2016 +0100
----------------------------------------------------------------------
docs/apis/streaming/connectors/index.md | 1 +
docs/apis/streaming/connectors/kinesis.md | 12 ++++++------
docs/apis/streaming/connectors/twitter.md | 6 +++---
docs/apis/streaming/event_time.md | 8 ++++----
docs/apis/streaming/event_timestamp_extractors.md | 16 ++++++++--------
docs/apis/streaming/event_timestamps_watermarks.md | 12 ++++++------
docs/apis/streaming/fault_tolerance.md | 2 +-
docs/apis/streaming/index.md | 4 ++--
docs/apis/streaming/state_backends.md | 4 ++--
docs/apis/streaming/windows.md | 14 +++++++-------
10 files changed, 40 insertions(+), 39 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/connectors/index.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/connectors/index.md b/docs/apis/streaming/connectors/index.md
index c62a2ca..20d6e41 100644
--- a/docs/apis/streaming/connectors/index.md
+++ b/docs/apis/streaming/connectors/index.md
@@ -35,6 +35,7 @@ Currently these systems are supported:
* [Elasticsearch 2x](https://elastic.com) (sink)
* [Hadoop FileSystem](http://hadoop.apache.org) (sink)
* [RabbitMQ](http://www.rabbitmq.com/) (sink/source)
+ * [Amazon Kinesis Streams](http://aws.amazon.com/kinesis/streams/) (sink/source)
* [Twitter Streaming API](https://dev.twitter.com/docs/streaming-apis) (source)
To run an application using one of these connectors, additional third party
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/connectors/kinesis.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/connectors/kinesis.md b/docs/apis/streaming/connectors/kinesis.md
index 923b898..66c078a 100644
--- a/docs/apis/streaming/connectors/kinesis.md
+++ b/docs/apis/streaming/connectors/kinesis.md
@@ -26,7 +26,7 @@ specific language governing permissions and limitations
under the License.
-->
-The Kinesis connector allows to produce data into an [Amazon AWS Kinesis Stream](http://aws.amazon.com/kinesis/streams/).
+The Kinesis connector provides access to [Amazon AWS Kinesis Streams](http://aws.amazon.com/kinesis/streams/).
To use the connector, add the following Maven dependency to your project:
@@ -57,7 +57,7 @@ See linking with them for cluster execution [here]({{site.baseurl}}/apis/cluster
#### Usage of Consumer
The `FlinkKinesisConsumer` can be used to pull data from multiple Kinesis streams within the same AWS region in parallel.
-It participates with Flink's distributed snapshot checkpointing and provides exactly-once processing guarantees. Note
+It participates in Flink's distributed snapshot checkpointing and provides exactly-once processing guarantees. Note
that the current version can not handle resharding of Kinesis streams. When Kinesis streams are resharded, the consumer
will fail and the Flink streaming job must be resubmitted.
@@ -85,8 +85,8 @@ DataStream<String> kinesisRecords = env.addSource(new FlinkKinesisConsumer<>(
</div>
The above is a simple example of using the consumer. Configuration for the consumer is supplied with a `java.util.Properties`
-instance, with which the configuration setting keys used can be found in `KinesisConfigConstants`. The example
-demonstrates consuming a single Kinesis stream in the AWS region "us-east-1". The AWS credentials is supplied using the basic method in which
+instance, the setting keys for which are enumerated in `KinesisConfigConstants`. The example
+demonstrates consuming a single Kinesis stream in the AWS region "us-east-1". The AWS credentials are supplied using the basic method in which
the AWS access key ID and secret key are directly supplied in the configuration (other options are setting
`KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_TYPE` to `ENV_VAR`, `SYS_PROP`, and `PROFILE`). Also, data is being consumed
from the newest position in the Kinesis stream (the other option will be setting `KinesisConfigConstants.CONFIG_STREAM_INIT_POSITION_TYPE`
@@ -94,11 +94,11 @@ to `TRIM_HORIZON`, which lets the consumer start reading the Kinesis stream from
#### Usage of Producer
-The `FlinkKinesisProducer` is used for sending data from a Flink stream into a Kinesis stream. Note that the producer is not participating in
+The `FlinkKinesisProducer` is used for putting data from a Flink stream onto a Kinesis stream. Note that the producer is not participating in
Flink's checkpointing and doesn't provide exactly-once processing guarantees. In case of a failure, data will be written again
to Kinesis, leading to duplicates. This behavior is usually called "at-least-once" semantics.
-To produce data into a Kinesis stream, make sure that you have a stream created with the status "ACTIVE" in the AWS dashboard.
+To put data onto a Kinesis stream, make sure the stream is marked as "ACTIVE" in the AWS dashboard.
For the monitoring to work, the user accessing the stream needs access to the Cloud watch service.
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/connectors/twitter.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/connectors/twitter.md b/docs/apis/streaming/connectors/twitter.md
index b896022..9e84481 100644
--- a/docs/apis/streaming/connectors/twitter.md
+++ b/docs/apis/streaming/connectors/twitter.md
@@ -26,7 +26,7 @@ specific language governing permissions and limitations
under the License.
-->
-Twitter Streaming API allows to connect to the stream of tweets made available by Twitter.
+The Twitter Streaming API provides access to the stream of tweets made available by Twitter.
Flink Streaming comes with a built-in `TwitterSource` class for establishing a connection to this stream.
To use this connector, add the following dependency to your project:
@@ -42,13 +42,13 @@ Note that the streaming connectors are currently not part of the binary distribu
See linking with them for cluster execution [here]({{site.baseurl}}/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution).
#### Authentication
-In order to connect to Twitter stream the user has to register their program and acquire the necessary information for the authentication. The process is described below.
+In order to connect to the Twitter stream the user has to register their program and acquire the necessary information for the authentication. The process is described below.
#### Acquiring the authentication information
First of all, a Twitter account is needed. Sign up for free at [twitter.com/signup](https://twitter.com/signup)
or sign in at Twitter's [Application Management](https://apps.twitter.com/) and register the application by
clicking on the "Create New App" button. Fill out a form about your program and accept the Terms and Conditions.
-After selecting the application, the API key and API secret (called `twitter-source.consumerKey` and `twitter-source.consumerSecret` in `TwitterSource` respectively) is located on the "API Keys" tab.
+After selecting the application, the API key and API secret (called `twitter-source.consumerKey` and `twitter-source.consumerSecret` in `TwitterSource` respectively) are located on the "API Keys" tab.
The necessary OAuth Access Token data (`twitter-source.token` and `twitter-source.tokenSecret` in `TwitterSource`) can be generated and acquired on the "Keys and Access Tokens" tab.
Remember to keep these pieces of information secret and do not push them to public repositories.
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/event_time.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/event_time.md b/docs/apis/streaming/event_time.md
index 7bc53e7..7f94d68 100644
--- a/docs/apis/streaming/event_time.md
+++ b/docs/apis/streaming/event_time.md
@@ -62,7 +62,7 @@ Flink supports different notions of *time* in streaming programs.
*processing time* operations.
- **Ingestion time:** Ingestion time is the time that events enter Flink. At the source operator, each
- records gets the source's current time as a timestamp, and time-based operations (like time windows)
+ record gets the source's current time as a timestamp, and time-based operations (like time windows)
refer to that timestamp.
*Ingestion Time* sits conceptually in between *Event Time* and *Processing Time*. Compared to
@@ -135,7 +135,7 @@ Note that in order to run this example in *Event Time*, the program needs to use
source, or inject a *Timestamp Assigner & Watermark Generator*. Those functions describe how to access
the event timestamps, and what timely out-of-orderness the event stream exhibits.
-The section below describes the general mechanism behind *Timestamps* and *Watermarks*. For a guide how
+The section below describes the general mechanism behind *Timestamps* and *Watermarks*. For a guide on how
to use timestamp assignment and watermark generation in the Flink DataStream API, please refer to
[Generating Timestamps / Watermarks]({{ site.baseurl }}/apis/streaming/event_timestamps_watermarks.html)
@@ -160,7 +160,7 @@ can progress by weeks in seconds.
------
-The mechanism in Flink to measure progress in event time are **Watermarks**.
+The mechanism in Flink to measure progress in event time is **Watermarks**.
Watermarks flow as part of the data stream and carry a timestamp *t*. A *Watermark(t)* declares that event time has reached time
*t* in that stream, meaning that all events with a timestamps *t' < t* have occurred.
@@ -197,7 +197,7 @@ The figure below shows an example of events and watermarks flowing through paral
It is possible that certain elements violate the watermark condition, meaning that even after the *Watermark(t)* has occurred,
more elements with timestamp *t' < t* will occur. In fact, in many real world setups, certain elements can be arbitrarily
-delayed, which it is impossible to define a time when all elements of a certain event timestamp have occurred.
+delayed, making it impossible to define a time when all elements of a certain event timestamp have occurred.
Further more, even if the lateness can be bounded, delaying the watermarks by too much is often not desirable, because it delays
the evaluation of the event time windows by too much.
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/event_timestamp_extractors.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/event_timestamp_extractors.md b/docs/apis/streaming/event_timestamp_extractors.md
index 06fb01b..83a90d2 100644
--- a/docs/apis/streaming/event_timestamp_extractors.md
+++ b/docs/apis/streaming/event_timestamp_extractors.md
@@ -2,7 +2,7 @@
title: "Pre-defined Timestamp Extractors / Watermark Emitters"
sub-nav-group: streaming
-sub-nav-pos: 1
+sub-nav-pos: 2
sub-nav-parent: eventtime
---
<!--
@@ -27,7 +27,7 @@ under the License.
* toc
{:toc}
-As described in the [timestamps and watermark handling]({{ site.baseurl }}/apis/streaming/event_timestamps_watermarks.html) page,
+As described in [timestamps and watermark handling]({{ site.baseurl }}/apis/streaming/event_timestamps_watermarks.html),
Flink provides abstractions that allow the programmer to assign their own timestamps and emit their own watermarks. More specifically,
one can do so by implementing one of the `AssignerWithPeriodicWatermarks` and `AssignerWithPunctuatedWatermarks` interfaces, depending
on their use-case. In a nutshell, the first will emit watermarks periodically, while the second does so based on some property of
@@ -40,8 +40,8 @@ for custom assigner implementations.
#### **Assigner with Ascending Timestamps**
The simplest special case for *periodic* watermark generation is the case where timestamps seen by a given source task
-occur in ascending order. In that case, the current timestamp can always act as a watermark, because no lower timestamps will
-occur any more.
+occur in ascending order. In that case, the current timestamp can always act as a watermark, because no earlier timestamps will
+arrive.
Note that it is only necessary that timestamps are ascending *per parallel data source task*. For example, if
in a specific setup one Kafka partition is read by one parallel data source instance, then it is only necessary that
@@ -74,11 +74,11 @@ val withTimestampsAndWatermarks = stream.assignAscendingTimestamps( _.getCreatio
#### **Assigner which allows a fixed amount of record lateness**
-Another example of periodic watermark generation is the one where the watermark lags behind the maximum (event-time) timestamp
-seen in the stream, by a fixed amount of time. This case covers scenarios where the maximum lateness that can be encountered in a
+Another example of periodic watermark generation is when the watermark lags behind the maximum (event-time) timestamp
+seen in the stream by a fixed amount of time. This case covers scenarios where the maximum lateness that can be encountered in a
stream is known in advance, e.g. when creating a custom source containing elements with timestamps spread within a fixed period of
-time for testing. For these cases, Flink provides the `BoundedOutOfOrdernessTimestampExtractor` which takes as argument
-the `maxOutOfOrderness`, i.e. the maximum amount of time an element is allowed to be late, before being ignored when computing the
+time for testing. For these cases, Flink provides the `BoundedOutOfOrdernessTimestampExtractor` which takes as an argument
+the `maxOutOfOrderness`, i.e. the maximum amount of time an element is allowed to be late before being ignored when computing the
final result for the given window. Lateness corresponds to the result of `t - t_w`, where `t` is the (event-time) timestamp of an
element, and `t_w` that of the previous watermark. If `lateness > 0` then the element is considered late and is ignored when computing
the result of the job for its corresponding window.
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/event_timestamps_watermarks.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/event_timestamps_watermarks.md b/docs/apis/streaming/event_timestamps_watermarks.md
index 0ec2810..fad84a6 100644
--- a/docs/apis/streaming/event_timestamps_watermarks.md
+++ b/docs/apis/streaming/event_timestamps_watermarks.md
@@ -114,11 +114,11 @@ those timestamps will be overwritten by the TimestampAssigner. Similarly, Waterm
### Timestamp Assigners / Watermark Generators
Timestamp Assigners take a stream and produce a new stream with timestamped elements and watermarks. If the
-original stream had timestamps or watermarks already, the timestamp assigner overwrites those.
+original stream had timestamps and/or watermarks already, the timestamp assigner overwrites them.
-The timestamp assigners occur usually immediately after the data source, but it is not strictly required to. A
-common pattern is for example to parse (*MapFunction*) and filter (*FilterFunction*) before the timestamp assigner.
-In any case, the timestamp assigner needs to occur before the first operation on event time
+The timestamp assigners usually are specified immediately after the data source, but it is not strictly required to do so. A
+common pattern is, for example, to parse (*MapFunction*) and filter (*FilterFunction*) before the timestamp assigner.
+In any case, the timestamp assigner needs to be specified before the first operation on event time
(such as the first window operation).
**NOTE:** The remainder of this section presents the main interfaces a programmer has
@@ -168,8 +168,8 @@ withTimestampsAndWatermarks
#### **With Periodic Watermarks**
-The `AssignerWithPeriodicWatermarks` assigns timestamps and generate watermarks periodically (possibly depending
-the stream elements, or purely based on processing time).
+The `AssignerWithPeriodicWatermarks` assigns timestamps and generates watermarks periodically (possibly depending
+on the stream elements, or purely based on processing time).
The interval (every *n* milliseconds) in which the watermark will be generated is defined via
`ExecutionConfig.setAutoWatermarkInterval(...)`. Each time, the assigner's `getCurrentWatermark()` method will be
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/fault_tolerance.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/fault_tolerance.md b/docs/apis/streaming/fault_tolerance.md
index f6e90a4..7f861fc 100644
--- a/docs/apis/streaming/fault_tolerance.md
+++ b/docs/apis/streaming/fault_tolerance.md
@@ -55,7 +55,7 @@ Other parameters for checkpointing include:
- *number of concurrent checkpoints*: By default, the system will not trigger another checkpoint while one is still in progress. This ensures that the topology does not spend too much time on checkpoints and not make progress with processing the streams. It is possible to allow for multiple overlapping checkpoints, which is interesting for pipelines that have a certain processing delay (for example because the functions call external services that need some time to respond) but that still want to do very frequent checkpoints (100s of milliseconds) to re-process very little upon failures.
-- *checkpoint timeout*: The time after which a checkpoint-in-progress is aborted, if it did not complete until then.
+- *checkpoint timeout*: The time after which a checkpoint-in-progress is aborted, if it did not complete by then.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/index.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/index.md b/docs/apis/streaming/index.md
index 9762962..acc6e9a 100644
--- a/docs/apis/streaming/index.md
+++ b/docs/apis/streaming/index.md
@@ -545,7 +545,7 @@ DataStream<Long> output = iterationBody.filter(new FilterFunction<Long>(){
<td>
<p>
Extracts timestamps from records in order to work with windows
- that use event time semantics. See <a href="{{ site.baseurl }}/apis/streaming/event_time.html">working with time</a>.
+ that use event time semantics. See <a href="{{ site.baseurl }}/apis/streaming/event_time.html">Event Time</a>.
{% highlight java %}
stream.assignTimestamps (new TimeStampExtractor() {...});
{% endhighlight %}
@@ -863,7 +863,7 @@ initialStream.iterate {
<p>
Extracts timestamps from records in order to work with windows
that use event time semantics.
- See <a href="{{ site.baseurl }}/apis/streaming/event_time.html">working with time</a>.
+ See <a href="{{ site.baseurl }}/apis/streaming/event_time.html">Event Time</a>.
{% highlight scala %}
stream.assignTimestamps { timestampExtractor }
{% endhighlight %}
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/state_backends.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/state_backends.md b/docs/apis/streaming/state_backends.md
index f9471dd..027148a 100644
--- a/docs/apis/streaming/state_backends.md
+++ b/docs/apis/streaming/state_backends.md
@@ -71,7 +71,7 @@ The MemoryStateBackend is encouraged for:
### The FsStateBackend
-The *FsStateBackend* is configured with a file system URL (type, address, path), such as for example "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
+The *FsStateBackend* is configured with a file system URL (type, address, path), such as "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
The FsStateBackend holds in-flight data in the TaskManager's memory. Upon checkpointing, it writes state snapshots into files in the configured file system and directory. Minimal metadata is stored in the JobManager's memory (or, in high-availability mode, in the metadata checkpoint).
@@ -82,7 +82,7 @@ The FsStateBackend is encouraged for:
### The RocksDBStateBackend
-The *RocksDBStateBackend* is configured with a file system URL (type, address, path), such as for example "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
+The *RocksDBStateBackend* is configured with a file system URL (type, address, path), such as "hdfs://namenode:40010/flink/checkpoints" or "file:///data/flink/checkpoints".
The RocksDBStateBackend holds in-flight data in a [RocksDB](http://rocksdb.org) data base
that is (per default) stored in the TaskManager data directories. Upon checkpointing, the whole
http://git-wip-us.apache.org/repos/asf/flink/blob/9c561fb4/docs/apis/streaming/windows.md
----------------------------------------------------------------------
diff --git a/docs/apis/streaming/windows.md b/docs/apis/streaming/windows.md
index 8cf5ddd..90ad0de 100644
--- a/docs/apis/streaming/windows.md
+++ b/docs/apis/streaming/windows.md
@@ -68,7 +68,7 @@ keyedStream.timeWindow(Time.seconds(5));
<td><strong>Sliding time window</strong><br>KeyedStream → WindowedStream</td>
<td>
<p>
- Defines a window of 5 seconds, that "slides" by 1 seconds. This means that elements are
+ Defines a window of 5 seconds, that "slides" by 1 second. This means that elements are
grouped according to their timestamp in groups of 5 second duration, and elements can belong to more than
one window (since windows overlap by at most 4 seconds)
The notion of time is specified by the selected TimeCharacteristic (see <a href="{{ site.baseurl }}/apis/streaming/event_time.html">time</a>).
@@ -138,7 +138,7 @@ keyedStream.timeWindow(Time.seconds(5))
<td><strong>Sliding time window</strong><br>KeyedStream → WindowedStream</td>
<td>
<p>
- Defines a window of 5 seconds, that "slides" by 1 seconds. This means that elements are
+ Defines a window of 5 seconds, that "slides" by 1 second. This means that elements are
grouped according to their timestamp in groups of 5 second duration, and elements can belong to more than
one window (since windows overlap by at most 4 seconds)
The notion of time is specified by the selected TimeCharacteristic (see <a href="{{ site.baseurl }}/apis/streaming/event_time.html">time</a>).
@@ -708,7 +708,7 @@ triggeredStream.evictor(CountEvictor.of(1000));
</p>
{% highlight java %}
triggeredStream.evictor(DeltaEvictor.of(5000, new DeltaFunction<Double>() {
- public double (Double oldValue, Double newValue) {
+ public double getDelta (Double oldValue, Double newValue) {
return newValue - oldValue;
}
}));
@@ -823,7 +823,7 @@ stream.timeWindow(Time.seconds(5))
</td>
<td>
{% highlight java %}
-stream.window(TumblingEventTimeWindows.of((Time.seconds(5)))
+stream.window(TumblingEventTimeWindows.of(Time.seconds(5))
.trigger(EventTimeTrigger.create())
{% endhighlight %}
</td>
@@ -851,7 +851,7 @@ stream.timeWindow(Time.seconds(5))
</td>
<td>
{% highlight java %}
-stream.window(TumblingProcessingTimeWindows.of((Time.seconds(5)))
+stream.window(TumblingProcessingTimeWindows.of(Time.seconds(5))
.trigger(ProcessingTimeTrigger.create())
{% endhighlight %}
</td>
@@ -933,7 +933,7 @@ nonKeyedStream.timeWindowAll(Time.seconds(5));
<td><strong>Sliding time window all</strong><br>DataStream → WindowedStream</td>
<td>
<p>
- Defines a window of 5 seconds, that "slides" by 1 seconds. This means that elements are
+ Defines a window of 5 seconds, that "slides" by 1 second. This means that elements are
grouped according to their timestamp in groups of 5 second duration, and elements can belong to more than
one window (since windows overlap by at least 4 seconds)
The notion of time used is controlled by the StreamExecutionEnvironment.
@@ -1003,7 +1003,7 @@ nonKeyedStream.timeWindowAll(Time.seconds(5));
<td><strong>Sliding time window all</strong><br>DataStream → WindowedStream</td>
<td>
<p>
- Defines a window of 5 seconds, that "slides" by 1 seconds. This means that elements are
+ Defines a window of 5 seconds, that "slides" by 1 second. This means that elements are
grouped according to their timestamp in groups of 5 second duration, and elements can belong to more than
one window (since windows overlap by at least 4 seconds)
The notion of time used is controlled by the StreamExecutionEnvironment.