You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/08/20 21:40:45 UTC
[beam-site] 08/11: Update doc links in blog post to point to latest
release
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 645574c9dcb59afe503702381781d6afdfc2b673
Author: Julien Phalip <jp...@gmail.com>
AuthorDate: Wed Aug 15 10:17:50 2018 -0700
Update doc links in blog post to point to latest release
---
...2018-08-16-review-input-streaming-connectors.md | 38 +++++++++++-----------
1 file changed, 19 insertions(+), 19 deletions(-)
diff --git a/src/_posts/2018-08-16-review-input-streaming-connectors.md b/src/_posts/2018-08-16-review-input-streaming-connectors.md
index 2b69a41..72983b8 100644
--- a/src/_posts/2018-08-16-review-input-streaming-connectors.md
+++ b/src/_posts/2018-08-16-review-input-streaming-connectors.md
@@ -9,13 +9,13 @@ authors:
- jphalip
---
-In this post, you'll learn about the current state of support for input streaming connectors in [Apache Beam](https://beam.apache.org/). For more context, you'll also learn about the corresponding state of support in [Apache Spark](https://spark.apache.org/).<!--more-->
+In this post, you'll learn about the current state of support for input streaming connectors in [Apache Beam]({{ site.baseurl }}/). For more context, you'll also learn about the corresponding state of support in [Apache Spark](https://spark.apache.org/).<!--more-->
With batch processing, you might load data from any source, including a database system. Even if there are no specific SDKs available for those database systems, you can often resort to using a [JDBC](https://en.wikipedia.org/wiki/Java_Database_Connectivity) driver. With streaming, implementing a proper data pipeline is arguably more challenging as generally fewer source types are available. For that reason, this article particularly focuses on the streaming use case.
## Connectors for Java
-Beam has an official [Java SDK](https://beam.apache.org/documentation/sdks/java/) and has several execution engines, called [runners](https://beam.apache.org/documentation/runners/capability-matrix/). In most cases it is fairly easy to transfer existing Beam pipelines written in Java or Scala to a Spark environment by using the [Spark Runner](https://beam.apache.org/documentation/runners/spark/).
+Beam has an official [Java SDK]({{ site.baseurl }}/documentation/sdks/java/) and has several execution engines, called [runners]({{ site.baseurl }}/documentation/runners/capability-matrix/). In most cases it is fairly easy to transfer existing Beam pipelines written in Java or Scala to a Spark environment by using the [Spark Runner]({{ site.baseurl }}/documentation/runners/spark/).
Spark is written in Scala and has a [Java API](https://spark.apache.org/docs/latest/api/java/). Spark's source code compiles to [Java bytecode](https://en.wikipedia.org/wiki/Java_(programming_language)#Java_JVM_and_Bytecode) and the binaries are run by a [Java Virtual Machine](https://en.wikipedia.org/wiki/Java_virtual_machine). Scala code is interoperable with Java and therefore has native compatibility with Java libraries (and vice versa).
@@ -41,7 +41,7 @@ Below are the main streaming input connectors for available for Beam and Spark D
</td>
<td>Local<br>(Using the <code>file://</code> URI)
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/TextIO.html">TextIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/StreamingContext.html#textFileStream-java.lang.String-">textFileStream</a><br>(Spark treats most Unix systems as HDFS-compatible, but the location should be accessible from all nodes)
</td>
@@ -49,7 +49,7 @@ Below are the main streaming input connectors for available for Beam and Spark D
<tr>
<td>HDFS<br>(Using the <code>hdfs://</code> URI)
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.html">HadoopFileSystemOptions</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/util/HdfsUtils.html">HdfsUtils</a>
</td>
@@ -59,7 +59,7 @@ Below are the main streaming input connectors for available for Beam and Spark D
</td>
<td>Cloud Storage<br>(Using the <code>gs://</code> URI)
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.html">GcsOptions</a>
</td>
<td rowspan="2" ><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a>
<p>
@@ -69,7 +69,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>S3<br>(Using the <code>s3://</code> URI)
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/FileIO.html">FileIO</a> + <a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/aws/options/S3Options.html">S3Options</a>
</td>
</tr>
<tr>
@@ -77,7 +77,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
</td>
<td>Kafka
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/kafka/KafkaIO.html">KafkaIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html">spark-streaming-kafka</a>
</td>
@@ -85,7 +85,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>Kinesis
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/kinesis/KinesisIO.html">KinesisIO</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-kinesis-integration.html">spark-streaming-kinesis</a>
</td>
@@ -93,7 +93,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a>
</td>
<td><a href="https://github.com/apache/bahir/tree/master/streaming-pubsub">spark-streaming-pubsub</a> from <a href="http://bahir.apache.org">Apache Bahir</a>
</td>
@@ -103,7 +103,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
</td>
<td>Custom receivers
</td>
- <td><a href="https://beam.apache.org/documentation/io/authoring-overview/#read-transforms">Read Transforms</a>
+ <td><a href="{{ site.baseurl }}/documentation/io/authoring-overview/#read-transforms">Read Transforms</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-custom-receivers.html">receiverStream</a>
</td>
@@ -112,7 +112,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
## Connectors for Python
-Beam has an official [Python SDK](https://beam.apache.org/documentation/sdks/python/) that currently supports a subset of the streaming features available in the Java SDK. Active development is underway to bridge the gap between the featuresets in the two SDKs. Currently for Python, the [Direct Runner](https://beam.apache.org/documentation/runners/direct/) and [Dataflow Runner](https://beam.apache.org/documentation/runners/dataflow/) are supported, and [several streaming options](https:/ [...]
+Beam has an official [Python SDK]({{ site.baseurl }}/documentation/sdks/python/) that currently supports a subset of the streaming features available in the Java SDK. Active development is underway to bridge the gap between the featuresets in the two SDKs. Currently for Python, the [Direct Runner]({{ site.baseurl }}/documentation/runners/direct/) and [Dataflow Runner]({{ site.baseurl }}/documentation/runners/dataflow/) are supported, and [several streaming options]({{ site.baseurl }}/doc [...]
Spark also has a Python SDK called [PySpark](http://spark.apache.org/docs/latest/api/python/pyspark.html). As mentioned earlier, Scala code compiles to a bytecode that is executed by the JVM. PySpark uses [Py4J](https://www.py4j.org/), a library that enables Python programs to interact with the JVM and therefore access Java libraries, interact with Java objects, and register callbacks from Java. This allows PySpark to access native Spark objects like RDDs. Spark Structured Streaming supp [...]
@@ -134,7 +134,7 @@ Below are the main streaming input connectors for available for Beam and Spark D
</td>
<td>Local
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/pydoc/2.6.0/apache_beam.io.textio.html">io.textio</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/pydoc/{{ site.release_latest }}/apache_beam.io.textio.html">io.textio</a>
</td>
<td><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -142,7 +142,7 @@ Below are the main streaming input connectors for available for Beam and Spark D
<tr>
<td>HDFS
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/pydoc/2.6.0/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/pydoc/{{ site.release_latest }}/apache_beam.io.hadoopfilesystem.html">io.hadoopfilesystem</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/SparkContext.html#hadoopConfiguration--">hadoopConfiguration</a> (Access through <code>sc._jsc</code> with Py4J)
and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
@@ -153,7 +153,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.ht
</td>
<td>Google Cloud Storage
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/pydoc/{{ site.release_latest }}/apache_beam.io.gcp.gcsio.html">io.gcp.gcsio</a>
</td>
<td rowspan="2" ><a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext.textFileStream">textFileStream</a>
</td>
@@ -185,7 +185,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.ht
<tr>
<td>Cloud Pub/Sub
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/pydoc/2.6.0/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/pydoc/{{ site.release_latest }}/apache_beam.io.gcp.pubsub.html">io.gcp.pubsub</a>
</td>
<td>N/A
</td>
@@ -195,7 +195,7 @@ and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.streaming.ht
</td>
<td>Custom receivers
</td>
- <td><a href="https://beam.apache.org/documentation/sdks/python-custom-io/">BoundedSource and RangeTracker</a>
+ <td><a href="{{ site.baseurl }}/documentation/sdks/python-custom-io/">BoundedSource and RangeTracker</a>
</td>
<td>N/A
</td>
@@ -210,7 +210,7 @@ Since Scala code is interoperable with Java and therefore has native compatibili
### **Go**
-A [Go SDK](https://beam.apache.org/documentation/sdks/go/) for Apache Beam is under active development. It is currently experimental and is not recommended for production. Spark does not have an official Go SDK.
+A [Go SDK]({{ site.baseurl }}/documentation/sdks/go/) for Apache Beam is under active development. It is currently experimental and is not recommended for production. Spark does not have an official Go SDK.
### **R**
@@ -222,5 +222,5 @@ We hope this article inspired you to try new and interesting ways of connecting
Check out the following links for further information:
-* See a full list of all built-in and in-progress [I/O Transforms](https://beam.apache.org/documentation/io/built-in/) for Apache Beam.
-* Learn about some Apache Beam mobile gaming pipeline [examples](https://beam.apache.org/get-started/mobile-gaming-example/).
+* See a full list of all built-in and in-progress [I/O Transforms]({{ site.baseurl }}/documentation/io/built-in/) for Apache Beam.
+* Learn about some Apache Beam mobile gaming pipeline [examples]({{ site.baseurl }}/get-started/mobile-gaming-example/).