You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by da...@apache.org on 2017/03/18 01:35:28 UTC
[1/4] beam-site git commit: Add Pipeline I/O section to website -
outline + move some existing content
Repository: beam-site
Updated Branches:
refs/heads/asf-site cb6d7d77e -> b5748765f
Add Pipeline I/O section to website - outline + move some existing content
* I did not to go with a single page for all this content b/c both java and python have enough unique content that they deserve their own separate sections (ie, just tabs on the code isn't enough), and the "click to the next page" model currently implemented allows the user to pick java vs python, but then after reading those pages, the next page for both points at the same place - the users mostly follow the same path, but for java vs python specific content, they will diverge then converge again.
* I moved the "list of built-in I/O" content over to it's own separate page since it'd be nice to have more content there - e.g. capabilities matrix, and it felt special enough to pull out of the programming guide.
* We decided not to put all of this content in the contribute section of the site since the expectation is we don't think all users will contribute their IO transforms, so we want most of the docs to just be about writing an IO transforms, and they lay out the expectations in the contribute part of the IO section.
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/f2171885
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/f2171885
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/f2171885
Branch: refs/heads/asf-site
Commit: f21718850c645c83767f9787d335964da142fda9
Parents: cb6d7d7
Author: Stephen Sisk <si...@google.com>
Authored: Wed Mar 8 17:49:37 2017 -0800
Committer: Davor Bonaci <da...@google.com>
Committed: Fri Mar 17 18:33:43 2017 -0700
----------------------------------------------------------------------
src/_includes/header.html | 1 +
src/documentation/io/authoring-java.md | 15 ++++++
src/documentation/io/authoring-overview.md | 44 ++++++++++++++++++
src/documentation/io/authoring-python.md | 18 ++++++++
src/documentation/io/built-in.md | 61 +++++++++++++++++++++++++
src/documentation/io/contributing.md | 15 ++++++
src/documentation/io/io-toc.md | 26 +++++++++++
src/documentation/io/testing.md | 19 ++++++++
src/documentation/programming-guide.md | 54 ++--------------------
src/documentation/sdks/java.md | 21 +--------
10 files changed, 204 insertions(+), 70 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/_includes/header.html
----------------------------------------------------------------------
diff --git a/src/_includes/header.html b/src/_includes/header.html
index 28000d8..1ea3496 100644
--- a/src/_includes/header.html
+++ b/src/_includes/header.html
@@ -42,6 +42,7 @@
<li><a href="{{ site.baseurl }}/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="{{ site.baseurl }}/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="{{ site.baseurl }}/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="{{ site.baseurl }}/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/io/authoring-java.md
----------------------------------------------------------------------
diff --git a/src/documentation/io/authoring-java.md b/src/documentation/io/authoring-java.md
new file mode 100644
index 0000000..6cdb6bd
--- /dev/null
+++ b/src/documentation/io/authoring-java.md
@@ -0,0 +1,15 @@
+---
+layout: default
+title: "Authoring I/O Transforms - Java"
+permalink: /documentation/io/authoring-java/
+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Authoring I/O Transforms - Java
+
+> Note: This guide is still in progress. There is an open issue to finish the guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
+
+# Next steps
+
+[Testing I/O Transforms]({{site.baseurl }}/documentation/io/testing/)
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/io/authoring-overview.md
----------------------------------------------------------------------
diff --git a/src/documentation/io/authoring-overview.md b/src/documentation/io/authoring-overview.md
new file mode 100644
index 0000000..dab6a85
--- /dev/null
+++ b/src/documentation/io/authoring-overview.md
@@ -0,0 +1,44 @@
+---
+layout: default
+title: "Authoring I/O Transforms - Overview"
+permalink: /documentation/io/authoring-overview/
+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Authoring I/O Transforms - Overview
+
+_A guide for users who need to connect to a data store that isn't supported by the [Built-in I/O Transforms]({{site.baseurl }}/documentation/io/built-in/)_
+
+> Note: This guide is still in progress. There is an open issue to finish the guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
+
+* TOC
+{:toc}
+
+## Introduction
+TODO
+
+## Example I/O Transforms
+TODO
+
+## Suggested steps for implementers
+TODO
+
+## Read transforms
+TODO
+
+### When to implement using the Source API
+TODO
+
+## Write transforms
+TODO
+
+### When to implement using the Sink API
+TODO
+
+# Next steps
+
+For more details on actual implementation, continue with one of the the language specific guides:
+
+* [Authoring I/O Transforms - Python]({{site.baseurl }}/documentation/io/authoring-python/)
+* [Authoring I/O Transforms - Java]({{site.baseurl }}/documentation/io/authoring-java/)
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/io/authoring-python.md
----------------------------------------------------------------------
diff --git a/src/documentation/io/authoring-python.md b/src/documentation/io/authoring-python.md
new file mode 100644
index 0000000..b6ccc56
--- /dev/null
+++ b/src/documentation/io/authoring-python.md
@@ -0,0 +1,18 @@
+---
+layout: default
+title: "Authoring I/O Transforms - Python"
+permalink: /documentation/io/authoring-python/
+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Authoring I/O Transforms - Python
+
+> Note: This guide is still in progress. There is an open issue to finish the guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
+
+TODO - move in the [current python SDK content]({{site.baseurl}}/documentation/sdks/python-custom-io/)
+
+
+# Next steps
+
+[Testing I/O Transforms]({{site.baseurl}}/documentation/io/testing/)
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/io/built-in.md
----------------------------------------------------------------------
diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
new file mode 100644
index 0000000..9f96968
--- /dev/null
+++ b/src/documentation/io/built-in.md
@@ -0,0 +1,61 @@
+---
+layout: default
+title: "Built-in I/O Transforms"
+permalink: /documentation/io/built-in/
+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Built-in I/O Transforms
+
+This table contains the currently available I/O transforms.
+
+Consult the [Programming Guide I/O section]({{site.baseurl }}/documentation/programming-guide#io) for general usage instructions, and see the javadoc/pydoc for the particular I/O transforms.
+
+
+<table class="table table-bordered">
+<tr>
+ <th>Language</th>
+ <th>File-based</th>
+ <th>Messaging</th>
+ <th>Database</th>
+</tr>
+<tr>
+ <td>Java</td>
+ <td>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java">AvroIO</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hdfs">Apache Hadoop HDFS</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java">TextIO</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/">XML</a></p>
+ </td>
+ <td>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jms">JMS</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kafka">Apache Kafka</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis">Amazon Kinesis</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io">Google Cloud PubSub</a></p>
+ </td>
+ <td>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hbase">Apache HBase</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/mongodb">MongoDB</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jdbc">JDBC</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery">Google BigQuery</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable">Google Cloud Bigtable</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore">Google Cloud Datastore</a></p>
+ </td>
+</tr>
+<tr>
+ <td>Python</td>
+ <td>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">avroio</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py">textio</a></p>
+ </td>
+ <td>
+ </td>
+ <td>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py">Google BigQuery</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/datastore">Google Cloud Datastore</a></p>
+ </td>
+
+</tr>
+</table>
+
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/io/contributing.md
----------------------------------------------------------------------
diff --git a/src/documentation/io/contributing.md b/src/documentation/io/contributing.md
new file mode 100644
index 0000000..949db3c
--- /dev/null
+++ b/src/documentation/io/contributing.md
@@ -0,0 +1,15 @@
+---
+layout: default
+title: "Contributing I/O Transforms"
+permalink: /documentation/io/contributing/
+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Contributing I/O Transforms
+
+* If you are planning to contribute your I/O transform to the Apache Beam community, you'll be going through the normal Beam contribution life cycle - see the [Apache Beam Contribution Guide]({{ site.baseurl }}/contribute/contribution-guide/) for more details.
+* Talk to the community!
+* Make sure you've implemented the appropriate tests as discussed in the [Testing I/O Transforms]({{site.baseurl }}/documentation/io/testing/) section.
+
+> Note: This guide is still in progress. There is an open issue to finish the guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/io/io-toc.md
----------------------------------------------------------------------
diff --git a/src/documentation/io/io-toc.md b/src/documentation/io/io-toc.md
new file mode 100644
index 0000000..ec6b244
--- /dev/null
+++ b/src/documentation/io/io-toc.md
@@ -0,0 +1,26 @@
+---
+layout: default
+title: "Pipeline I/O"
+permalink: /documentation/io/io-toc/
+---
+
+# Pipeline I/O
+
+## Using Pipeline I/O
+* [Programming Guide: Using I/O Transforms]({{site.baseurl }}/documentation/programming-guide#io)
+* [Built-in I/O Transforms]({{site.baseurl }}/documentation/io/built-in/)
+
+
+## Authoring Read & Write I/O Transforms
+
+> Note: This guide is still in progress. There is an open issue to finish the guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
+
+<!-- TODO: commented out until this content is ready.
+
+This series of articles will walk you through the process of creating a new I/O transform.
+
+* [Authoring I/O Transforms - Overview]({{site.baseurl }}/documentation/io/authoring-overview/)
+* [Authoring I/O Transforms - Python]({{site.baseurl }}/documentation/io/authoring-python/)
+* [Authoring I/O Transforms - Java]({{site.baseurl }}/documentation/io/authoring-java/)
+* [Testing I/O Transforms]({{site.baseurl }}/documentation/io/testing/)
+* [Contributing I/O Transforms]({{site.baseurl }}/documentation/io/contributing/) -->
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/io/testing.md
----------------------------------------------------------------------
diff --git a/src/documentation/io/testing.md b/src/documentation/io/testing.md
new file mode 100644
index 0000000..e43c628
--- /dev/null
+++ b/src/documentation/io/testing.md
@@ -0,0 +1,19 @@
+---
+layout: default
+title: "Testing I/O Transforms"
+permalink: /documentation/io/testing/
+---
+
+[Pipeline I/O Table of Contents]({{site.baseurl}}/documentation/io/io-toc/)
+
+# Testing I/O Transforms
+
+> Note: This guide is still in progress. There is an open issue to finish the guide: [BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025).
+
+
+# Next steps
+
+If you have a well tested I/O transform, why not contribute it to Apache Beam? Read all about it:
+
+[Contributing I/O Transforms]({{site.baseurl }}/documentation/io/contributing/)
+
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/programming-guide.md
----------------------------------------------------------------------
diff --git a/src/documentation/programming-guide.md b/src/documentation/programming-guide.md
index 65a3062..57b49e8 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -921,9 +921,8 @@ While `ParDo` always produces a main output `PCollection` (as the return value f
## <a name="io"></a>Pipeline I/O
-When you create a pipeline, you often need to read data from some external source, such as a file in external data sink or a database. Likewise, you may want your pipeline to output its result data to a similar external data sink. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn't supported by the built-in transforms, you can implement your own read and write transforms.
+When you create a pipeline, you often need to read data from some external source, such as a file in external data sink or a database. Likewise, you may want your pipeline to output its result data to a similar external data sink. Beam provides read and write transforms for a [number of common data storage types]({{site.baseurl }}/documentation/io/built-in/). If you want your pipeline to read from or write to a data storage format that isn't supported by the built-in transforms, you can [implement your own read and write transforms]({{site.baseurl }}/documentation/io/io-toc/).
-> A guide that covers how to implement your own Beam IO transforms is in progress ([BEAM-1025](https://issues.apache.org/jira/browse/BEAM-1025)).
### Reading input data
@@ -988,55 +987,8 @@ records.apply("WriteToText",
%}
```
-### Beam-provided I/O APIs
-
-See the language specific source code directories for the Beam supported I/O APIs. Specific documentation for each of these I/O sources will be added in the future. ([BEAM-1054](https://issues.apache.org/jira/browse/BEAM-1054))
-
-<table class="table table-bordered">
-<tr>
- <th>Language</th>
- <th>File-based</th>
- <th>Messaging</th>
- <th>Database</th>
-</tr>
-<tr>
- <td>Java</td>
- <td>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java">AvroIO</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hdfs">HDFS</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java">TextIO</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/">XML</a></p>
- </td>
- <td>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jms">JMS</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kafka">Kafka</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis">Kinesis</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io">Google Cloud PubSub</a></p>
- </td>
- <td>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hbase">Apache HBase</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/mongodb">MongoDB</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jdbc">JDBC</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery">Google BigQuery</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable">Google Cloud Bigtable</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore">Google Cloud Datastore</a></p>
- </td>
-</tr>
-<tr>
- <td>Python</td>
- <td>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">avroio</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py">textio</a></p>
- </td>
- <td>
- </td>
- <td>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py">Google BigQuery</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/datastore">Google Cloud Datastore</a></p>
- </td>
-
-</tr>
-</table>
+### Beam-provided I/O Transforms
+See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms.
## <a name="running"></a>Running the pipeline
http://git-wip-us.apache.org/repos/asf/beam-site/blob/f2171885/src/documentation/sdks/java.md
----------------------------------------------------------------------
diff --git a/src/documentation/sdks/java.md b/src/documentation/sdks/java.md
index 1a3d856..474dc93 100644
--- a/src/documentation/sdks/java.md
+++ b/src/documentation/sdks/java.md
@@ -21,22 +21,5 @@ See the [Java API Reference]({{ site.baseurl }}/documentation/sdks/javadoc/) for
The Java SDK supports all features currently supported by the Beam model.
-## Supported IO Connectors
-
-* Amazon Kinesis
-* Apache Hadoop's `FileInputFormat` in Hadoop Distributed File System (HDFS)
-* Apache HBase
-* Apache Kafka
-* Avro Files
-* Google BigQuery
-* Google Cloud Bigtable
-* Google Cloud Datastore
-* Google Cloud Pub/Sub
-* Google Cloud Storage
-* Java Database Connectivity (JDBC)
-* Java Message Service (JMS)
-* MongoDB
-* Text Files
-* XML Files
-
-
+## Pipeline I/O
+See the [Beam-provided I/O Transforms]({{site.baseurl }}/documentation/io/built-in/) page for a list of the currently available I/O transforms.
[2/4] beam-site git commit: Regenerate website
Posted by da...@apache.org.
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/contributing/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/contributing/index.html b/content/documentation/io/contributing/index.html
new file mode 100644
index 0000000..43ace0c
--- /dev/null
+++ b/content/documentation/io/contributing/index.html
@@ -0,0 +1,193 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Contributing I/O Transforms</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.
+">
+
+ <link rel="stylesheet" href="/styles/site.css">
+ <link rel="stylesheet" href="/css/theme.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/contributing/" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+ <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+ <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Example Walkthroughs</li>
+ <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+ <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Resources</li>
+ <li><a href="/get-started/downloads">Downloads</a></li>
+ <li><a href="/get-started/support">Support</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/documentation">Using the Documentation</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Beam Concepts</li>
+ <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+ <li><a href="/documentation/resources/">Additional Resources</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Pipeline Fundamentals</li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SDKs</li>
+ <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+ <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+ <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Runners</li>
+ <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+ <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+ <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+ <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+ <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+ <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute">Get Started Contributing</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Guides</li>
+ <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+ <li><a href="/contribute/testing/">Testing Guide</a></li>
+ <li><a href="/contribute/release-guide/">Release Guide</a></li>
+ <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Technical References</li>
+ <li><a href="/contribute/design-principles/">Design Principles</a></li>
+ <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+ <li><a href="/contribute/source-repository/">Source Repository</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Promotion</li>
+ <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+ <li><a href="/contribute/logos/">Logos and Design</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+ <li><a href="/contribute/team/">Team</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+ <p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
+
+<h1 id="contributing-io-transforms">Contributing I/O Transforms</h1>
+
+<ul>
+ <li>If you are planning to contribute your I/O transform to the Apache Beam community, you\u2019ll be going through the normal Beam contribution life cycle - see the <a href="/contribute/contribution-guide/">Apache Beam Contribution Guide</a> for more details.</li>
+ <li>Talk to the community!</li>
+ <li>Make sure you\u2019ve implemented the appropriate tests as discussed in the <a href="/documentation/io/testing/">Testing I/O Transforms</a> section.</li>
+</ul>
+
+<blockquote>
+ <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
+</blockquote>
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">
+ © Copyright
+ <a href="http://www.apache.org">The Apache Software Foundation</a>,
+ 2017. All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="/privacy_policy">Privacy Policy</a> |
+ <a href="/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ </body>
+
+</html>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/io-toc/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/io-toc/index.html b/content/documentation/io/io-toc/index.html
new file mode 100644
index 0000000..3e35922
--- /dev/null
+++ b/content/documentation/io/io-toc/index.html
@@ -0,0 +1,203 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Pipeline I/O</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.
+">
+
+ <link rel="stylesheet" href="/styles/site.css">
+ <link rel="stylesheet" href="/css/theme.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/io-toc/" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+ <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+ <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Example Walkthroughs</li>
+ <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+ <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Resources</li>
+ <li><a href="/get-started/downloads">Downloads</a></li>
+ <li><a href="/get-started/support">Support</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/documentation">Using the Documentation</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Beam Concepts</li>
+ <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+ <li><a href="/documentation/resources/">Additional Resources</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Pipeline Fundamentals</li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SDKs</li>
+ <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+ <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+ <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Runners</li>
+ <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+ <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+ <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+ <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+ <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+ <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute">Get Started Contributing</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Guides</li>
+ <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+ <li><a href="/contribute/testing/">Testing Guide</a></li>
+ <li><a href="/contribute/release-guide/">Release Guide</a></li>
+ <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Technical References</li>
+ <li><a href="/contribute/design-principles/">Design Principles</a></li>
+ <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+ <li><a href="/contribute/source-repository/">Source Repository</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Promotion</li>
+ <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+ <li><a href="/contribute/logos/">Logos and Design</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+ <li><a href="/contribute/team/">Team</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+ <h1 id="pipeline-io">Pipeline I/O</h1>
+
+<h2 id="using-pipeline-io">Using Pipeline I/O</h2>
+<ul>
+ <li><a href="/documentation/programming-guide#io">Programming Guide: Using I/O Transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O Transforms</a></li>
+</ul>
+
+<h2 id="authoring-read-amp-write-io-transforms">Authoring Read & Write I/O Transforms</h2>
+
+<blockquote>
+ <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
+</blockquote>
+
+<!-- TODO: commented out until this content is ready.
+
+This series of articles will walk you through the process of creating a new I/O transform.
+
+* [Authoring I/O Transforms - Overview](/documentation/io/authoring-overview/)
+* [Authoring I/O Transforms - Python](/documentation/io/authoring-python/)
+* [Authoring I/O Transforms - Java](/documentation/io/authoring-java/)
+* [Testing I/O Transforms](/documentation/io/testing/)
+* [Contributing I/O Transforms](/documentation/io/contributing/) -->
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">
+ © Copyright
+ <a href="http://www.apache.org">The Apache Software Foundation</a>,
+ 2017. All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="/privacy_policy">Privacy Policy</a> |
+ <a href="/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ </body>
+
+</html>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/testing/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/testing/index.html b/content/documentation/io/testing/index.html
new file mode 100644
index 0000000..03ba859
--- /dev/null
+++ b/content/documentation/io/testing/index.html
@@ -0,0 +1,194 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Testing I/O Transforms</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.
+">
+
+ <link rel="stylesheet" href="/styles/site.css">
+ <link rel="stylesheet" href="/css/theme.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/testing/" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+ <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+ <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Example Walkthroughs</li>
+ <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+ <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Resources</li>
+ <li><a href="/get-started/downloads">Downloads</a></li>
+ <li><a href="/get-started/support">Support</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/documentation">Using the Documentation</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Beam Concepts</li>
+ <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+ <li><a href="/documentation/resources/">Additional Resources</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Pipeline Fundamentals</li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SDKs</li>
+ <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+ <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+ <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Runners</li>
+ <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+ <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+ <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+ <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+ <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+ <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute">Get Started Contributing</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Guides</li>
+ <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+ <li><a href="/contribute/testing/">Testing Guide</a></li>
+ <li><a href="/contribute/release-guide/">Release Guide</a></li>
+ <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Technical References</li>
+ <li><a href="/contribute/design-principles/">Design Principles</a></li>
+ <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+ <li><a href="/contribute/source-repository/">Source Repository</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Promotion</li>
+ <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+ <li><a href="/contribute/logos/">Logos and Design</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+ <li><a href="/contribute/team/">Team</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+ <p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
+
+<h1 id="testing-io-transforms">Testing I/O Transforms</h1>
+
+<blockquote>
+ <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
+</blockquote>
+
+<h1 id="next-steps">Next steps</h1>
+
+<p>If you have a well tested I/O transform, why not contribute it to Apache Beam? Read all about it:</p>
+
+<p><a href="/documentation/io/contributing/">Contributing I/O Transforms</a></p>
+
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">
+ © Copyright
+ <a href="http://www.apache.org">The Apache Software Foundation</a>,
+ 2017. All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="/privacy_policy">Privacy Policy</a> |
+ <a href="/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ </body>
+
+</html>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/pipelines/create-your-pipeline/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/pipelines/create-your-pipeline/index.html b/content/documentation/pipelines/create-your-pipeline/index.html
index a8bca14..8911488 100644
--- a/content/documentation/pipelines/create-your-pipeline/index.html
+++ b/content/documentation/pipelines/create-your-pipeline/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/pipelines/design-your-pipeline/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/pipelines/design-your-pipeline/index.html b/content/documentation/pipelines/design-your-pipeline/index.html
index 84186c8..c575446 100644
--- a/content/documentation/pipelines/design-your-pipeline/index.html
+++ b/content/documentation/pipelines/design-your-pipeline/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/pipelines/test-your-pipeline/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/pipelines/test-your-pipeline/index.html b/content/documentation/pipelines/test-your-pipeline/index.html
index 022cae5..112700b 100644
--- a/content/documentation/pipelines/test-your-pipeline/index.html
+++ b/content/documentation/pipelines/test-your-pipeline/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/programming-guide/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/programming-guide/index.html b/content/documentation/programming-guide/index.html
index 707c525..19853df 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
@@ -1220,11 +1221,7 @@ tree, [2]
<h2 id="a-nameioapipeline-io"><a name="io"></a>Pipeline I/O</h2>
-<p>When you create a pipeline, you often need to read data from some external source, such as a file in external data sink or a database. Likewise, you may want your pipeline to output its result data to a similar external data sink. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn\u2019t supported by the built-in transforms, you can implement your own read and write transforms.</p>
-
-<blockquote>
- <p>A guide that covers how to implement your own Beam IO transforms is in progress (<a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>).</p>
-</blockquote>
+<p>When you create a pipeline, you often need to read data from some external source, such as a file in external data sink or a database. Likewise, you may want your pipeline to output its result data to a similar external data sink. Beam provides read and write transforms for a <a href="/documentation/io/built-in/">number of common data storage types</a>. If you want your pipeline to read from or write to a data storage format that isn\u2019t supported by the built-in transforms, you can <a href="/documentation/io/io-toc/">implement your own read and write transforms</a>.</p>
<h3 id="reading-input-data">Reading input data</h3>
@@ -1290,55 +1287,8 @@ tree, [2]
</code></pre>
</div>
-<h3 id="beam-provided-io-apis">Beam-provided I/O APIs</h3>
-
-<p>See the language specific source code directories for the Beam supported I/O APIs. Specific documentation for each of these I/O sources will be added in the future. (<a href="https://issues.apache.org/jira/browse/BEAM-1054">BEAM-1054</a>)</p>
-
-<table class="table table-bordered">
-<tr>
- <th>Language</th>
- <th>File-based</th>
- <th>Messaging</th>
- <th>Database</th>
-</tr>
-<tr>
- <td>Java</td>
- <td>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java">AvroIO</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hdfs">HDFS</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java">TextIO</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/">XML</a></p>
- </td>
- <td>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jms">JMS</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kafka">Kafka</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis">Kinesis</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io">Google Cloud PubSub</a></p>
- </td>
- <td>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hbase">Apache HBase</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/mongodb">MongoDB</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jdbc">JDBC</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery">Google BigQuery</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable">Google Cloud Bigtable</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore">Google Cloud Datastore</a></p>
- </td>
-</tr>
-<tr>
- <td>Python</td>
- <td>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">avroio</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py">textio</a></p>
- </td>
- <td>
- </td>
- <td>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py">Google BigQuery</a></p>
- <p><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/datastore">Google Cloud Datastore</a></p>
- </td>
-
-</tr>
-</table>
+<h3 id="beam-provided-io-transforms">Beam-provided I/O Transforms</h3>
+<p>See the <a href="/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms.</p>
<h2 id="a-namerunningarunning-the-pipeline"><a name="running"></a>Running the pipeline</h2>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/resources/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/resources/index.html b/content/documentation/resources/index.html
index ae6c691..dc74031 100644
--- a/content/documentation/resources/index.html
+++ b/content/documentation/resources/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/apex/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/apex/index.html b/content/documentation/runners/apex/index.html
index f85c270..875f3e0 100644
--- a/content/documentation/runners/apex/index.html
+++ b/content/documentation/runners/apex/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/capability-matrix/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/capability-matrix/index.html b/content/documentation/runners/capability-matrix/index.html
index efce223..04174c7 100644
--- a/content/documentation/runners/capability-matrix/index.html
+++ b/content/documentation/runners/capability-matrix/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/dataflow/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/dataflow/index.html b/content/documentation/runners/dataflow/index.html
index 259e99f..2f3d9b0 100644
--- a/content/documentation/runners/dataflow/index.html
+++ b/content/documentation/runners/dataflow/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/direct/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/direct/index.html b/content/documentation/runners/direct/index.html
index 11bbf6e..dd1151a 100644
--- a/content/documentation/runners/direct/index.html
+++ b/content/documentation/runners/direct/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/flink/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/flink/index.html b/content/documentation/runners/flink/index.html
index f451eba..b0b8fdc 100644
--- a/content/documentation/runners/flink/index.html
+++ b/content/documentation/runners/flink/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/runners/spark/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/runners/spark/index.html b/content/documentation/runners/spark/index.html
index 29fce06..16c3e5b 100644
--- a/content/documentation/runners/spark/index.html
+++ b/content/documentation/runners/spark/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/java/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/java/index.html b/content/documentation/sdks/java/index.html
index 4901fb3..18f406d 100644
--- a/content/documentation/sdks/java/index.html
+++ b/content/documentation/sdks/java/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
@@ -164,26 +165,8 @@
<p>The Java SDK supports all features currently supported by the Beam model.</p>
-<h2 id="supported-io-connectors">Supported IO Connectors</h2>
-
-<ul>
- <li>Amazon Kinesis</li>
- <li>Apache Hadoop\u2019s <code class="highlighter-rouge">FileInputFormat</code> in Hadoop Distributed File System (HDFS)</li>
- <li>Apache HBase</li>
- <li>Apache Kafka</li>
- <li>Avro Files</li>
- <li>Google BigQuery</li>
- <li>Google Cloud Bigtable</li>
- <li>Google Cloud Datastore</li>
- <li>Google Cloud Pub/Sub</li>
- <li>Google Cloud Storage</li>
- <li>Java Database Connectivity (JDBC)</li>
- <li>Java Message Service (JMS)</li>
- <li>MongoDB</li>
- <li>Text Files</li>
- <li>XML Files</li>
-</ul>
-
+<h2 id="pipeline-io">Pipeline I/O</h2>
+<p>See the <a href="/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms.</p>
</div>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python-custom-io/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/python-custom-io/index.html b/content/documentation/sdks/python-custom-io/index.html
index d6195b6..d078e33 100644
--- a/content/documentation/sdks/python-custom-io/index.html
+++ b/content/documentation/sdks/python-custom-io/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python-pipeline-dependencies/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/python-pipeline-dependencies/index.html b/content/documentation/sdks/python-pipeline-dependencies/index.html
index c0d9bdd..dba6ff7 100644
--- a/content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python-type-safety/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/python-type-safety/index.html b/content/documentation/sdks/python-type-safety/index.html
index 092e02f..24900b8 100644
--- a/content/documentation/sdks/python-type-safety/index.html
+++ b/content/documentation/sdks/python-type-safety/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/sdks/python/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/sdks/python/index.html b/content/documentation/sdks/python/index.html
index d7f7240..e5c9db7 100644
--- a/content/documentation/sdks/python/index.html
+++ b/content/documentation/sdks/python/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/beam-overview/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/beam-overview/index.html b/content/get-started/beam-overview/index.html
index 7a2b1f1..caac276 100644
--- a/content/get-started/beam-overview/index.html
+++ b/content/get-started/beam-overview/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/downloads/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/downloads/index.html b/content/get-started/downloads/index.html
index 3438ee7..9aedfeb 100644
--- a/content/get-started/downloads/index.html
+++ b/content/get-started/downloads/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/index.html b/content/get-started/index.html
index b7cc440..b33241c 100644
--- a/content/get-started/index.html
+++ b/content/get-started/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/mobile-gaming-example/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/mobile-gaming-example/index.html b/content/get-started/mobile-gaming-example/index.html
index 0489219..5b7e487 100644
--- a/content/get-started/mobile-gaming-example/index.html
+++ b/content/get-started/mobile-gaming-example/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/quickstart-java/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/quickstart-java/index.html b/content/get-started/quickstart-java/index.html
index d4b6053..51f5e4c 100644
--- a/content/get-started/quickstart-java/index.html
+++ b/content/get-started/quickstart-java/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/quickstart-py/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/quickstart-py/index.html b/content/get-started/quickstart-py/index.html
index 153eac6..b717987 100644
--- a/content/get-started/quickstart-py/index.html
+++ b/content/get-started/quickstart-py/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/support/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/support/index.html b/content/get-started/support/index.html
index 9a00712..bf9702c 100644
--- a/content/get-started/support/index.html
+++ b/content/get-started/support/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/get-started/wordcount-example/index.html
----------------------------------------------------------------------
diff --git a/content/get-started/wordcount-example/index.html b/content/get-started/wordcount-example/index.html
index a5be22d..030c3e2 100644
--- a/content/get-started/wordcount-example/index.html
+++ b/content/get-started/wordcount-example/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/index.html
----------------------------------------------------------------------
diff --git a/content/index.html b/content/index.html
index dcb2f23..7f896cf 100644
--- a/content/index.html
+++ b/content/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/privacy_policy/index.html
----------------------------------------------------------------------
diff --git a/content/privacy_policy/index.html b/content/privacy_policy/index.html
index 951f387..eaef599 100644
--- a/content/privacy_policy/index.html
+++ b/content/privacy_policy/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
[4/4] beam-site git commit: This closes #173
Posted by da...@apache.org.
This closes #173
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/b5748765
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/b5748765
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/b5748765
Branch: refs/heads/asf-site
Commit: b5748765ffa80aa71b1023ef2e25625c734ebde8
Parents: cb6d7d7 35c2d97
Author: Davor Bonaci <da...@google.com>
Authored: Fri Mar 17 18:35:17 2017 -0700
Committer: Davor Bonaci <da...@google.com>
Committed: Fri Mar 17 18:35:17 2017 -0700
----------------------------------------------------------------------
.../2016/03/17/capability-matrix.html | 1 +
.../2016/04/03/presentation-materials.html | 1 +
.../sdk/2016/02/25/python-sdk-now-public.html | 1 +
.../beam/release/2016/06/15/first-release.html | 1 +
.../10/11/strata-hadoop-world-and-beam.html | 1 +
.../website/2016/02/22/beam-has-a-logo.html | 1 +
.../blog/2016/05/18/splitAtFraction-method.html | 1 +
.../05/27/where-is-my-pcollection-dot-map.html | 1 +
.../06/13/flink-batch-runner-milestone.html | 1 +
content/blog/2016/08/03/six-months.html | 1 +
content/blog/2016/10/20/test-stream.html | 1 +
content/blog/2017/01/09/added-apex-runner.html | 1 +
content/blog/2017/01/10/beam-graduates.html | 1 +
.../blog/2017/02/01/graduation-media-recap.html | 1 +
.../blog/2017/02/13/stateful-processing.html | 1 +
content/blog/2017/03/16/python-sdk-release.html | 1 +
content/blog/index.html | 1 +
content/coming-soon.html | 1 +
.../contribute/contribution-guide/index.html | 1 +
content/contribute/design-principles/index.html | 1 +
content/contribute/index.html | 1 +
content/contribute/logos/index.html | 1 +
content/contribute/maturity-model/index.html | 1 +
.../presentation-materials/index.html | 1 +
.../ptransform-style-guide/index.html | 1 +
content/contribute/release-guide/index.html | 1 +
content/contribute/source-repository/index.html | 1 +
content/contribute/team/index.html | 1 +
content/contribute/testing/index.html | 1 +
content/contribute/work-in-progress/index.html | 1 +
content/documentation/index.html | 1 +
.../documentation/io/authoring-java/index.html | 191 +++++++++++++++
.../io/authoring-overview/index.html | 233 ++++++++++++++++++
.../io/authoring-python/index.html | 193 +++++++++++++++
content/documentation/io/built-in/index.html | 234 +++++++++++++++++++
.../documentation/io/contributing/index.html | 193 +++++++++++++++
content/documentation/io/io-toc/index.html | 203 ++++++++++++++++
content/documentation/io/testing/index.html | 194 +++++++++++++++
.../pipelines/create-your-pipeline/index.html | 1 +
.../pipelines/design-your-pipeline/index.html | 1 +
.../pipelines/test-your-pipeline/index.html | 1 +
.../documentation/programming-guide/index.html | 58 +----
content/documentation/resources/index.html | 1 +
content/documentation/runners/apex/index.html | 1 +
.../runners/capability-matrix/index.html | 1 +
.../documentation/runners/dataflow/index.html | 1 +
content/documentation/runners/direct/index.html | 1 +
content/documentation/runners/flink/index.html | 1 +
content/documentation/runners/spark/index.html | 1 +
content/documentation/sdks/java/index.html | 23 +-
.../sdks/python-custom-io/index.html | 1 +
.../python-pipeline-dependencies/index.html | 1 +
.../sdks/python-type-safety/index.html | 1 +
content/documentation/sdks/python/index.html | 1 +
content/get-started/beam-overview/index.html | 1 +
content/get-started/downloads/index.html | 1 +
content/get-started/index.html | 1 +
.../mobile-gaming-example/index.html | 1 +
content/get-started/quickstart-java/index.html | 1 +
content/get-started/quickstart-py/index.html | 1 +
content/get-started/support/index.html | 1 +
.../get-started/wordcount-example/index.html | 1 +
content/index.html | 1 +
content/privacy_policy/index.html | 1 +
src/_includes/header.html | 1 +
src/documentation/io/authoring-java.md | 15 ++
src/documentation/io/authoring-overview.md | 44 ++++
src/documentation/io/authoring-python.md | 18 ++
src/documentation/io/built-in.md | 61 +++++
src/documentation/io/contributing.md | 15 ++
src/documentation/io/io-toc.md | 26 +++
src/documentation/io/testing.md | 19 ++
src/documentation/programming-guide.md | 54 +----
src/documentation/sdks/java.md | 21 +-
74 files changed, 1707 insertions(+), 144 deletions(-)
----------------------------------------------------------------------
[3/4] beam-site git commit: Regenerate website
Posted by da...@apache.org.
Regenerate website
Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/35c2d97b
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/35c2d97b
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/35c2d97b
Branch: refs/heads/asf-site
Commit: 35c2d97b0713a4d69c5dabdf39a3254cb8789cc1
Parents: f217188
Author: Davor Bonaci <da...@google.com>
Authored: Fri Mar 17 18:35:16 2017 -0700
Committer: Davor Bonaci <da...@google.com>
Committed: Fri Mar 17 18:35:16 2017 -0700
----------------------------------------------------------------------
.../2016/03/17/capability-matrix.html | 1 +
.../2016/04/03/presentation-materials.html | 1 +
.../sdk/2016/02/25/python-sdk-now-public.html | 1 +
.../beam/release/2016/06/15/first-release.html | 1 +
.../10/11/strata-hadoop-world-and-beam.html | 1 +
.../website/2016/02/22/beam-has-a-logo.html | 1 +
.../blog/2016/05/18/splitAtFraction-method.html | 1 +
.../05/27/where-is-my-pcollection-dot-map.html | 1 +
.../06/13/flink-batch-runner-milestone.html | 1 +
content/blog/2016/08/03/six-months.html | 1 +
content/blog/2016/10/20/test-stream.html | 1 +
content/blog/2017/01/09/added-apex-runner.html | 1 +
content/blog/2017/01/10/beam-graduates.html | 1 +
.../blog/2017/02/01/graduation-media-recap.html | 1 +
.../blog/2017/02/13/stateful-processing.html | 1 +
content/blog/2017/03/16/python-sdk-release.html | 1 +
content/blog/index.html | 1 +
content/coming-soon.html | 1 +
.../contribute/contribution-guide/index.html | 1 +
content/contribute/design-principles/index.html | 1 +
content/contribute/index.html | 1 +
content/contribute/logos/index.html | 1 +
content/contribute/maturity-model/index.html | 1 +
.../presentation-materials/index.html | 1 +
.../ptransform-style-guide/index.html | 1 +
content/contribute/release-guide/index.html | 1 +
content/contribute/source-repository/index.html | 1 +
content/contribute/team/index.html | 1 +
content/contribute/testing/index.html | 1 +
content/contribute/work-in-progress/index.html | 1 +
content/documentation/index.html | 1 +
.../documentation/io/authoring-java/index.html | 191 +++++++++++++++
.../io/authoring-overview/index.html | 233 ++++++++++++++++++
.../io/authoring-python/index.html | 193 +++++++++++++++
content/documentation/io/built-in/index.html | 234 +++++++++++++++++++
.../documentation/io/contributing/index.html | 193 +++++++++++++++
content/documentation/io/io-toc/index.html | 203 ++++++++++++++++
content/documentation/io/testing/index.html | 194 +++++++++++++++
.../pipelines/create-your-pipeline/index.html | 1 +
.../pipelines/design-your-pipeline/index.html | 1 +
.../pipelines/test-your-pipeline/index.html | 1 +
.../documentation/programming-guide/index.html | 58 +----
content/documentation/resources/index.html | 1 +
content/documentation/runners/apex/index.html | 1 +
.../runners/capability-matrix/index.html | 1 +
.../documentation/runners/dataflow/index.html | 1 +
content/documentation/runners/direct/index.html | 1 +
content/documentation/runners/flink/index.html | 1 +
content/documentation/runners/spark/index.html | 1 +
content/documentation/sdks/java/index.html | 23 +-
.../sdks/python-custom-io/index.html | 1 +
.../python-pipeline-dependencies/index.html | 1 +
.../sdks/python-type-safety/index.html | 1 +
content/documentation/sdks/python/index.html | 1 +
content/get-started/beam-overview/index.html | 1 +
content/get-started/downloads/index.html | 1 +
content/get-started/index.html | 1 +
.../mobile-gaming-example/index.html | 1 +
content/get-started/quickstart-java/index.html | 1 +
content/get-started/quickstart-py/index.html | 1 +
content/get-started/support/index.html | 1 +
.../get-started/wordcount-example/index.html | 1 +
content/index.html | 1 +
content/privacy_policy/index.html | 1 +
64 files changed, 1503 insertions(+), 74 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/beam/capability/2016/03/17/capability-matrix.html
----------------------------------------------------------------------
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html b/content/beam/capability/2016/03/17/capability-matrix.html
index 3f8b610..82da83a 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/beam/capability/2016/04/03/presentation-materials.html
----------------------------------------------------------------------
diff --git a/content/beam/capability/2016/04/03/presentation-materials.html b/content/beam/capability/2016/04/03/presentation-materials.html
index 6b75135..fb85487 100644
--- a/content/beam/capability/2016/04/03/presentation-materials.html
+++ b/content/beam/capability/2016/04/03/presentation-materials.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
----------------------------------------------------------------------
diff --git a/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html b/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
index 91df159..df9411b 100644
--- a/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
+++ b/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/beam/release/2016/06/15/first-release.html
----------------------------------------------------------------------
diff --git a/content/beam/release/2016/06/15/first-release.html b/content/beam/release/2016/06/15/first-release.html
index 87551f5..c2bdb95 100644
--- a/content/beam/release/2016/06/15/first-release.html
+++ b/content/beam/release/2016/06/15/first-release.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
----------------------------------------------------------------------
diff --git a/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html b/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
index 55a9c23..5db85b2 100644
--- a/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
+++ b/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/beam/update/website/2016/02/22/beam-has-a-logo.html
----------------------------------------------------------------------
diff --git a/content/beam/update/website/2016/02/22/beam-has-a-logo.html b/content/beam/update/website/2016/02/22/beam-has-a-logo.html
index f950d93..5861622 100644
--- a/content/beam/update/website/2016/02/22/beam-has-a-logo.html
+++ b/content/beam/update/website/2016/02/22/beam-has-a-logo.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2016/05/18/splitAtFraction-method.html
----------------------------------------------------------------------
diff --git a/content/blog/2016/05/18/splitAtFraction-method.html b/content/blog/2016/05/18/splitAtFraction-method.html
index 98069af..07b3cb1 100644
--- a/content/blog/2016/05/18/splitAtFraction-method.html
+++ b/content/blog/2016/05/18/splitAtFraction-method.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
----------------------------------------------------------------------
diff --git a/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html b/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
index 64a3f01..a728b2c 100644
--- a/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
+++ b/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2016/06/13/flink-batch-runner-milestone.html
----------------------------------------------------------------------
diff --git a/content/blog/2016/06/13/flink-batch-runner-milestone.html b/content/blog/2016/06/13/flink-batch-runner-milestone.html
index ff81e76..e909e6b 100644
--- a/content/blog/2016/06/13/flink-batch-runner-milestone.html
+++ b/content/blog/2016/06/13/flink-batch-runner-milestone.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2016/08/03/six-months.html
----------------------------------------------------------------------
diff --git a/content/blog/2016/08/03/six-months.html b/content/blog/2016/08/03/six-months.html
index 75d49a2..5115abe 100644
--- a/content/blog/2016/08/03/six-months.html
+++ b/content/blog/2016/08/03/six-months.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2016/10/20/test-stream.html
----------------------------------------------------------------------
diff --git a/content/blog/2016/10/20/test-stream.html b/content/blog/2016/10/20/test-stream.html
index c89dd65..4862955 100644
--- a/content/blog/2016/10/20/test-stream.html
+++ b/content/blog/2016/10/20/test-stream.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2017/01/09/added-apex-runner.html
----------------------------------------------------------------------
diff --git a/content/blog/2017/01/09/added-apex-runner.html b/content/blog/2017/01/09/added-apex-runner.html
index 0531a1e..5ff6b41 100644
--- a/content/blog/2017/01/09/added-apex-runner.html
+++ b/content/blog/2017/01/09/added-apex-runner.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2017/01/10/beam-graduates.html
----------------------------------------------------------------------
diff --git a/content/blog/2017/01/10/beam-graduates.html b/content/blog/2017/01/10/beam-graduates.html
index 438541c..31bd0f8 100644
--- a/content/blog/2017/01/10/beam-graduates.html
+++ b/content/blog/2017/01/10/beam-graduates.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2017/02/01/graduation-media-recap.html
----------------------------------------------------------------------
diff --git a/content/blog/2017/02/01/graduation-media-recap.html b/content/blog/2017/02/01/graduation-media-recap.html
index 0b12147..a98a325 100644
--- a/content/blog/2017/02/01/graduation-media-recap.html
+++ b/content/blog/2017/02/01/graduation-media-recap.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2017/02/13/stateful-processing.html
----------------------------------------------------------------------
diff --git a/content/blog/2017/02/13/stateful-processing.html b/content/blog/2017/02/13/stateful-processing.html
index ba03b4a..33a4338 100644
--- a/content/blog/2017/02/13/stateful-processing.html
+++ b/content/blog/2017/02/13/stateful-processing.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/2017/03/16/python-sdk-release.html
----------------------------------------------------------------------
diff --git a/content/blog/2017/03/16/python-sdk-release.html b/content/blog/2017/03/16/python-sdk-release.html
index cb1320c..186a4d4 100644
--- a/content/blog/2017/03/16/python-sdk-release.html
+++ b/content/blog/2017/03/16/python-sdk-release.html
@@ -76,6 +76,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/blog/index.html
----------------------------------------------------------------------
diff --git a/content/blog/index.html b/content/blog/index.html
index ca0732f..387f6f6 100644
--- a/content/blog/index.html
+++ b/content/blog/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/coming-soon.html
----------------------------------------------------------------------
diff --git a/content/coming-soon.html b/content/coming-soon.html
index 4bd74f8..d429688 100644
--- a/content/coming-soon.html
+++ b/content/coming-soon.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/contribution-guide/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/contribution-guide/index.html b/content/contribute/contribution-guide/index.html
index 2a5ec60..4d4a782 100644
--- a/content/contribute/contribution-guide/index.html
+++ b/content/contribute/contribution-guide/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/design-principles/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/design-principles/index.html b/content/contribute/design-principles/index.html
index e6df0f3..78cecea 100644
--- a/content/contribute/design-principles/index.html
+++ b/content/contribute/design-principles/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/index.html b/content/contribute/index.html
index 907f4f2..d315380 100644
--- a/content/contribute/index.html
+++ b/content/contribute/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/logos/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/logos/index.html b/content/contribute/logos/index.html
index 20b2a91..ae914cf 100644
--- a/content/contribute/logos/index.html
+++ b/content/contribute/logos/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/maturity-model/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/maturity-model/index.html b/content/contribute/maturity-model/index.html
index 3096031..3f18ea1 100644
--- a/content/contribute/maturity-model/index.html
+++ b/content/contribute/maturity-model/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/presentation-materials/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/presentation-materials/index.html b/content/contribute/presentation-materials/index.html
index 46f1015..f0c6f46 100644
--- a/content/contribute/presentation-materials/index.html
+++ b/content/contribute/presentation-materials/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/ptransform-style-guide/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/ptransform-style-guide/index.html b/content/contribute/ptransform-style-guide/index.html
index 3b32f9b..66921a3 100644
--- a/content/contribute/ptransform-style-guide/index.html
+++ b/content/contribute/ptransform-style-guide/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/release-guide/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/release-guide/index.html b/content/contribute/release-guide/index.html
index d98ac85..18ccaa8 100644
--- a/content/contribute/release-guide/index.html
+++ b/content/contribute/release-guide/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/source-repository/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/source-repository/index.html b/content/contribute/source-repository/index.html
index 2d70585..289e8be 100644
--- a/content/contribute/source-repository/index.html
+++ b/content/contribute/source-repository/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/team/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/team/index.html b/content/contribute/team/index.html
index 81fb4d7..f0ef73d 100644
--- a/content/contribute/team/index.html
+++ b/content/contribute/team/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/testing/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/testing/index.html b/content/contribute/testing/index.html
index fb004c9..c0ebd6d 100644
--- a/content/contribute/testing/index.html
+++ b/content/contribute/testing/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/contribute/work-in-progress/index.html
----------------------------------------------------------------------
diff --git a/content/contribute/work-in-progress/index.html b/content/contribute/work-in-progress/index.html
index 7cef411..90eff24 100644
--- a/content/contribute/work-in-progress/index.html
+++ b/content/contribute/work-in-progress/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/index.html b/content/documentation/index.html
index af5afb0..7d006b8 100644
--- a/content/documentation/index.html
+++ b/content/documentation/index.html
@@ -77,6 +77,7 @@
<li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
<li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
<li role="separator" class="divider"></li>
<li class="dropdown-header">SDKs</li>
<li><a href="/documentation/sdks/java/">Java SDK</a></li>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/authoring-java/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/authoring-java/index.html b/content/documentation/io/authoring-java/index.html
new file mode 100644
index 0000000..5128d93
--- /dev/null
+++ b/content/documentation/io/authoring-java/index.html
@@ -0,0 +1,191 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Authoring I/O Transforms - Java</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.
+">
+
+ <link rel="stylesheet" href="/styles/site.css">
+ <link rel="stylesheet" href="/css/theme.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/authoring-java/" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+ <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+ <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Example Walkthroughs</li>
+ <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+ <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Resources</li>
+ <li><a href="/get-started/downloads">Downloads</a></li>
+ <li><a href="/get-started/support">Support</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/documentation">Using the Documentation</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Beam Concepts</li>
+ <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+ <li><a href="/documentation/resources/">Additional Resources</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Pipeline Fundamentals</li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SDKs</li>
+ <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+ <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+ <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Runners</li>
+ <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+ <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+ <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+ <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+ <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+ <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute">Get Started Contributing</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Guides</li>
+ <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+ <li><a href="/contribute/testing/">Testing Guide</a></li>
+ <li><a href="/contribute/release-guide/">Release Guide</a></li>
+ <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Technical References</li>
+ <li><a href="/contribute/design-principles/">Design Principles</a></li>
+ <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+ <li><a href="/contribute/source-repository/">Source Repository</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Promotion</li>
+ <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+ <li><a href="/contribute/logos/">Logos and Design</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+ <li><a href="/contribute/team/">Team</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+ <p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
+
+<h1 id="authoring-io-transforms---java">Authoring I/O Transforms - Java</h1>
+
+<blockquote>
+ <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
+</blockquote>
+
+<h1 id="next-steps">Next steps</h1>
+
+<p><a href="/documentation/io/testing/">Testing I/O Transforms</a></p>
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">
+ © Copyright
+ <a href="http://www.apache.org">The Apache Software Foundation</a>,
+ 2017. All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="/privacy_policy">Privacy Policy</a> |
+ <a href="/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ </body>
+
+</html>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/authoring-overview/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/authoring-overview/index.html b/content/documentation/io/authoring-overview/index.html
new file mode 100644
index 0000000..73fffa2
--- /dev/null
+++ b/content/documentation/io/authoring-overview/index.html
@@ -0,0 +1,233 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Authoring I/O Transforms - Overview</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.
+">
+
+ <link rel="stylesheet" href="/styles/site.css">
+ <link rel="stylesheet" href="/css/theme.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/authoring-overview/" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+ <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+ <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Example Walkthroughs</li>
+ <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+ <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Resources</li>
+ <li><a href="/get-started/downloads">Downloads</a></li>
+ <li><a href="/get-started/support">Support</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/documentation">Using the Documentation</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Beam Concepts</li>
+ <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+ <li><a href="/documentation/resources/">Additional Resources</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Pipeline Fundamentals</li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SDKs</li>
+ <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+ <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+ <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Runners</li>
+ <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+ <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+ <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+ <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+ <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+ <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute">Get Started Contributing</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Guides</li>
+ <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+ <li><a href="/contribute/testing/">Testing Guide</a></li>
+ <li><a href="/contribute/release-guide/">Release Guide</a></li>
+ <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Technical References</li>
+ <li><a href="/contribute/design-principles/">Design Principles</a></li>
+ <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+ <li><a href="/contribute/source-repository/">Source Repository</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Promotion</li>
+ <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+ <li><a href="/contribute/logos/">Logos and Design</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+ <li><a href="/contribute/team/">Team</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+ <p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
+
+<h1 id="authoring-io-transforms---overview">Authoring I/O Transforms - Overview</h1>
+
+<p><em>A guide for users who need to connect to a data store that isn\u2019t supported by the <a href="/documentation/io/built-in/">Built-in I/O Transforms</a></em></p>
+
+<blockquote>
+ <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
+</blockquote>
+
+<ul id="markdown-toc">
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#example-io-transforms" id="markdown-toc-example-io-transforms">Example I/O Transforms</a></li>
+ <li><a href="#suggested-steps-for-implementers" id="markdown-toc-suggested-steps-for-implementers">Suggested steps for implementers</a></li>
+ <li><a href="#read-transforms" id="markdown-toc-read-transforms">Read transforms</a> <ul>
+ <li><a href="#when-to-implement-using-the-source-api" id="markdown-toc-when-to-implement-using-the-source-api">When to implement using the Source API</a></li>
+ </ul>
+ </li>
+ <li><a href="#write-transforms" id="markdown-toc-write-transforms">Write transforms</a> <ul>
+ <li><a href="#when-to-implement-using-the-sink-api" id="markdown-toc-when-to-implement-using-the-sink-api">When to implement using the Sink API</a></li>
+ </ul>
+ </li>
+</ul>
+
+<h2 id="introduction">Introduction</h2>
+<p>TODO</p>
+
+<h2 id="example-io-transforms">Example I/O Transforms</h2>
+<p>TODO</p>
+
+<h2 id="suggested-steps-for-implementers">Suggested steps for implementers</h2>
+<p>TODO</p>
+
+<h2 id="read-transforms">Read transforms</h2>
+<p>TODO</p>
+
+<h3 id="when-to-implement-using-the-source-api">When to implement using the Source API</h3>
+<p>TODO</p>
+
+<h2 id="write-transforms">Write transforms</h2>
+<p>TODO</p>
+
+<h3 id="when-to-implement-using-the-sink-api">When to implement using the Sink API</h3>
+<p>TODO</p>
+
+<h1 id="next-steps">Next steps</h1>
+
+<p>For more details on actual implementation, continue with one of the the language specific guides:</p>
+
+<ul>
+ <li><a href="/documentation/io/authoring-python/">Authoring I/O Transforms - Python</a></li>
+ <li><a href="/documentation/io/authoring-java/">Authoring I/O Transforms - Java</a></li>
+</ul>
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">
+ © Copyright
+ <a href="http://www.apache.org">The Apache Software Foundation</a>,
+ 2017. All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="/privacy_policy">Privacy Policy</a> |
+ <a href="/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ </body>
+
+</html>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/authoring-python/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/authoring-python/index.html b/content/documentation/io/authoring-python/index.html
new file mode 100644
index 0000000..0730406
--- /dev/null
+++ b/content/documentation/io/authoring-python/index.html
@@ -0,0 +1,193 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Authoring I/O Transforms - Python</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.
+">
+
+ <link rel="stylesheet" href="/styles/site.css">
+ <link rel="stylesheet" href="/css/theme.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/authoring-python/" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+ <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+ <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Example Walkthroughs</li>
+ <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+ <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Resources</li>
+ <li><a href="/get-started/downloads">Downloads</a></li>
+ <li><a href="/get-started/support">Support</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/documentation">Using the Documentation</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Beam Concepts</li>
+ <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+ <li><a href="/documentation/resources/">Additional Resources</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Pipeline Fundamentals</li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SDKs</li>
+ <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+ <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+ <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Runners</li>
+ <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+ <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+ <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+ <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+ <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+ <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute">Get Started Contributing</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Guides</li>
+ <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+ <li><a href="/contribute/testing/">Testing Guide</a></li>
+ <li><a href="/contribute/release-guide/">Release Guide</a></li>
+ <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Technical References</li>
+ <li><a href="/contribute/design-principles/">Design Principles</a></li>
+ <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+ <li><a href="/contribute/source-repository/">Source Repository</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Promotion</li>
+ <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+ <li><a href="/contribute/logos/">Logos and Design</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+ <li><a href="/contribute/team/">Team</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+ <p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
+
+<h1 id="authoring-io-transforms---python">Authoring I/O Transforms - Python</h1>
+
+<blockquote>
+ <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
+</blockquote>
+
+<p>TODO - move in the <a href="/documentation/sdks/python-custom-io/">current python SDK content</a></p>
+
+<h1 id="next-steps">Next steps</h1>
+
+<p><a href="/documentation/io/testing/">Testing I/O Transforms</a></p>
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">
+ © Copyright
+ <a href="http://www.apache.org">The Apache Software Foundation</a>,
+ 2017. All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="/privacy_policy">Privacy Policy</a> |
+ <a href="/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ </body>
+
+</html>
http://git-wip-us.apache.org/repos/asf/beam-site/blob/35c2d97b/content/documentation/io/built-in/index.html
----------------------------------------------------------------------
diff --git a/content/documentation/io/built-in/index.html b/content/documentation/io/built-in/index.html
new file mode 100644
index 0000000..fe5405a
--- /dev/null
+++ b/content/documentation/io/built-in/index.html
@@ -0,0 +1,234 @@
+<!DOCTYPE html>
+<html lang="en">
+
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Built-in I/O Transforms</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.
+">
+
+ <link rel="stylesheet" href="/styles/site.css">
+ <link rel="stylesheet" href="/css/theme.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/built-in/" data-proofer-ignore>
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+</head>
+
+
+ <body role="document">
+
+ <nav class="navbar navbar-default navbar-fixed-top">
+ <div class="container">
+ <div class="navbar-header">
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Get Started <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/get-started/beam-overview/">Beam Overview</a></li>
+ <li><a href="/get-started/quickstart-java/">Quickstart - Java</a></li>
+ <li><a href="/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Example Walkthroughs</li>
+ <li><a href="/get-started/wordcount-example/">WordCount</a></li>
+ <li><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Resources</li>
+ <li><a href="/get-started/downloads">Downloads</a></li>
+ <li><a href="/get-started/support">Support</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Documentation <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/documentation">Using the Documentation</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Beam Concepts</li>
+ <li><a href="/documentation/programming-guide/">Programming Guide</a></li>
+ <li><a href="/documentation/resources/">Additional Resources</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Pipeline Fundamentals</li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ <li><a href="/documentation/io/io-toc/">Pipeline I/O</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">SDKs</li>
+ <li><a href="/documentation/sdks/java/">Java SDK</a></li>
+ <li><a href="/documentation/sdks/javadoc/0.6.0/" target="_blank">Java SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li><a href="/documentation/sdks/python/">Python SDK</a></li>
+ <li><a href="/documentation/sdks/pydoc/0.6.0/" target="_blank">Python SDK API Reference <img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Runners</li>
+ <li><a href="/documentation/runners/capability-matrix/">Capability Matrix</a></li>
+ <li><a href="/documentation/runners/direct/">Direct Runner</a></li>
+ <li><a href="/documentation/runners/apex/">Apache Apex Runner</a></li>
+ <li><a href="/documentation/runners/flink/">Apache Flink Runner</a></li>
+ <li><a href="/documentation/runners/spark/">Apache Spark Runner</a></li>
+ <li><a href="/documentation/runners/dataflow/">Cloud Dataflow Runner</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Contribute <span class="caret"></span></a>
+ <ul class="dropdown-menu">
+ <li><a href="/contribute">Get Started Contributing</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Guides</li>
+ <li><a href="/contribute/contribution-guide/">Contribution Guide</a></li>
+ <li><a href="/contribute/testing/">Testing Guide</a></li>
+ <li><a href="/contribute/release-guide/">Release Guide</a></li>
+ <li><a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Technical References</li>
+ <li><a href="/contribute/design-principles/">Design Principles</a></li>
+ <li><a href="/contribute/work-in-progress/">Ongoing Projects</a></li>
+ <li><a href="/contribute/source-repository/">Source Repository</a></li>
+ <li role="separator" class="divider"></li>
+ <li class="dropdown-header">Promotion</li>
+ <li><a href="/contribute/presentation-materials/">Presentation Materials</a></li>
+ <li><a href="/contribute/logos/">Logos and Design</a></li>
+ <li role="separator" class="divider"></li>
+ <li><a href="/contribute/maturity-model/">Maturity Model</a></li>
+ <li><a href="/contribute/team/">Team</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:24px;">Apache Software Foundation<span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+</nav>
+
+
+<link rel="stylesheet" href="">
+
+
+ <div class="container" role="main">
+
+ <div class="row">
+ <p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
+
+<h1 id="built-in-io-transforms">Built-in I/O Transforms</h1>
+
+<p>This table contains the currently available I/O transforms.</p>
+
+<p>Consult the <a href="/documentation/programming-guide#io">Programming Guide I/O section</a> for general usage instructions, and see the javadoc/pydoc for the particular I/O transforms.</p>
+
+<table class="table table-bordered">
+<tr>
+ <th>Language</th>
+ <th>File-based</th>
+ <th>Messaging</th>
+ <th>Database</th>
+</tr>
+<tr>
+ <td>Java</td>
+ <td>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java">AvroIO</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hdfs">Apache Hadoop HDFS</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java">TextIO</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/">XML</a></p>
+ </td>
+ <td>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jms">JMS</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kafka">Apache Kafka</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis">Amazon Kinesis</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io">Google Cloud PubSub</a></p>
+ </td>
+ <td>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/hbase">Apache HBase</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/mongodb">MongoDB</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jdbc">JDBC</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery">Google BigQuery</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable">Google Cloud Bigtable</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore">Google Cloud Datastore</a></p>
+ </td>
+</tr>
+<tr>
+ <td>Python</td>
+ <td>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">avroio</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py">textio</a></p>
+ </td>
+ <td>
+ </td>
+ <td>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py">Google BigQuery</a></p>
+ <p><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/datastore">Google Cloud Datastore</a></p>
+ </td>
+
+</tr>
+</table>
+
+
+ </div>
+
+
+ <hr>
+ <div class="row">
+ <div class="col-xs-12">
+ <footer>
+ <p class="text-center">
+ © Copyright
+ <a href="http://www.apache.org">The Apache Software Foundation</a>,
+ 2017. All Rights Reserved.
+ </p>
+ <p class="text-center">
+ <a href="/privacy_policy">Privacy Policy</a> |
+ <a href="/feed.xml">RSS Feed</a>
+ </p>
+ </footer>
+ </div>
+ </div>
+ <!-- container div end -->
+</div>
+
+
+ </body>
+
+</html>