You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2019/01/08 22:01:08 UTC
[beam] branch asf-site updated: Publishing website 2019/01/08
22:01:03 at commit b06f635
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0467e93 Publishing website 2019/01/08 22:01:03 at commit b06f635
0467e93 is described below
commit 0467e938b010b5c79cc567253eb02d1dcaa70e77
Author: jenkins <bu...@apache.org>
AuthorDate: Tue Jan 8 22:01:04 2019 +0000
Publishing website 2019/01/08 22:01:03 at commit b06f635
---
.../08/20/review-input-streaming-connectors.html | 2 +-
.../dsls/sql/aggregate-functions/index.html | 1 -
.../dsls/sql/create-external-table/index.html | 1 -
.../documentation/dsls/sql/data-types/index.html | 1 -
.../documentation/dsls/sql/joins/index.html | 1 -
.../documentation/dsls/sql/lexical/index.html | 1 -
.../documentation/dsls/sql/overview/index.html | 1 -
.../dsls/sql/scalar-functions/index.html | 1 -
.../documentation/dsls/sql/select/index.html | 1 -
.../documentation/dsls/sql/set/index.html | 1 -
.../documentation/dsls/sql/shell/index.html | 1 -
.../dsls/sql/user-defined-functions/index.html | 1 -
.../documentation/dsls/sql/walkthrough/index.html | 1 -
.../dsls/sql/windowing-and-triggering/index.html | 1 -
.../documentation/execution-model/index.html | 18 +-
website/generated-content/documentation/index.html | 18 +-
.../documentation/io/authoring-java/index.html | 400 +---------
.../documentation/io/authoring-overview/index.html | 497 +------------
.../documentation/io/authoring-python/index.html | 390 +---------
.../io/built-in/google-bigquery/index.html | 18 +-
.../documentation/io/built-in/hadoop/index.html | 20 +-
.../documentation/io/built-in/hcatalog/index.html | 20 +-
.../documentation/io/built-in/index.html | 20 +-
.../documentation/io/contributing/index.html | 384 ----------
.../documentation/io/developing-io-java/index.html | 789 ++++++++++++++++++++
.../index.html | 256 ++++---
.../developing-io-python}/index.html | 501 ++++++-------
.../documentation/io/io-toc/index.html | 406 +----------
.../documentation/io/testing/index.html | 20 +-
.../pipelines/create-your-pipeline/index.html | 18 +-
.../pipelines/design-your-pipeline/index.html | 18 +-
.../pipelines/test-your-pipeline/index.html | 18 +-
.../documentation/programming-guide/index.html | 20 +-
.../resources/learning-resources/index.html | 18 +-
.../resources/videos-and-podcasts/index.html | 18 +-
.../documentation/runners/jstorm/index.html | 18 +-
.../sdks/feature-comparison/index.html | 1 -
.../documentation/sdks/go/index.html | 1 -
.../sdks/java-dependencies/index.html | 1 -
.../documentation/sdks/java-extensions/index.html | 1 -
.../documentation/sdks/java-thirdparty/index.html | 1 -
.../documentation/sdks/java/euphoria/index.html | 1 -
.../documentation/sdks/java/index.html | 1 -
.../sdks/java/testing/nexmark/index.html | 1 -
.../documentation/sdks/python-custom-io/index.html | 809 +--------------------
.../sdks/python-dependencies/index.html | 1 -
.../sdks/python-pipeline-dependencies/index.html | 1 -
.../documentation/sdks/python-streaming/index.html | 1 -
.../sdks/python-type-safety/index.html | 1 -
.../documentation/sdks/python/index.html | 11 +-
website/generated-content/feed.xml | 2 +-
51 files changed, 1435 insertions(+), 3299 deletions(-)
diff --git a/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html b/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
index 9333f39..af3d90b 100644
--- a/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
+++ b/website/generated-content/blog/2018/08/20/review-input-streaming-connectors.html
@@ -265,7 +265,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/spark/stre
</td>
<td>Custom receivers
</td>
- <td><a href="/documentation/io/authoring-overview/#read-transforms">Read Transforms</a>
+ <td><a href="/documentation/io/developing-io-overview/">Read Transforms</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-custom-receivers.html">receiverStream</a>
</td>
diff --git a/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html b/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
index 9e8b123..77e06bc 100644
--- a/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/aggregate-functions/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/create-external-table/index.html b/website/generated-content/documentation/dsls/sql/create-external-table/index.html
index 685e499..13571e9 100644
--- a/website/generated-content/documentation/dsls/sql/create-external-table/index.html
+++ b/website/generated-content/documentation/dsls/sql/create-external-table/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/data-types/index.html b/website/generated-content/documentation/dsls/sql/data-types/index.html
index cf2f374..acfb216 100644
--- a/website/generated-content/documentation/dsls/sql/data-types/index.html
+++ b/website/generated-content/documentation/dsls/sql/data-types/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/joins/index.html b/website/generated-content/documentation/dsls/sql/joins/index.html
index 48d6944..3f705a9 100644
--- a/website/generated-content/documentation/dsls/sql/joins/index.html
+++ b/website/generated-content/documentation/dsls/sql/joins/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/lexical/index.html b/website/generated-content/documentation/dsls/sql/lexical/index.html
index deddbc4..519ca52 100644
--- a/website/generated-content/documentation/dsls/sql/lexical/index.html
+++ b/website/generated-content/documentation/dsls/sql/lexical/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/overview/index.html b/website/generated-content/documentation/dsls/sql/overview/index.html
index 17ac9ef..86bab96 100644
--- a/website/generated-content/documentation/dsls/sql/overview/index.html
+++ b/website/generated-content/documentation/dsls/sql/overview/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/scalar-functions/index.html b/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
index 8d2306e..7d2d05e 100644
--- a/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/scalar-functions/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/select/index.html b/website/generated-content/documentation/dsls/sql/select/index.html
index 29abe90..943e20a 100644
--- a/website/generated-content/documentation/dsls/sql/select/index.html
+++ b/website/generated-content/documentation/dsls/sql/select/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/set/index.html b/website/generated-content/documentation/dsls/sql/set/index.html
index 704cfb0..74b75d1 100644
--- a/website/generated-content/documentation/dsls/sql/set/index.html
+++ b/website/generated-content/documentation/dsls/sql/set/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/shell/index.html b/website/generated-content/documentation/dsls/sql/shell/index.html
index d71bccb..2903a8d 100644
--- a/website/generated-content/documentation/dsls/sql/shell/index.html
+++ b/website/generated-content/documentation/dsls/sql/shell/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html b/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
index 900ce91..3551e7d 100644
--- a/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
+++ b/website/generated-content/documentation/dsls/sql/user-defined-functions/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/walkthrough/index.html b/website/generated-content/documentation/dsls/sql/walkthrough/index.html
index 6a5e5fc..1f06fba 100644
--- a/website/generated-content/documentation/dsls/sql/walkthrough/index.html
+++ b/website/generated-content/documentation/dsls/sql/walkthrough/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html b/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
index ca9654d..adc4e7b 100644
--- a/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
+++ b/website/generated-content/documentation/dsls/sql/windowing-and-triggering/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/execution-model/index.html b/website/generated-content/documentation/execution-model/index.html
index a8c90e9..3d69522 100644
--- a/website/generated-content/documentation/execution-model/index.html
+++ b/website/generated-content/documentation/execution-model/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/index.html b/website/generated-content/documentation/index.html
index 3e0d8f0..1e25b8c 100644
--- a/website/generated-content/documentation/index.html
+++ b/website/generated-content/documentation/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/io/authoring-java/index.html b/website/generated-content/documentation/io/authoring-java/index.html
index 836d43a..216d587 100644
--- a/website/generated-content/documentation/io/authoring-java/index.html
+++ b/website/generated-content/documentation/io/authoring-java/index.html
@@ -1,394 +1,10 @@
<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Authoring I/O Transforms - Java</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/io/authoring-java/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">Languages</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/roadmap/">Roadmap</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- <li>
- <!--
- data-proofer-ignore disables link checking from website test automation.
- GitHub links will not resolve until the markdown source is available on the master branch.
- New pages would fail validation during development / PR test automation.
- -->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/authoring-java.md" data-proofer-ignore>
- <i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
- </a>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Documentation</span></li>
-<li><a href="/documentation">Using the Documentation</a></li>
-<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline development lifecycle</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Beam programming guide</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/">Overview</a></li>
- <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">PCollections</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li>
- <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li>
- <li>
- <span class="section-nav-list-title">Core Beam transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li>
- <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#combine">Combine</a></li>
- <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li>
- <li><a href="/documentation/programming-guide/#partition">Partition</a></li>
- </ul>
- </li>
-
- <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li>
- <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li>
- <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li>
- <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline I/O</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
- <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Data encoding and type safety</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li>
- <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li>
- <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Windowing</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li>
- <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li>
- <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li>
- <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li>
- <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Triggers</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li>
- <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li>
- <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li>
- <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li>
- <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li>
- <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
- </ul>
- </li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Learning Resources</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
- <li><a href="/documentation/resources/learning-resources/#articles">Articles</a></li>
- <li><a href="/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
- <li><a href="/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
- <li><a href="/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
- <li><a href="/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
- <li><a href="/documentation/resources/learning-resources/#how-to-contribute">How to Contribute</a></li>
- <li><a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a></li>
- </ul>
-</li>
-<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki</a></li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#example-io-transforms">Example I/O Transforms</a></li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
-<h1 id="authoring-io-transforms---java">Authoring I/O Transforms - Java</h1>
-
-<blockquote>
- <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
-</blockquote>
-
-<h2 id="example-io-transforms">Example I/O Transforms</h2>
-<p>Currently, Apache Beam’s I/O transforms use a variety of different
-styles. These transforms are good examples to follow:</p>
-<ul>
- <li><a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreIO.java"><code class="highlighter-rouge">DatastoreIO</code></a> - <code class="highlighter-rouge">ParDo</code> based database read and write that conforms to the PTransform style guide</li>
- <li><a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java"><code class="highlighter-rouge">BigtableIO</code></a> - Good test examples, and demonstrates Dynamic Work Rebalancing</li>
- <li><a href="https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java"><code class="highlighter-rouge">JdbcIO</code></a> - Demonstrates reading using single <code class="highlighter-rouge">ParDo</code>+<code class="highlighter-rouge">GroupByKey</code> when data stores cannot be read in parallel</li>
-</ul>
-
-<h1 id="next-steps">Next steps</h1>
-
-<p><a href="/documentation/io/testing/">Testing I/O Transforms</a></p>
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
+<html lang="en-US">
+<meta charset="utf-8">
+<title>Redirecting…</title>
+<link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-java/">
+<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/io/developing-io-java/">
+<h1>Redirecting…</h1>
+<a href="https://beam.apache.org/documentation/io/developing-io-java/">Click here if you are not redirected.</a>
+<script>location="https://beam.apache.org/documentation/io/developing-io-java/"</script>
</html>
diff --git a/website/generated-content/documentation/io/authoring-overview/index.html b/website/generated-content/documentation/io/authoring-overview/index.html
index 99d28d8..d3e173b 100644
--- a/website/generated-content/documentation/io/authoring-overview/index.html
+++ b/website/generated-content/documentation/io/authoring-overview/index.html
@@ -1,491 +1,10 @@
<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Authoring I/O Transforms - Overview</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/io/authoring-overview/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">Languages</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/roadmap/">Roadmap</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- <li>
- <!--
- data-proofer-ignore disables link checking from website test automation.
- GitHub links will not resolve until the markdown source is available on the master branch.
- New pages would fail validation during development / PR test automation.
- -->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/authoring-overview.md" data-proofer-ignore>
- <i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
- </a>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Documentation</span></li>
-<li><a href="/documentation">Using the Documentation</a></li>
-<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline development lifecycle</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Beam programming guide</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/">Overview</a></li>
- <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">PCollections</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li>
- <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li>
- <li>
- <span class="section-nav-list-title">Core Beam transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li>
- <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#combine">Combine</a></li>
- <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li>
- <li><a href="/documentation/programming-guide/#partition">Partition</a></li>
- </ul>
- </li>
-
- <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li>
- <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li>
- <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li>
- <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline I/O</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
- <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Data encoding and type safety</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li>
- <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li>
- <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Windowing</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li>
- <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li>
- <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li>
- <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li>
- <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Triggers</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li>
- <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li>
- <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li>
- <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li>
- <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li>
- <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
- </ul>
- </li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Learning Resources</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
- <li><a href="/documentation/resources/learning-resources/#articles">Articles</a></li>
- <li><a href="/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
- <li><a href="/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
- <li><a href="/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
- <li><a href="/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
- <li><a href="/documentation/resources/learning-resources/#how-to-contribute">How to Contribute</a></li>
- <li><a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a></li>
- </ul>
-</li>
-<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki</a></li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#introduction">Introduction</a></li>
- <li><a href="#suggested-steps-for-implementers">Suggested steps for implementers</a></li>
- <li><a href="#read-transforms">Read transforms</a>
- <ul>
- <li><a href="#when-to-implement-using-the-source-api">When to implement using the <code class="highlighter-rouge">Source</code> API</a></li>
- </ul>
- </li>
- <li><a href="#write-transforms">Write transforms</a>
- <ul>
- <li><a href="#when-to-implement-using-the-sink-api">When to implement using the <code class="highlighter-rouge">Sink</code> API</a></li>
- </ul>
- </li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
-<h1 id="authoring-io-transforms---overview">Authoring I/O Transforms - Overview</h1>
-
-<p><em>A guide for users who need to connect to a data store that isn’t supported by the <a href="/documentation/io/built-in/">Built-in I/O Transforms</a></em></p>
-
-<ul id="markdown-toc">
- <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
- <li><a href="#suggested-steps-for-implementers" id="markdown-toc-suggested-steps-for-implementers">Suggested steps for implementers</a></li>
- <li><a href="#read-transforms" id="markdown-toc-read-transforms">Read transforms</a> <ul>
- <li><a href="#when-to-implement-using-the-source-api" id="markdown-toc-when-to-implement-using-the-source-api">When to implement using the <code class="highlighter-rouge">Source</code> API</a></li>
- </ul>
- </li>
- <li><a href="#write-transforms" id="markdown-toc-write-transforms">Write transforms</a> <ul>
- <li><a href="#when-to-implement-using-the-sink-api" id="markdown-toc-when-to-implement-using-the-sink-api">When to implement using the <code class="highlighter-rouge">Sink</code> API</a></li>
- </ul>
- </li>
-</ul>
-
-<h2 id="introduction">Introduction</h2>
-<p>This guide covers how to implement I/O transforms in the Beam model. Beam pipelines use these read and write transforms to import data for processing, and write data to a store.</p>
-
-<p>Reading and writing data in Beam is a parallel task, and using <code class="highlighter-rouge">ParDo</code>s, <code class="highlighter-rouge">GroupByKey</code>s, etc… is usually sufficient. Rarely, you will need the more specialized <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Sink</code> classes for specific features. There are changes coming soon (<code class="highlighter-rouge">SplittableDoFn</code>, <a href="https://issues.apache.org/jira/brows [...]
-
-<p>As you work on your I/O Transform, be aware that the Beam community is excited to help those building new I/O Transforms and that there are many examples and helper classes.</p>
-
-<h2 id="suggested-steps-for-implementers">Suggested steps for implementers</h2>
-<ol>
- <li>Check out this guide and come up with your design. If you’d like, you can email the <a href="/get-started/support">Beam dev mailing list</a> with any questions you might have. It’s good to check there to see if anyone else is working on the same I/O Transform.</li>
- <li>If you are planning to contribute your I/O transform to the Beam community, you’ll be going through the normal Beam contribution life cycle - see the <a href="/contribute/contribution-guide/">Apache Beam Contribution Guide</a> for more details.</li>
- <li>As you’re working on your IO transform, see the <a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a> for specific information about writing I/O Transforms.</li>
-</ol>
-
-<h2 id="read-transforms">Read transforms</h2>
-<p>Read transforms take data from outside of the Beam pipeline and produce <code class="highlighter-rouge">PCollection</code>s of data.</p>
-
-<p>For data stores or file types where the data can be read in parallel, you can think of the process as a mini-pipeline. This often consists of two steps:</p>
-<ol>
- <li>Splitting the data into parts to be read in parallel</li>
- <li>Reading from each of those parts</li>
-</ol>
-
-<p>Each of those steps will be a <code class="highlighter-rouge">ParDo</code>, with a <code class="highlighter-rouge">GroupByKey</code> in between. The <code class="highlighter-rouge">GroupByKey</code> is an implementation detail, but for most runners it allows the runner to use different numbers of workers for:</p>
-<ul>
- <li>Determining how to split up the data to be read into chunks - this will likely occur on very few workers</li>
- <li>Reading - will likely benefit from more workers</li>
-</ul>
-
-<p>The <code class="highlighter-rouge">GroupByKey</code> will also allow Dynamic Work Rebalancing to occur (on supported runners).</p>
-
-<p>Here are some examples of read transform implementations that use the “reading as a mini-pipeline” model when data can be read in parallel:</p>
-<ul>
- <li><strong>Reading from a file glob</strong> - For example reading all files in “~/data/**”
- <ul>
- <li>Get File Paths <code class="highlighter-rouge">ParDo</code>: As input, take in a file glob. Produce a <code class="highlighter-rouge">PCollection</code> of strings, each of which is a file path.</li>
- <li>Reading <code class="highlighter-rouge">ParDo</code>: Given the <code class="highlighter-rouge">PCollection</code> of file paths, read each one, producing a <code class="highlighter-rouge">PCollection</code> of records.</li>
- </ul>
- </li>
- <li><strong>Reading from a NoSQL Database</strong> (eg Apache HBase) - these databases often allow reading from ranges in parallel.
- <ul>
- <li>Determine Key Ranges <code class="highlighter-rouge">ParDo</code>: As input, receive connection information for the database and the key range to read from. Produce a <code class="highlighter-rouge">PCollection</code> of key ranges that can be read in parallel efficiently.</li>
- <li>Read Key Range <code class="highlighter-rouge">ParDo</code>: Given the <code class="highlighter-rouge">PCollection</code> of key ranges, read the key range, producing a <code class="highlighter-rouge">PCollection</code> of records.</li>
- </ul>
- </li>
-</ul>
-
-<p>For data stores or files where reading cannot occur in parallel, reading is a simple task that can be accomplished with a single <code class="highlighter-rouge">ParDo</code>+<code class="highlighter-rouge">GroupByKey</code>. For example:</p>
-<ul>
- <li><strong>Reading from a database query</strong> - traditional SQL database queries often can only be read in sequence. The <code class="highlighter-rouge">ParDo</code> in this case would establish a connection to the database and read batches of records, producing a <code class="highlighter-rouge">PCollection</code> of those records.</li>
- <li><strong>Reading from a gzip file</strong> - a gzip file has to be read in order, so it cannot be parallelized. The <code class="highlighter-rouge">ParDo</code> in this case would open the file and read in sequence, producing a <code class="highlighter-rouge">PCollection</code> of records from the file.</li>
-</ul>
-
-<h3 id="when-to-implement-using-the-source-api">When to implement using the <code class="highlighter-rouge">Source</code> API</h3>
-<p>The above discussion is in terms of <code class="highlighter-rouge">ParDo</code>s - this is because <code class="highlighter-rouge">Source</code>s have proven to be tricky to implement. At this point in time, the recommendation is to <strong>use <code class="highlighter-rouge">Source</code> only if <code class="highlighter-rouge">ParDo</code> doesn’t meet your needs</strong>. A class derived from <code class="highlighter-rouge">FileBasedSource</code> is often the best option when rea [...]
-
-<p>If you’re trying to decide on whether or not to use <code class="highlighter-rouge">Source</code>, feel free to email the <a href="/get-started/support">Beam dev mailing list</a> and we can discuss the specific pros and cons of your case.</p>
-
-<p>In some cases implementing a <code class="highlighter-rouge">Source</code> may be necessary or result in better performance.</p>
-<ul>
- <li><code class="highlighter-rouge">ParDo</code>s will not work for reading from unbounded sources - they do not support checkpointing and don’t support mechanisms like de-duping that have proven useful for streaming data sources.</li>
- <li><code class="highlighter-rouge">ParDo</code>s cannot provide hints to runners about their progress or the size of data they are reading - without size estimation of the data or progress on your read, the runner doesn’t have any way to guess how large your read will be, and thus if it attempts to dynamically allocate workers, it does not have any clues as to how many workers you may need for your pipeline.</li>
- <li><code class="highlighter-rouge">ParDo</code>s do not support Dynamic Work Rebalancing - these are features used by some readers to improve the processing speed of jobs (but may not be possible with your data source).</li>
- <li><code class="highlighter-rouge">ParDo</code>s do not receive ‘desired_bundle_size’ as a hint from runners when performing initial splitting.
-<code class="highlighter-rouge">SplittableDoFn</code> (<a href="https://issues.apache.org/jira/browse/BEAM-65">BEAM-65</a>) will mitigate many of these concerns.</li>
-</ul>
-
-<h2 id="write-transforms">Write transforms</h2>
-<p>Write transforms are responsible for taking the contents of a <code class="highlighter-rouge">PCollection</code> and transferring that data outside of the Beam pipeline.</p>
-
-<p>Write transforms can usually be implemented using a single <code class="highlighter-rouge">ParDo</code> that writes the records received to the data store.</p>
-
-<p>TODO: this section needs further explanation.</p>
-
-<h3 id="when-to-implement-using-the-sink-api">When to implement using the <code class="highlighter-rouge">Sink</code> API</h3>
-<p>You are strongly discouraged from using the <code class="highlighter-rouge">Sink</code> class unless you are creating a <code class="highlighter-rouge">FileBasedSink</code>. Most of the time, a simple <code class="highlighter-rouge">ParDo</code> is all that’s necessary. If you think you have a case that is only possible using a <code class="highlighter-rouge">Sink</code>, please email the <a href="/get-started/support">Beam dev mailing list</a>.</p>
-
-<h1 id="next-steps">Next steps</h1>
-
-<p>This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
-
-<!-- TODO: commented out until this content is ready.
-For more details on actual implementation, continue with one of the the language specific guides:
-
-* [Authoring I/O Transforms - Python](/documentation/io/authoring-python/)
-* [Authoring I/O Transforms - Java](/documentation/io/authoring-java/)
--->
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
+<html lang="en-US">
+<meta charset="utf-8">
+<title>Redirecting…</title>
+<link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-overview/">
+<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/io/developing-io-overview/">
+<h1>Redirecting…</h1>
+<a href="https://beam.apache.org/documentation/io/developing-io-overview/">Click here if you are not redirected.</a>
+<script>location="https://beam.apache.org/documentation/io/developing-io-overview/"</script>
</html>
diff --git a/website/generated-content/documentation/io/authoring-python/index.html b/website/generated-content/documentation/io/authoring-python/index.html
index 60fd5c1..e74d71d 100644
--- a/website/generated-content/documentation/io/authoring-python/index.html
+++ b/website/generated-content/documentation/io/authoring-python/index.html
@@ -1,384 +1,10 @@
<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Authoring I/O Transforms - Python</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/io/authoring-python/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">Languages</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/roadmap/">Roadmap</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- <li>
- <!--
- data-proofer-ignore disables link checking from website test automation.
- GitHub links will not resolve until the markdown source is available on the master branch.
- New pages would fail validation during development / PR test automation.
- -->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/authoring-python.md" data-proofer-ignore>
- <i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
- </a>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Documentation</span></li>
-<li><a href="/documentation">Using the Documentation</a></li>
-<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline development lifecycle</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Beam programming guide</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/">Overview</a></li>
- <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">PCollections</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li>
- <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li>
- <li>
- <span class="section-nav-list-title">Core Beam transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li>
- <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#combine">Combine</a></li>
- <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li>
- <li><a href="/documentation/programming-guide/#partition">Partition</a></li>
- </ul>
- </li>
-
- <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li>
- <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li>
- <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li>
- <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline I/O</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
- <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Data encoding and type safety</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li>
- <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li>
- <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Windowing</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li>
- <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li>
- <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li>
- <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li>
- <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Triggers</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li>
- <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li>
- <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li>
- <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li>
- <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li>
- <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
- </ul>
- </li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Learning Resources</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
- <li><a href="/documentation/resources/learning-resources/#articles">Articles</a></li>
- <li><a href="/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
- <li><a href="/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
- <li><a href="/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
- <li><a href="/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
- <li><a href="/documentation/resources/learning-resources/#how-to-contribute">How to Contribute</a></li>
- <li><a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a></li>
- </ul>
-</li>
-<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki</a></li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
-<h1 id="authoring-io-transforms---python">Authoring I/O Transforms - Python</h1>
-
-<blockquote>
- <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
-</blockquote>
-
-<p>TODO - move in the <a href="/documentation/sdks/python-custom-io/">current python SDK content</a></p>
-
-<h1 id="next-steps">Next steps</h1>
-
-<p><a href="/documentation/io/testing/">Testing I/O Transforms</a></p>
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
+<html lang="en-US">
+<meta charset="utf-8">
+<title>Redirecting…</title>
+<link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-python/">
+<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/io/developing-io-python/">
+<h1>Redirecting…</h1>
+<a href="https://beam.apache.org/documentation/io/developing-io-python/">Click here if you are not redirected.</a>
+<script>location="https://beam.apache.org/documentation/io/developing-io-python/"</script>
</html>
diff --git a/website/generated-content/documentation/io/built-in/google-bigquery/index.html b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
index 8d58f00..47dd15c 100644
--- a/website/generated-content/documentation/io/built-in/google-bigquery/index.html
+++ b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/io/built-in/hadoop/index.html b/website/generated-content/documentation/io/built-in/hadoop/index.html
index ae21e57..19c4b65 100644
--- a/website/generated-content/documentation/io/built-in/hadoop/index.html
+++ b/website/generated-content/documentation/io/built-in/hadoop/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
@@ -306,8 +316,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
<h1 id="hadoop-inputformat-io">Hadoop InputFormat IO</h1>
<p>A <code class="highlighter-rouge">HadoopInputFormatIO</code> is a transform for reading data from any source that implements Hadoop’s <code class="highlighter-rouge">InputFormat</code>. For example, Cassandra, Elasticsearch, HBase, Redis, Postgres, etc.</p>
diff --git a/website/generated-content/documentation/io/built-in/hcatalog/index.html b/website/generated-content/documentation/io/built-in/hcatalog/index.html
index be2299c..58de197 100644
--- a/website/generated-content/documentation/io/built-in/hcatalog/index.html
+++ b/website/generated-content/documentation/io/built-in/hcatalog/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
@@ -303,8 +313,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
<h1 id="hcatalog-io">HCatalog IO</h1>
<p>An <code class="highlighter-rouge">HCatalogIO</code> is a transform for reading and writing data to an HCatalog managed source.</p>
diff --git a/website/generated-content/documentation/io/built-in/index.html b/website/generated-content/documentation/io/built-in/index.html
index 92d4ff9..a208d00 100644
--- a/website/generated-content/documentation/io/built-in/index.html
+++ b/website/generated-content/documentation/io/built-in/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
@@ -298,8 +308,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
<h1 id="built-in-io-transforms">Built-in I/O Transforms</h1>
<p>This table contains the currently available I/O transforms.</p>
diff --git a/website/generated-content/documentation/io/contributing/index.html b/website/generated-content/documentation/io/contributing/index.html
deleted file mode 100644
index 6aa620d..0000000
--- a/website/generated-content/documentation/io/contributing/index.html
+++ /dev/null
@@ -1,384 +0,0 @@
-<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Contributing I/O Transforms</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/io/contributing/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">Languages</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/roadmap/">Roadmap</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- <li>
- <!--
- data-proofer-ignore disables link checking from website test automation.
- GitHub links will not resolve until the markdown source is available on the master branch.
- New pages would fail validation during development / PR test automation.
- -->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/contributing.md" data-proofer-ignore>
- <i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
- </a>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Documentation</span></li>
-<li><a href="/documentation">Using the Documentation</a></li>
-<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline development lifecycle</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Beam programming guide</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/">Overview</a></li>
- <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">PCollections</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li>
- <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li>
- <li>
- <span class="section-nav-list-title">Core Beam transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li>
- <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#combine">Combine</a></li>
- <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li>
- <li><a href="/documentation/programming-guide/#partition">Partition</a></li>
- </ul>
- </li>
-
- <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li>
- <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li>
- <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li>
- <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline I/O</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
- <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Data encoding and type safety</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li>
- <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li>
- <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Windowing</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li>
- <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li>
- <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li>
- <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li>
- <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Triggers</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li>
- <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li>
- <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li>
- <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li>
- <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li>
- <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
- </ul>
- </li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Learning Resources</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
- <li><a href="/documentation/resources/learning-resources/#articles">Articles</a></li>
- <li><a href="/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
- <li><a href="/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
- <li><a href="/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
- <li><a href="/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
- <li><a href="/documentation/resources/learning-resources/#how-to-contribute">How to Contribute</a></li>
- <li><a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a></li>
- </ul>
-</li>
-<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki</a></li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
-<h1 id="contributing-io-transforms">Contributing I/O Transforms</h1>
-
-<ul>
- <li>If you are planning to contribute your I/O transform to the Apache Beam community, you’ll be going through the normal Beam contribution life cycle - see the <a href="/contribute/contribution-guide/">Apache Beam Contribution Guide</a> for more details.</li>
- <li>Talk to the community!</li>
- <li>Make sure you’ve implemented the appropriate tests as discussed in the <a href="/documentation/io/testing/">Testing I/O Transforms</a> section.</li>
-</ul>
-
-<blockquote>
- <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
-</blockquote>
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
-</html>
diff --git a/website/generated-content/documentation/io/developing-io-java/index.html b/website/generated-content/documentation/io/developing-io-java/index.html
new file mode 100644
index 0000000..31d11de
--- /dev/null
+++ b/website/generated-content/documentation/io/developing-io-java/index.html
@@ -0,0 +1,789 @@
+<!DOCTYPE html>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<html lang="en">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <title>Apache Beam: Developing I/O connectors for Java</title>
+ <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
+">
+ <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
+ <link rel="stylesheet" href="/css/site.css">
+ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="/js/bootstrap.min.js"></script>
+ <script src="/js/language-switch.js"></script>
+ <script src="/js/fix-menu.js"></script>
+ <script src="/js/section-nav.js"></script>
+ <script src="/js/page-nav.js"></script>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-java/" data-proofer-ignore>
+ <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
+ <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
+ <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-73650088-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+</head>
+
+ <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<nav class="header navbar navbar-fixed-top">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ <a href="/" class="navbar-brand" >
+ <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
+ </a>
+ </div>
+
+ <div class="navbar-mask closed"></div>
+
+ <div id="navbar" class="navbar-container closed">
+ <ul class="nav navbar-nav">
+ <li>
+ <a href="/get-started/beam-overview/">Get Started</a>
+ </li>
+ <li>
+ <a href="/documentation/">Documentation</a>
+ </li>
+ <li>
+ <a href="/documentation/sdks/java/">Languages</a>
+ </li>
+ <li>
+ <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
+ </li>
+ <li>
+ <a href="/roadmap/">Roadmap</a>
+ </li>
+ <li>
+ <a href="/contribute/">Contribute</a>
+ </li>
+ <li>
+ <a href="/community/contact-us/">Community</a>
+ </li>
+ <li><a href="/blog">Blog</a></li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
+ <ul class="dropdown-menu dropdown-menu-right">
+ <li><a href="http://www.apache.org/">ASF Homepage</a></li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
+ </ul>
+ </li>
+ <li>
+ <!--
+ data-proofer-ignore disables link checking from website test automation.
+ GitHub links will not resolve until the markdown source is available on the master branch.
+ New pages would fail validation during development / PR test automation.
+ -->
+ <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/developing-io-java.md" data-proofer-ignore>
+ <i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
+ </a>
+ </li>
+ </ul>
+ </div>
+</nav>
+
+ <div class="clearfix container-main-content">
+ <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
+ <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
+ <nav>
+ <ul class="section-nav-list" data-section-nav>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<li><span class="section-nav-list-main-title">Documentation</span></li>
+<li><a href="/documentation">Using the Documentation</a></li>
+<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
+<li>
+ <span class="section-nav-list-title">Pipeline development lifecycle</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
+ </ul>
+</li>
+<li>
+ <span class="section-nav-list-title">Beam programming guide</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/">Overview</a></li>
+ <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">PCollections</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li>
+ <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Transforms</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li>
+ <li>
+ <span class="section-nav-list-title">Core Beam transforms</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li>
+ <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li>
+ <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li>
+ <li><a href="/documentation/programming-guide/#combine">Combine</a></li>
+ <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li>
+ <li><a href="/documentation/programming-guide/#partition">Partition</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li>
+ <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li>
+ <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li>
+ <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Pipeline I/O</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Data encoding and type safety</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li>
+ <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li>
+ <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Windowing</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li>
+ <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li>
+ <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li>
+ <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li>
+ <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Triggers</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li>
+ <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li>
+ <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li>
+ <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li>
+ <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li>
+ <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
+ </ul>
+ </li>
+ </ul>
+</li>
+<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Learning Resources</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
+ <li><a href="/documentation/resources/learning-resources/#articles">Articles</a></li>
+ <li><a href="/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
+ <li><a href="/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
+ <li><a href="/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
+ <li><a href="/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
+ <li><a href="/documentation/resources/learning-resources/#how-to-contribute">How to Contribute</a></li>
+ <li><a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a></li>
+ </ul>
+</li>
+<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki</a></li>
+
+ </ul>
+ </nav>
+ </div>
+
+ <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+
+
+<ul class="nav">
+ <li><a href="#basic-code-reqs">Basic code requirements</a></li>
+ <li><a href="#implementing-the-source-interface">Implementing the Source interface</a>
+ <ul>
+ <li><a href="#implementing-the-source-subclass">Implementing the Source subclass</a></li>
+ <li><a href="#implementing-the-reader-subclass">Implementing the Reader subclass</a></li>
+ <li><a href="#convenience-source-and-reader-base-classes">Convenience Source and Reader base classes</a></li>
+ </ul>
+ </li>
+ <li><a href="#using-filebasedsink">Using the FileBasedSink abstraction</a></li>
+ <li><a href="#ptransform-wrappers">PTransform wrappers</a></li>
+</ul>
+
+
+ </nav>
+
+ <div class="body__contained body__section-nav">
+ <!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<h1 id="developing-io-connectors-for-java">Developing I/O connectors for Java</h1>
+
+<p>To connect to a data store that isn’t supported by Beam’s existing I/O
+connectors, you must create a custom I/O connector that usually consist of a
+source and a sink. All Beam sources and sinks are composite transforms; however,
+the implementation of your custom I/O depends on your use case. Before you
+start, read the
+<a href="/documentation/io/developing-io-overview/">new I/O connector overview</a>
+for an overview of developing a new I/O connector, the available implementation
+options, and how to choose the right option for your use case.</p>
+
+<p>This guide covers using the <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">FileBasedSink</code> interfaces using Java.
+The Python SDK offers the same functionality, but uses a slightly different API.
+See <a href="/documentation/io/developing-io-python/">Developing I/O connectors for Python</a>
+for information specific to the Python SDK.</p>
+
+<h2 id="basic-code-reqs">Basic code requirements</h2>
+
+<p>Beam runners use the classes you provide to read and/or write data using
+multiple worker instances in parallel. As such, the code you provide for
+<code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">FileBasedSink</code> subclasses must meet some basic requirements:</p>
+
+<ol>
+ <li>
+ <p><strong>Serializability:</strong> Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">FileBasedSink</code> subclass, whether
+bounded or unbounded, must be Serializable. A runner might create multiple
+instances of your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">FileBasedSink</code> subclass to be sent to
+multiple remote workers to facilitate reading or writing in parallel.</p>
+ </li>
+ <li>
+ <p><strong>Immutability:</strong>
+Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">FileBasedSink</code> subclass must be effectively immutable.
+All private fields must be declared final, and all private variables of
+collection type must be effectively immutable. If your class has setter
+methods, those methods must return an independent copy of the object with
+the relevant field modified.</p>
+
+ <p>You should only use mutable state in your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">FileBasedSink</code>
+subclass if you are using lazy evaluation of expensive computations that
+you need to implement the source or sink; in that case, you must declare
+all mutable instance variables transient.</p>
+ </li>
+ <li>
+ <p><strong>Thread-Safety:</strong> Your code must be thread-safe. If you build your source
+to work with dynamic work rebalancing, it is critical that you make your
+code thread-safe. The Beam SDK provides a helper class to make this easier.
+See <a href="#bounded-dynamic">Using Your BoundedSource with dynamic work rebalancing</a>
+for more details.</p>
+ </li>
+ <li>
+ <p><strong>Testability:</strong> It is critical to exhaustively unit test all of your
+<code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">FileBasedSink</code> subclasses, especially if you build your
+classes to work with advanced features such as dynamic work rebalancing. A
+minor implementation error can lead to data corruption or data loss (such
+as skipping or duplicating records) that can be hard to detect.</p>
+
+ <p>To assist in testing <code class="highlighter-rouge">BoundedSource</code> implementations, you can use the
+SourceTestUtils class. <code class="highlighter-rouge">SourceTestUtils</code> contains utilities for automatically
+verifying some of the properties of your <code class="highlighter-rouge">BoundedSource</code> implementation. You
+can use <code class="highlighter-rouge">SourceTestUtils</code> to increase your implementation’s test coverage
+using a wide range of inputs with relatively few lines of code. For
+examples that use <code class="highlighter-rouge">SourceTestUtils</code>, see the
+<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroSourceTest.java">AvroSourceTest</a> and
+<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java">TextIOReadTest</a>
+source code.</p>
+ </li>
+</ol>
+
+<p>In addition, see the <a href="/contribute/ptransform-style-guide/">PTransform style guide</a>
+for Beam’s transform style guidance.</p>
+
+<h2 id="implementing-the-source-interface">Implementing the Source interface</h2>
+
+<p>To create a data source for your pipeline, you must provide the format-specific
+logic that tells a runner how to read data from your input source, and how to
+split your data source into multiple parts so that multiple worker instances can
+read your data in parallel. If you’re creating a data source that reads
+unbounded data, you must provide additional logic for managing your source’s
+watermark and optional checkpointing.</p>
+
+<p>Supply the logic for your source by creating the following classes:</p>
+
+<ul>
+ <li>
+ <p>A subclass of <code class="highlighter-rouge">BoundedSource</code> if you want to read a finite (batch) data set,
+or a subclass of <code class="highlighter-rouge">UnboundedSource</code> if you want to read an infinite (streaming)
+data set. These subclasses describe the data you want to read, including the
+data’s location and parameters (such as how much data to read).</p>
+ </li>
+ <li>
+ <p>A subclass of <code class="highlighter-rouge">Source.Reader</code>. Each Source must have an associated Reader that
+captures all the state involved in reading from that <code class="highlighter-rouge">Source</code>. This can
+include things like file handles, RPC connections, and other parameters that
+depend on the specific requirements of the data format you want to read.</p>
+ </li>
+ <li>
+ <p>The <code class="highlighter-rouge">Reader</code> class hierarchy mirrors the Source hierarchy. If you’re extending
+<code class="highlighter-rouge">BoundedSource</code>, you’ll need to provide an associated <code class="highlighter-rouge">BoundedReader</code>. if you’re
+extending <code class="highlighter-rouge">UnboundedSource</code>, you’ll need to provide an associated
+<code class="highlighter-rouge">UnboundedReader</code>.</p>
+ </li>
+ <li>
+ <p>One or more user-facing wrapper composite transforms (<code class="highlighter-rouge">PTransform</code>) that
+wrap read operations. <a href="#ptransform-wrappers">PTransform wrappers</a> discusses
+why you should avoid exposing your sources.</p>
+ </li>
+</ul>
+
+<h3 id="implementing-the-source-subclass">Implementing the Source subclass</h3>
+
+<p>You must create a subclass of either <code class="highlighter-rouge">BoundedSource</code> or <code class="highlighter-rouge">UnboundedSource</code>,
+depending on whether your data is a finite batch or an infinite stream. In
+either case, your <code class="highlighter-rouge">Source</code> subclass must override the abstract methods in the
+superclass. A runner might call these methods when using your data source. For
+example, when reading from a bounded source, a runner uses these methods to
+estimate the size of your data set and to split it up for parallel reading.</p>
+
+<p>Your <code class="highlighter-rouge">Source</code> subclass should also manage basic information about your data
+source, such as the location. For example, the example <code class="highlighter-rouge">Source</code> implementation
+in Beam’s <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/io/gcp/datastore/DatastoreIO.html">DatastoreIO</a>
+class takes host, datasetID, and query as arguments. The connector uses these
+values to obtain data from Cloud Datastore.</p>
+
+<h4 id="boundedsource">BoundedSource</h4>
+
+<p><code class="highlighter-rouge">BoundedSource</code> represents a finite data set from which a Beam runner may read,
+possibly in parallel. <code class="highlighter-rouge">BoundedSource</code> contains a set of abstract methods that
+the runner uses to split the data set for reading by multiple workers.</p>
+
+<p>To implement a <code class="highlighter-rouge">BoundedSource</code>, your subclass must override the following
+abstract methods:</p>
+
+<ul>
+ <li>
+ <p><code class="highlighter-rouge">split</code>: The runner uses this method to split your finite data
+into bundles of a given size.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">getEstimatedSizeBytes</code>: The runner uses this method to estimate the total
+size of your data, in bytes.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">createReader</code>: Creates the associated <code class="highlighter-rouge">BoundedReader</code> for this
+<code class="highlighter-rouge">BoundedSource</code>.</p>
+ </li>
+</ul>
+
+<p>You can see a model of how to implement <code class="highlighter-rouge">BoundedSource</code> and the required
+abstract methods in Beam’s implementations for Cloud BigTable
+(<a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java">BigtableIO.java</a>)
+and BigQuery (<a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java">BigQuerySourceBase.java</a>).</p>
+
+<h4 id="unboundedsource">UnboundedSource</h4>
+
+<p><code class="highlighter-rouge">UnboundedSource</code> represents an infinite data stream from which the runner may
+read, possibly in parallel. <code class="highlighter-rouge">UnboundedSource</code> contains a set of abstract methods
+that the runner uses to support streaming reads in parallel; these include
+<em>checkpointing</em> for failure recovery, <em>record IDs</em> to prevent data duplication,
+and <em>watermarking</em> for estimating data completeness in downstream parts of your
+pipeline.</p>
+
+<p>To implement an <code class="highlighter-rouge">UnboundedSource</code>, your subclass must override the following
+abstract methods:</p>
+
+<ul>
+ <li>
+ <p><code class="highlighter-rouge">split</code>: The runner uses this method to generate a list of
+<code class="highlighter-rouge">UnboundedSource</code> objects which represent the number of sub-stream instances
+from which the service should read in parallel.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">getCheckpointMarkCoder</code>: The runner uses this method to obtain the Coder for
+the checkpoints for your source (if any).</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">requiresDeduping</code>: The runner uses this method to determine whether the data
+requires explicit removal of duplicate records. If this method returns true,
+the runner will automatically insert a step to remove duplicates from your
+source’s output. This should return true if and only if your source
+provides record IDs for each record. See <code class="highlighter-rouge">UnboundedReader.getCurrentRecordId</code>
+for when this should be done.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">createReader</code>: Creates the associated <code class="highlighter-rouge">UnboundedReader</code> for this
+<code class="highlighter-rouge">UnboundedSource</code>.</p>
+ </li>
+</ul>
+
+<h3 id="implementing-the-reader-subclass">Implementing the Reader subclass</h3>
+
+<p>You must create a subclass of either <code class="highlighter-rouge">BoundedReader</code> or <code class="highlighter-rouge">UnboundedReader</code> to be
+returned by your source subclass’s <code class="highlighter-rouge">createReader</code> method. The runner uses the
+methods in your <code class="highlighter-rouge">Reader</code> (whether bounded or unbounded) to do the actual reading
+of your dataset.</p>
+
+<p><code class="highlighter-rouge">BoundedReader</code> and <code class="highlighter-rouge">UnboundedReader</code> have similar basic interfaces, which
+you’ll need to define. In addition, there are some additional methods unique to
+<code class="highlighter-rouge">UnboundedReader</code> that you’ll need to implement for working with unbounded data,
+and an optional method you can implement if you want your <code class="highlighter-rouge">BoundedReader</code> to
+take advantage of dynamic work rebalancing. There are also minor differences in
+the semantics for the <code class="highlighter-rouge">start()</code> and <code class="highlighter-rouge">advance()</code> methods when using
+<code class="highlighter-rouge">UnboundedReader</code>.</p>
+
+<h4 id="reader-methods-common-to-both-boundedreader-and-unboundedreader">Reader methods common to both BoundedReader and UnboundedReader</h4>
+
+<p>A runner uses the following methods to read data using <code class="highlighter-rouge">BoundedReader</code> or
+<code class="highlighter-rouge">UnboundedReader</code>:</p>
+
+<ul>
+ <li>
+ <p><code class="highlighter-rouge">start</code>: Initializes the <code class="highlighter-rouge">Reader</code> and advances to the first record to be read.
+This method is called exactly once when the runner begins reading your data,
+and is a good place to put expensive operations needed for initialization.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">advance</code>: Advances the reader to the next valid record. This method must
+return false if there is no more input available. <code class="highlighter-rouge">BoundedReader</code> should stop
+reading once advance returns false, but <code class="highlighter-rouge">UnboundedReader</code> can return true in
+future calls once more data is available from your stream.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">getCurrent</code>: Returns the data record at the current position, last read by
+start or advance.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">getCurrentTimestamp</code>: Returns the timestamp for the current data record. You
+only need to override <code class="highlighter-rouge">getCurrentTimestamp</code> if your source reads data that has
+intrinsic timestamps. The runner uses this value to set the intrinsic
+timestamp for each element in the resulting output <code class="highlighter-rouge">PCollection</code>.</p>
+ </li>
+</ul>
+
+<h4 id="reader-methods-unique-to-unboundedreader">Reader methods unique to UnboundedReader</h4>
+
+<p>In addition to the basic <code class="highlighter-rouge">Reader</code> interface, <code class="highlighter-rouge">UnboundedReader</code> has some
+additional methods for managing reads from an unbounded data source:</p>
+
+<ul>
+ <li>
+ <p><code class="highlighter-rouge">getCurrentRecordId</code>: Returns a unique identifier for the current record.
+The runner uses these record IDs to filter out duplicate records. If your
+data has logical IDs present in each record, you can have this method return
+them; otherwise, you can return a hash of the record contents, using at
+least a 128-bit hash. It is incorrect to use Java’s <code class="highlighter-rouge">Object.hashCode()</code>, as
+a 32-bit hash is generally insufficient for preventing collisions, and
+<code class="highlighter-rouge">hasCode()</code> is not guaranteed to be stable across processes.</p>
+
+ <p>Implementing <code class="highlighter-rouge">getCurrentRecordId</code> is optional if your source uses a
+checkpointing scheme that uniquely identifies each record. For example, if
+your splits are files and the checkpoints are file positions up to which all
+data has been read, you do not need record IDs. However, record IDs can
+still be useful if upstream systems writing data to your source occasionally
+produce duplicate records that your source might then read.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">getWatermark</code>: Returns a watermark that your <code class="highlighter-rouge">Reader</code> provides. The watermark
+is the approximate lower bound on timestamps of future elements to be read
+by your <code class="highlighter-rouge">Reader</code>. The runner uses the watermark as an estimate of data
+completeness. Watermarks are used in windowing and triggers.</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">getCheckpointMark</code>: The runner uses this method to create a checkpoint in
+your data stream. The checkpoint represents the progress of the
+<code class="highlighter-rouge">UnboundedReader</code>, which can be used for failure recovery. Different data
+streams may use different checkpointing methods; some sources might require
+received records to be acknowledged, while others might use positional
+checkpointing. You’ll need to tailor this method to the most appropriate
+checkpointing scheme. For example, you might have this method return the
+most recently acked record(s).</p>
+ </li>
+ <li>
+ <p><code class="highlighter-rouge">getCheckpointMark</code> is optional; you don’t need to implement it if your data
+does not have meaningful checkpoints. However, if you choose not to
+implement checkpointing in your source, you may encounter duplicate data or
+data loss in your pipeline, depending on whether your data source tries to
+re-send records in case of errors.</p>
+ </li>
+</ul>
+
+<p>You can read a bounded <code class="highlighter-rouge">PCollection</code> from an <code class="highlighter-rouge">UnboundedSource</code> by specifying
+either <code class="highlighter-rouge">.withMaxNumRecords</code> or <code class="highlighter-rouge">.withMaxReadTime</code> when you read from your
+source. <code class="highlighter-rouge">.withMaxNumRecords</code> reads a fixed maximum number of records from your
+unbounded source, while <code class="highlighter-rouge">.withMaxReadTime</code> reads from your unbounded source for
+a fixed maximum time duration.</p>
+
+<h4 id="bounded-dynamic">Using your BoundedSource with dynamic work rebalancing</h4>
+
+<p>If your source provides bounded data, you can have your <code class="highlighter-rouge">BoundedReader</code> work
+with dynamic work rebalancing by implementing the method <code class="highlighter-rouge">splitAtFraction</code>. The
+runner may call <code class="highlighter-rouge">splitAtFraction</code> concurrently with start or advance on a given
+reader so that the remaining data in your <code class="highlighter-rouge">Source</code> can be split and
+redistributed to other workers.</p>
+
+<p>When you implement <code class="highlighter-rouge">splitAtFraction</code>, your code must produce a
+mutually-exclusive set of splits where the union of those splits matches the
+total data set.</p>
+
+<p>If you implement <code class="highlighter-rouge">splitAtFraction</code>, you must implement both <code class="highlighter-rouge">splitAtFraction</code>
+and <code class="highlighter-rouge">getFractionConsumed</code> in a thread-safe manner, or data loss is possible. You
+should also unit-test your implementation exhaustively to avoid data duplication
+or data loss.</p>
+
+<p>To ensure that your code is thread-safe, use the <code class="highlighter-rouge">RangeTracker</code> thread-safe
+helper object to manage positions in your data source when implementing
+<code class="highlighter-rouge">splitAtFraction</code> and <code class="highlighter-rouge">getFractionConsumed</code>.</p>
+
+<p>We highly recommended that you unit test your implementations of
+<code class="highlighter-rouge">splitAtFraction</code> using the <code class="highlighter-rouge">SourceTestUtils</code> class. <code class="highlighter-rouge">SourceTestUtils</code> contains
+a number of methods for testing your implementation of <code class="highlighter-rouge">splitAtFraction</code>,
+including exhaustive automatic testing.</p>
+
+<h3 id="convenience-source-and-reader-base-classes">Convenience Source and Reader base classes</h3>
+
+<p>The Beam SDK contains some convenient abstract base classes to help you create
+<code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Reader</code> classes that work with common data storage formats, like
+files.</p>
+
+<h4 id="filebasedsource">FileBasedSource</h4>
+
+<p>If your data source uses files, you can derive your <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Reader</code>
+classes from the <code class="highlighter-rouge">FileBasedSource</code> and <code class="highlighter-rouge">FileBasedReader</code> abstract base classes.
+<code class="highlighter-rouge">FileBasedSource</code> is a bounded source subclass that implements code common to
+Beam sources that interact with files, including:</p>
+
+<ul>
+ <li>File pattern expansion</li>
+ <li>Sequential record reading</li>
+ <li>Split points</li>
+</ul>
+
+<h2 id="using-filebasedsink">Using the FileBasedSink abstraction</h2>
+
+<p>If your data source uses files, you can implement the <code class="highlighter-rouge">FileBasedSink</code>
+abstraction to create a file-based sink. For other sinks, use <code class="highlighter-rouge">ParDo</code>,
+<code class="highlighter-rouge">GroupByKey</code>, and other transforms offered by the Beam SDK for Java. See the
+<a href="/documentation/io/developing-io-overview/">developing I/O connectors overview</a>
+for more details.</p>
+
+<p>When using the <code class="highlighter-rouge">FileBasedSink</code> interface, you must provide the format-specific
+logic that tells the runner how to write bounded data from your pipeline’s
+<code class="highlighter-rouge">PCollection</code>s to an output sink. The runner writes bundles of data in parallel
+using multiple workers.</p>
+
+<p>Supply the logic for your file-based sink by implementing the following classes:</p>
+
+<ul>
+ <li>
+ <p>A subclass of the abstract base class <code class="highlighter-rouge">FileBasedSink</code>. <code class="highlighter-rouge">FileBasedSink</code>
+describes a location or resource that your pipeline can write to in
+parallel. To avoid exposing your sink to end-users, your <code class="highlighter-rouge">FileBasedSink</code>
+subclass should be protected or private.</p>
+ </li>
+ <li>
+ <p>A user-facing wrapper <code class="highlighter-rouge">PTransform</code> that, as part of the logic, calls
+<a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java">WriteFiles</a>
+and passes your <code class="highlighter-rouge">FileBasedSink</code> as a parameter. A user should not need to
+call <code class="highlighter-rouge">WriteFiles</code> directly.</p>
+ </li>
+</ul>
+
+<p>The <code class="highlighter-rouge">FileBasedSink</code> abstract base class implements code that is common to Beam
+sinks that interact with files, including:</p>
+
+<ul>
+ <li>Setting file headers and footers</li>
+ <li>Sequential record writing</li>
+ <li>Setting the output MIME type</li>
+</ul>
+
+<p><code class="highlighter-rouge">FileBasedSink</code> and its subclasses support writing files to any Beam-supported
+<code class="highlighter-rouge">FileSystem</code> implementations. See the following Beam-provided <code class="highlighter-rouge">FileBasedSink</code>
+implementations for examples:</p>
+
+<ul>
+ <li><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSink.java">TextSink</a> and</li>
+ <li><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSink.java">AvroSink</a>.</li>
+</ul>
+
+<h2 id="ptransform-wrappers">PTransform wrappers</h2>
+
+<p>When you create a source or sink that end-users will use, avoid exposing your
+source or sink code. To avoid exposing your sources and sinks to end-users, your
+new classes should be protected or private. Then, implement a user-facing
+wrapper <code class="highlighter-rouge">PTransform</code>. By exposing your source or sink as a transform, your
+implementation is hidden and can be arbitrarily complex or simple. The greatest
+benefit of not exposing implementation details is that later on, you can add
+additional functionality without breaking the existing implementation for users.</p>
+
+<p>For example, if your users’ pipelines read from your source using
+<code class="highlighter-rouge">read</code> and you want to insert a reshard into the pipeline, all
+users would need to add the reshard themselves (using the <code class="highlighter-rouge">GroupByKey</code>
+transform). To solve this, we recommended that you expose the source as a
+composite <code class="highlighter-rouge">PTransform</code> that performs both the read operation and the reshard.</p>
+
+<p>See Beam’s <a href="/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else">PTransform style guide</a>
+for additional information about wrapping with a <code class="highlighter-rouge">PTransform</code>.</p>
+
+
+ </div>
+ </div>
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+<footer class="footer">
+ <div class="footer__contained">
+ <div class="footer__cols">
+ <div class="footer__cols__col">
+ <div class="footer__cols__col__logo">
+ <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
+ </div>
+ <div class="footer__cols__col__logo">
+ <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
+ </div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Start</div>
+ <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Docs</div>
+ <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
+ <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Community</div>
+ <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
+ <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a></div>
+ <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
+ </div>
+ <div class="footer__cols__col footer__cols__col--md">
+ <div class="footer__cols__col__title">Resources</div>
+ <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
+ <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
+ <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
+ </div>
+ </div>
+ </div>
+ <div class="footer__bottom">
+ ©
+ <a href="http://www.apache.org">The Apache Software Foundation</a>
+ | <a href="/privacy_policy">Privacy Policy</a>
+ | <a href="/feed.xml">RSS Feed</a>
+ <br><br>
+ Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
+ either registered trademarks or trademarks of The Apache Software
+ Foundation. All other products or name brands are trademarks of their
+ respective holders, including The Apache Software Foundation.
+ </div>
+</footer>
+
+ </body>
+</html>
diff --git a/website/generated-content/documentation/io/authoring-overview/index.html b/website/generated-content/documentation/io/developing-io-overview/index.html
similarity index 66%
copy from website/generated-content/documentation/io/authoring-overview/index.html
copy to website/generated-content/documentation/io/developing-io-overview/index.html
index 99d28d8..33dfe8b 100644
--- a/website/generated-content/documentation/io/authoring-overview/index.html
+++ b/website/generated-content/documentation/io/developing-io-overview/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Authoring I/O Transforms - Overview</title>
+ <title>Overview: Developing a new I/O connector</title>
<meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/io/authoring-overview/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-overview/" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
@@ -125,7 +125,7 @@
GitHub links will not resolve until the markdown source is available on the master branch.
New pages would fail validation during development / PR test automation.
-->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/authoring-overview.md" data-proofer-ignore>
+ <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/developing-io-overview.md" data-proofer-ignore>
<i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
</a>
</li>
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
@@ -280,18 +290,13 @@
<ul class="nav">
- <li><a href="#introduction">Introduction</a></li>
- <li><a href="#suggested-steps-for-implementers">Suggested steps for implementers</a></li>
- <li><a href="#read-transforms">Read transforms</a>
- <ul>
- <li><a href="#when-to-implement-using-the-source-api">When to implement using the <code class="highlighter-rouge">Source</code> API</a></li>
- </ul>
- </li>
- <li><a href="#write-transforms">Write transforms</a>
+ <li><a href="#sources">Sources</a>
<ul>
- <li><a href="#when-to-implement-using-the-sink-api">When to implement using the <code class="highlighter-rouge">Sink</code> API</a></li>
+ <li><a href="#when-to-use-source">When to use the Source interface</a></li>
+ <li><a href="#using-pardo-and-groupbykey">Using ParDo and GroupByKey</a></li>
</ul>
</li>
+ <li><a href="#sinks">Sinks</a></li>
</ul>
@@ -312,112 +317,191 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
+<h1 id="overview-developing-a-new-io-connector">Overview: Developing a new I/O connector</h1>
-<h1 id="authoring-io-transforms---overview">Authoring I/O Transforms - Overview</h1>
+<p><em>A guide for users who need to connect to a data store that isn’t supported by
+the <a href="/documentation/io/built-in/">Built-in I/O connectors</a></em></p>
-<p><em>A guide for users who need to connect to a data store that isn’t supported by the <a href="/documentation/io/built-in/">Built-in I/O Transforms</a></em></p>
+<p>To connect to a data store that isn’t supported by Beam’s existing I/O
+connectors, you must create a custom I/O connector. A connector usually consists
+of a source and a sink. All Beam sources and sinks are composite transforms;
+however, the implementation of your custom I/O depends on your use case. Here
+are the recommended steps to get started:</p>
-<ul id="markdown-toc">
- <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
- <li><a href="#suggested-steps-for-implementers" id="markdown-toc-suggested-steps-for-implementers">Suggested steps for implementers</a></li>
- <li><a href="#read-transforms" id="markdown-toc-read-transforms">Read transforms</a> <ul>
- <li><a href="#when-to-implement-using-the-source-api" id="markdown-toc-when-to-implement-using-the-source-api">When to implement using the <code class="highlighter-rouge">Source</code> API</a></li>
- </ul>
+<ol>
+ <li>
+ <p>Read this overview and choose your implementation. You can email the
+<a href="/get-started/support">Beam dev mailing list</a> with any
+questions you might have. In addition, you can check if anyone else is
+working on the same I/O connector.</p>
</li>
- <li><a href="#write-transforms" id="markdown-toc-write-transforms">Write transforms</a> <ul>
- <li><a href="#when-to-implement-using-the-sink-api" id="markdown-toc-when-to-implement-using-the-sink-api">When to implement using the <code class="highlighter-rouge">Sink</code> API</a></li>
- </ul>
+ <li>
+ <p>If you plan to contribute your I/O connector to the Beam community, see the
+<a href="/contribute/contribution-guide/">Apache Beam contribution guide</a>.</p>
</li>
-</ul>
-
-<h2 id="introduction">Introduction</h2>
-<p>This guide covers how to implement I/O transforms in the Beam model. Beam pipelines use these read and write transforms to import data for processing, and write data to a store.</p>
+ <li>
+ <p>Read the <a href="/contribute/ptransform-style-guide/">PTransform style guide</a>
+for additional style guide recommendations.</p>
+ </li>
+</ol>
-<p>Reading and writing data in Beam is a parallel task, and using <code class="highlighter-rouge">ParDo</code>s, <code class="highlighter-rouge">GroupByKey</code>s, etc… is usually sufficient. Rarely, you will need the more specialized <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Sink</code> classes for specific features. There are changes coming soon (<code class="highlighter-rouge">SplittableDoFn</code>, <a href="https://issues.apache.org/jira/brows [...]
+<h2 id="sources">Sources</h2>
-<p>As you work on your I/O Transform, be aware that the Beam community is excited to help those building new I/O Transforms and that there are many examples and helper classes.</p>
+<p>For <strong>bounded (batch) sources</strong>, there are currently two options for creating a
+Beam source:</p>
-<h2 id="suggested-steps-for-implementers">Suggested steps for implementers</h2>
<ol>
- <li>Check out this guide and come up with your design. If you’d like, you can email the <a href="/get-started/support">Beam dev mailing list</a> with any questions you might have. It’s good to check there to see if anyone else is working on the same I/O Transform.</li>
- <li>If you are planning to contribute your I/O transform to the Beam community, you’ll be going through the normal Beam contribution life cycle - see the <a href="/contribute/contribution-guide/">Apache Beam Contribution Guide</a> for more details.</li>
- <li>As you’re working on your IO transform, see the <a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a> for specific information about writing I/O Transforms.</li>
+ <li>
+ <p>Use <code class="highlighter-rouge">ParDo</code> and <code class="highlighter-rouge">GroupByKey</code>.</p>
+ </li>
+ <li>
+ <p>Use the <code class="highlighter-rouge">Source</code> interface and extend the <code class="highlighter-rouge">BoundedSource</code> abstract subclass.</p>
+ </li>
</ol>
-<h2 id="read-transforms">Read transforms</h2>
-<p>Read transforms take data from outside of the Beam pipeline and produce <code class="highlighter-rouge">PCollection</code>s of data.</p>
+<p><code class="highlighter-rouge">ParDo</code> is the recommended option, as implementing a <code class="highlighter-rouge">Source</code> can be tricky. See
+<a href="#when-to-use-source">When to use the Source interface</a> for a list of some use
+cases where you might want to use a <code class="highlighter-rouge">Source</code> (such as
+<a href="/blog/2016/05/18/splitAtFraction-method.html">dynamic work rebalancing</a>).</p>
+
+<p>(Java only) For <strong>unbounded (streaming) sources</strong>, you must use the <code class="highlighter-rouge">Source</code>
+interface and extend the <code class="highlighter-rouge">UnboundedSource</code> abstract subclass. <code class="highlighter-rouge">UnboundedSource</code>
+supports features that are useful for streaming pipelines, such as
+checkpointing.</p>
+
+<p>Splittable DoFn is a new sources framework that is under development and will
+replace the other options for developing bounded and unbounded sources. For more
+information, see the
+<a href="/roadmap/connectors-multi-sdk/">roadmap for multi-SDK connector efforts</a>.</p>
+
+<h3 id="when-to-use-source">When to use the Source interface</h3>
+
+<p>If you are not sure whether to use <code class="highlighter-rouge">Source</code>, feel free to email the <a href="/get-started/support">Beam dev
+mailing list</a> and we can discuss the
+specific pros and cons of your case.</p>
+
+<p>In some cases, implementing a <code class="highlighter-rouge">Source</code> might be necessary or result in better
+performance:</p>
+
+<ul>
+ <li>
+ <p><strong>Unbounded sources:</strong> <code class="highlighter-rouge">ParDo</code> does not work for reading from unbounded
+sources. <code class="highlighter-rouge">ParDo</code> does not support checkpointing or mechanisms like de-duping
+that are useful for streaming data sources.</p>
+ </li>
+ <li>
+ <p><strong>Progress and size estimation:</strong> <code class="highlighter-rouge">ParDo</code> can’t provide hints to runners about
+progress or the size of data they are reading. Without size estimation of the
+data or progress on your read, the runner doesn’t have any way to guess how
+large your read will be. Therefore, if the runner attempts to dynamically
+allocate workers, it does not have any clues as to how many workers you might
+need for your pipeline.</p>
+ </li>
+ <li>
+ <p><strong>Dynamic work rebalancing:</strong> <code class="highlighter-rouge">ParDo</code> does not support dynamic work
+rebalancing, which is used by some readers to improve the processing speed of
+jobs. Depending on your data source, dynamic work rebalancing might not be
+possible.</p>
+ </li>
+ <li>
+ <p><strong>Splitting into parts of particular size recommended by the runner:</strong> <code class="highlighter-rouge">ParDo</code>
+does not receive <code class="highlighter-rouge">desired_bundle_size</code> as a hint from runners when performing
+initial splitting.</p>
+ </li>
+</ul>
+
+<p>For example, if you’d like to read from a new file format that contains many
+records per file, or if you’d like to read from a key-value store that supports
+read operations in sorted key order.</p>
+
+<h3 id="using-pardo-and-groupbykey">Using ParDo and GroupByKey</h3>
+
+<p>For data stores or file types where the data can be read in parallel, you can
+think of the process as a mini-pipeline. This often consists of two steps:</p>
-<p>For data stores or file types where the data can be read in parallel, you can think of the process as a mini-pipeline. This often consists of two steps:</p>
<ol>
- <li>Splitting the data into parts to be read in parallel</li>
- <li>Reading from each of those parts</li>
+ <li>
+ <p>Splitting the data into parts to be read in parallel</p>
+ </li>
+ <li>
+ <p>Reading from each of those parts</p>
+ </li>
</ol>
-<p>Each of those steps will be a <code class="highlighter-rouge">ParDo</code>, with a <code class="highlighter-rouge">GroupByKey</code> in between. The <code class="highlighter-rouge">GroupByKey</code> is an implementation detail, but for most runners it allows the runner to use different numbers of workers for:</p>
+<p>Each of those steps will be a <code class="highlighter-rouge">ParDo</code>, with a <code class="highlighter-rouge">GroupByKey</code> in between. The
+<code class="highlighter-rouge">GroupByKey</code> is an implementation detail, but for most runners <code class="highlighter-rouge">GroupByKey</code>
+allows the runner to use different numbers of workers in some situations:</p>
+
<ul>
- <li>Determining how to split up the data to be read into chunks - this will likely occur on very few workers</li>
- <li>Reading - will likely benefit from more workers</li>
+ <li>
+ <p>Determining how to split up the data to be read into chunks</p>
+ </li>
+ <li>
+ <p>Reading data, which often benefits from more workers</p>
+ </li>
</ul>
-<p>The <code class="highlighter-rouge">GroupByKey</code> will also allow Dynamic Work Rebalancing to occur (on supported runners).</p>
+<p>In addition, <code class="highlighter-rouge">GroupByKey</code> also allows dynamic work rebalancing to happen on
+runners that support the feature.</p>
+
+<p>Here are some examples of read transform implementations that use the “reading
+as a mini-pipeline” model when data can be read in parallel:</p>
-<p>Here are some examples of read transform implementations that use the “reading as a mini-pipeline” model when data can be read in parallel:</p>
<ul>
- <li><strong>Reading from a file glob</strong> - For example reading all files in “~/data/**”
+ <li><strong>Reading from a file glob</strong>: For example, reading all files in “~/data/**”.
<ul>
- <li>Get File Paths <code class="highlighter-rouge">ParDo</code>: As input, take in a file glob. Produce a <code class="highlighter-rouge">PCollection</code> of strings, each of which is a file path.</li>
- <li>Reading <code class="highlighter-rouge">ParDo</code>: Given the <code class="highlighter-rouge">PCollection</code> of file paths, read each one, producing a <code class="highlighter-rouge">PCollection</code> of records.</li>
+ <li>Get File Paths <code class="highlighter-rouge">ParDo</code>: As input, take in a file glob. Produce a
+<code class="highlighter-rouge">PCollection</code> of strings, each of which is a file path.</li>
+ <li>Reading <code class="highlighter-rouge">ParDo</code>: Given the <code class="highlighter-rouge">PCollection</code> of file paths, read each one,
+producing a <code class="highlighter-rouge">PCollection</code> of records.</li>
</ul>
</li>
- <li><strong>Reading from a NoSQL Database</strong> (eg Apache HBase) - these databases often allow reading from ranges in parallel.
+ <li><strong>Reading from a NoSQL database</strong> (such as Apache HBase): These databases
+often allow reading from ranges in parallel.
<ul>
- <li>Determine Key Ranges <code class="highlighter-rouge">ParDo</code>: As input, receive connection information for the database and the key range to read from. Produce a <code class="highlighter-rouge">PCollection</code> of key ranges that can be read in parallel efficiently.</li>
- <li>Read Key Range <code class="highlighter-rouge">ParDo</code>: Given the <code class="highlighter-rouge">PCollection</code> of key ranges, read the key range, producing a <code class="highlighter-rouge">PCollection</code> of records.</li>
+ <li>Determine Key Ranges <code class="highlighter-rouge">ParDo</code>: As input, receive connection information for
+the database and the key range to read from. Produce a <code class="highlighter-rouge">PCollection</code> of key
+ranges that can be read in parallel efficiently.</li>
+ <li>Read Key Range <code class="highlighter-rouge">ParDo</code>: Given the <code class="highlighter-rouge">PCollection</code> of key ranges, read the key
+range, producing a <code class="highlighter-rouge">PCollection</code> of records.</li>
</ul>
</li>
</ul>
-<p>For data stores or files where reading cannot occur in parallel, reading is a simple task that can be accomplished with a single <code class="highlighter-rouge">ParDo</code>+<code class="highlighter-rouge">GroupByKey</code>. For example:</p>
-<ul>
- <li><strong>Reading from a database query</strong> - traditional SQL database queries often can only be read in sequence. The <code class="highlighter-rouge">ParDo</code> in this case would establish a connection to the database and read batches of records, producing a <code class="highlighter-rouge">PCollection</code> of those records.</li>
- <li><strong>Reading from a gzip file</strong> - a gzip file has to be read in order, so it cannot be parallelized. The <code class="highlighter-rouge">ParDo</code> in this case would open the file and read in sequence, producing a <code class="highlighter-rouge">PCollection</code> of records from the file.</li>
-</ul>
-
-<h3 id="when-to-implement-using-the-source-api">When to implement using the <code class="highlighter-rouge">Source</code> API</h3>
-<p>The above discussion is in terms of <code class="highlighter-rouge">ParDo</code>s - this is because <code class="highlighter-rouge">Source</code>s have proven to be tricky to implement. At this point in time, the recommendation is to <strong>use <code class="highlighter-rouge">Source</code> only if <code class="highlighter-rouge">ParDo</code> doesn’t meet your needs</strong>. A class derived from <code class="highlighter-rouge">FileBasedSource</code> is often the best option when rea [...]
+<p>For data stores or files where reading cannot occur in parallel, reading is a
+simple task that can be accomplished with a single <code class="highlighter-rouge">ParDo</code>+<code class="highlighter-rouge">GroupByKey</code>. For
+example:</p>
-<p>If you’re trying to decide on whether or not to use <code class="highlighter-rouge">Source</code>, feel free to email the <a href="/get-started/support">Beam dev mailing list</a> and we can discuss the specific pros and cons of your case.</p>
-
-<p>In some cases implementing a <code class="highlighter-rouge">Source</code> may be necessary or result in better performance.</p>
<ul>
- <li><code class="highlighter-rouge">ParDo</code>s will not work for reading from unbounded sources - they do not support checkpointing and don’t support mechanisms like de-duping that have proven useful for streaming data sources.</li>
- <li><code class="highlighter-rouge">ParDo</code>s cannot provide hints to runners about their progress or the size of data they are reading - without size estimation of the data or progress on your read, the runner doesn’t have any way to guess how large your read will be, and thus if it attempts to dynamically allocate workers, it does not have any clues as to how many workers you may need for your pipeline.</li>
- <li><code class="highlighter-rouge">ParDo</code>s do not support Dynamic Work Rebalancing - these are features used by some readers to improve the processing speed of jobs (but may not be possible with your data source).</li>
- <li><code class="highlighter-rouge">ParDo</code>s do not receive ‘desired_bundle_size’ as a hint from runners when performing initial splitting.
-<code class="highlighter-rouge">SplittableDoFn</code> (<a href="https://issues.apache.org/jira/browse/BEAM-65">BEAM-65</a>) will mitigate many of these concerns.</li>
+ <li>
+ <p><strong>Reading from a database query</strong>: Traditional SQL database queries often
+can only be read in sequence. In this case, the <code class="highlighter-rouge">ParDo</code> would establish a
+connection to the database and read batches of records, producing a
+<code class="highlighter-rouge">PCollection</code> of those records.</p>
+ </li>
+ <li>
+ <p><strong>Reading from a gzip file</strong>: A gzip file must be read in order, so the read
+cannot be parallelized. In this case, the <code class="highlighter-rouge">ParDo</code> would open the file and
+read in sequence, producing a <code class="highlighter-rouge">PCollection</code> of records from the file.</p>
+ </li>
</ul>
-<h2 id="write-transforms">Write transforms</h2>
-<p>Write transforms are responsible for taking the contents of a <code class="highlighter-rouge">PCollection</code> and transferring that data outside of the Beam pipeline.</p>
+<h2 id="sinks">Sinks</h2>
-<p>Write transforms can usually be implemented using a single <code class="highlighter-rouge">ParDo</code> that writes the records received to the data store.</p>
+<p>To create a Beam sink, we recommend that you use a <code class="highlighter-rouge">ParDo</code> that writes the
+received records to the data store. To develop more complex sinks (for example,
+to support data de-duplication when failures are retried by a runner), use
+<code class="highlighter-rouge">ParDo</code>, <code class="highlighter-rouge">GroupByKey</code>, and other available Beam transforms.</p>
-<p>TODO: this section needs further explanation.</p>
+<p>For <strong>file-based sinks</strong>, you can use the <code class="highlighter-rouge">FileBasedSink</code> abstraction that is
+provided by both the Java and Python SDKs. See our language specific
+implementation guides for more details:</p>
-<h3 id="when-to-implement-using-the-sink-api">When to implement using the <code class="highlighter-rouge">Sink</code> API</h3>
-<p>You are strongly discouraged from using the <code class="highlighter-rouge">Sink</code> class unless you are creating a <code class="highlighter-rouge">FileBasedSink</code>. Most of the time, a simple <code class="highlighter-rouge">ParDo</code> is all that’s necessary. If you think you have a case that is only possible using a <code class="highlighter-rouge">Sink</code>, please email the <a href="/get-started/support">Beam dev mailing list</a>.</p>
-
-<h1 id="next-steps">Next steps</h1>
-
-<p>This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
-
-<!-- TODO: commented out until this content is ready.
-For more details on actual implementation, continue with one of the the language specific guides:
+<ul>
+ <li><a href="/documentation/io/developing-io-java/">Developing I/O connectors for Java</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing I/O connectors for Python</a></li>
+</ul>
-* [Authoring I/O Transforms - Python](/documentation/io/authoring-python/)
-* [Authoring I/O Transforms - Java](/documentation/io/authoring-java/)
--->
</div>
</div>
diff --git a/website/generated-content/documentation/sdks/python-custom-io/index.html b/website/generated-content/documentation/io/developing-io-python/index.html
similarity index 55%
copy from website/generated-content/documentation/sdks/python-custom-io/index.html
copy to website/generated-content/documentation/io/developing-io-python/index.html
index 04d3a0e..42660ed 100644
--- a/website/generated-content/documentation/sdks/python-custom-io/index.html
+++ b/website/generated-content/documentation/io/developing-io-python/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Apache Beam: Creating New Sources and Sinks with the Python SDK</title>
+ <title>Apache Beam: Developing I/O connectors for Python</title>
<meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/sdks/python-custom-io/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-python/" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
@@ -125,7 +125,7 @@
GitHub links will not resolve until the markdown source is available on the master branch.
New pages would fail validation during development / PR test automation.
-->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/sdks/python-custom-io.md" data-proofer-ignore>
+ <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/developing-io-python.md" data-proofer-ignore>
<i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
</a>
</li>
@@ -150,74 +150,125 @@
limitations under the License. See accompanying LICENSE file.
-->
-<li><span class="section-nav-list-main-title">Languages</span></li>
-
+<li><span class="section-nav-list-main-title">Documentation</span></li>
+<li><a href="/documentation">Using the Documentation</a></li>
+<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
<li>
- <span class="section-nav-list-title">Java</span>
+ <span class="section-nav-list-title">Pipeline development lifecycle</span>
+
<ul class="section-nav-list">
- <li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.9.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
- </li>
- <li><a href="/documentation/sdks/java-dependencies/">Java SDK dependencies</a></li>
- <li><a href="/documentation/sdks/java-extensions/">Java SDK extensions</a></li>
- <li><a href="/documentation/sdks/java-thirdparty/">Java 3rd party extensions</a></li>
- <li><a href="/documentation/sdks/java/testing/nexmark/">Nexmark benchmark suite</a></li>
+ <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
+ <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-
<li>
- <span class="section-nav-list-title">Python</span>
+ <span class="section-nav-list-title">Beam programming guide</span>
+
<ul class="section-nav-list">
- <li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.9.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
+ <li><a href="/documentation/programming-guide/">Overview</a></li>
+ <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">PCollections</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li>
+ <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li>
+ </ul>
</li>
- <li><a href="/documentation/sdks/python-dependencies/">Python SDK dependencies</a></li>
- <li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
- <li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
- <li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
- </ul>
-</li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Transforms</span>
-<li>
- <span class="section-nav-list-title">Go</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/sdks/go/">Go SDK overview</a></li>
- <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam" target="_blank">Go SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li>
+ <li>
+ <span class="section-nav-list-title">Core Beam transforms</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li>
+ <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li>
+ <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li>
+ <li><a href="/documentation/programming-guide/#combine">Combine</a></li>
+ <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li>
+ <li><a href="/documentation/programming-guide/#partition">Partition</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li>
+ <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li>
+ <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li>
+ <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li>
+ </ul>
</li>
- </ul>
-</li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Pipeline I/O</span>
-<li>
- <span class="section-nav-list-title">SQL</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
- <li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
- <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
+ <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Data encoding and type safety</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li>
+ <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li>
+ <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li>
+ </ul>
+ </li>
<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">SQL Reference</span>
+ <span class="section-nav-list-title">Windowing</span>
<ul class="section-nav-list">
- <li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
- <li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
- <li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
- <li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
- <li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
- <li><a href="/documentation/dsls/sql/scalar-functions/">Scalar functions</a></li>
- <li><a href="/documentation/dsls/sql/aggregate-functions/">Aggregate functions</a></li>
- <li><a href="/documentation/dsls/sql/user-defined-functions/">User-defined functions</a></li>
- <li><a href="/documentation/dsls/sql/set/">SET Pipeline Options</a></li>
+ <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li>
+ <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li>
+ <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li>
+ <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li>
+ <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Triggers</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li>
+ <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li>
+ <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li>
+ <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li>
+ <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li>
+ <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
</ul>
</li>
</ul>
</li>
+<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Learning Resources</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
+ <li><a href="/documentation/resources/learning-resources/#articles">Articles</a></li>
+ <li><a href="/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
+ <li><a href="/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
+ <li><a href="/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
+ <li><a href="/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
+ <li><a href="/documentation/resources/learning-resources/#how-to-contribute">How to Contribute</a></li>
+ <li><a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a></li>
+ </ul>
+</li>
+<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki</a></li>
</ul>
</nav>
@@ -239,32 +290,17 @@
<ul class="nav">
- <li><a href="#why-create-a-new-source-or-sink">Why Create a New Source or Sink</a></li>
- <li><a href="#basic-code-reqs">Basic Code Requirements for New Sources and Sinks</a>
+ <li><a href="#basic-code-reqs">Basic code requirements</a></li>
+ <li><a href="#implementing-the-source-interface">Implementing the Source interface</a>
<ul>
- <li><a href="#serializability">Serializability</a></li>
- <li><a href="#immutability">Immutability</a></li>
- <li><a href="#thread-safety">Thread-Safety</a></li>
- <li><a href="#testability">Testability</a></li>
+ <li><a href="#implementing-the-boundedsource-subclass">Implementing the BoundedSource subclass</a></li>
+ <li><a href="#implementing-the-rangetracker-subclass">Implementing the RangeTracker subclass</a></li>
+ <li><a href="#convenience-source-base-classes">Convenience Source base classes</a></li>
+ <li><a href="#reading-from-a-new-source">Reading from a new Source</a></li>
</ul>
</li>
- <li><a href="#creating-a-new-source">Creating a New Source</a>
- <ul>
- <li><a href="#implementing-the-boundedsource-subclass">Implementing the BoundedSource Subclass</a></li>
- <li><a href="#implementing-the-rangetracker-subclass">Implementing the RangeTracker Subclass</a></li>
- <li><a href="#convenience-source-base-classes">Convenience Source Base Classes</a></li>
- </ul>
- </li>
- <li><a href="#reading-from-a-new-source">Reading from a New Source</a></li>
- <li><a href="#creating-a-new-sink">Creating a New Sink</a>
- <ul>
- <li><a href="#implementing-the-sink-subclass">Implementing the Sink Subclass</a></li>
- <li><a href="#implementing-the-writer-subclass">Implementing the Writer Subclass</a></li>
- <li><a href="#convenience-sink-and-writer-base-classes">Convenience Sink and Writer Base Classes</a></li>
- </ul>
- </li>
- <li><a href="#writing-to-a-new-sink">Writing to a New Sink</a></li>
- <li><a href="#ptransform-wrappers">PTransform Wrappers</a></li>
+ <li><a href="#using-the-filebasedsink-abstraction">Using the FileBasedSink abstraction</a></li>
+ <li><a href="#ptransform-wrappers">PTransform wrappers</a></li>
</ul>
@@ -284,68 +320,81 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<h1 id="creating-new-sources-and-sinks-with-the-python-sdk">Creating New Sources and Sinks with the Python SDK</h1>
-
-<p>The Apache Beam SDK for Python provides an extensible API that you can use to create new data sources and sinks. This tutorial shows how to create new sources and sinks using <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">Beam’s Source and Sink API</a>.</p>
-
-<ul>
- <li>Create a new source by extending the <code class="highlighter-rouge">BoundedSource</code> and <code class="highlighter-rouge">RangeTracker</code> interfaces.</li>
- <li>Create a new sink by implementing the <code class="highlighter-rouge">Sink</code> and <code class="highlighter-rouge">Writer</code> classes.</li>
-</ul>
+<h1 id="developing-io-connectors-for-python">Developing I/O connectors for Python</h1>
-<h2 id="why-create-a-new-source-or-sink">Why Create a New Source or Sink</h2>
+<p>To connect to a data store that isn’t supported by Beam’s existing I/O
+connectors, you must create a custom I/O connector that usually consist of a
+source and a sink. All Beam sources and sinks are composite transforms; however,
+the implementation of your custom I/O depends on your use case. Before you
+start, read the <a href="/documentation/io/developing-io-overview/">new I/O connector overview</a>
+for an overview of developing a new I/O connector, the available implementation
+options, and how to choose the right option for your use case.</p>
-<p>You’ll need to create a new source or sink if you want your pipeline to read data from (or write data to) a storage system for which the Beam SDK for Python does not provide <a href="/documentation/programming-guide/#pipeline-io">native support</a>.</p>
+<p>This guide covers using the <a href="https://beam.apache.org/releases/pydoc/2.9.0/apache_beam.io.iobase.html">Source and FileBasedSink interfaces</a>
+for Python. The Java SDK offers the same functionality, but uses a slightly
+different API. See <a href="/documentation/io/developing-io-java/">Developing I/O connectors for Java</a>
+for information specific to the Java SDK.</p>
-<p>In simple cases, you may not need to create a new source or sink. For example, if you need to read data from an SQL database using an arbitrary query, none of the advanced Source API features would benefit you. Likewise, if you’d like to write data to a third-party API via a protocol that lacks deduplication support, the Sink API wouldn’t benefit you. In such cases it makes more sense to use a <code class="highlighter-rouge">ParDo</code>.</p>
+<h2 id="basic-code-reqs">Basic code requirements</h2>
-<p>However, if you’d like to use advanced features such as dynamic splitting and size estimation, you should use Beam’s APIs and create a new source or sink.</p>
+<p>Beam runners use the classes you provide to read and/or write data using
+multiple worker instances in parallel. As such, the code you provide for
+<code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">FileBasedSink</code> subclasses must meet some basic requirements:</p>
-<h2 id="basic-code-reqs">Basic Code Requirements for New Sources and Sinks</h2>
-
-<p>Services use the classes you provide to read and/or write data using multiple worker instances in parallel. As such, the code you provide for <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Sink</code> subclasses must meet some basic requirements:</p>
-
-<h3 id="serializability">Serializability</h3>
-
-<p>Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass must be serializable. The service may create multiple instances of your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass to be sent to multiple remote workers to facilitate reading or writing in parallel. Note that the <em>way</em> the source and sink objects are serialized is runner specific.</p>
-
-<h3 id="immutability">Immutability</h3>
-
-<p>Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass must be effectively immutable. You should only use mutable state in your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass if you are using lazy evaluation of expensive computations that you need to implement the source.</p>
-
-<h3 id="thread-safety">Thread-Safety</h3>
-
-<p>Your code must be thread-safe. The Beam SDK for Python provides the <code class="highlighter-rouge">RangeTracker</code> class to make this easier.</p>
-
-<h3 id="testability">Testability</h3>
-
-<p>It is critical to exhaustively unit-test all of your <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Sink</code> subclasses. A minor implementation error can lead to data corruption or data loss (such as skipping or duplicating records) that can be hard to detect.</p>
-
-<p>You can use test harnesses and utility methods available in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/source_test_utils.py">source_test_utils module</a> to develop tests for your source.</p>
-
-<h2 id="creating-a-new-source">Creating a New Source</h2>
-
-<p>You should create a new source if you’d like to use the advanced features that the Source API provides:</p>
+<ol>
+ <li>
+ <p><strong>Serializability:</strong> Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">FileBasedSink</code> subclass must be
+serializable. The service may create multiple instances of your <code class="highlighter-rouge">Source</code>
+or <code class="highlighter-rouge">FileBasedSink</code> subclass to be sent to multiple remote workers to
+facilitate reading or writing in parallel. The <em>way</em> the source and sink
+objects are serialized is runner specific.</p>
+ </li>
+ <li>
+ <p><strong>Immutability:</strong> Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">FileBasedSink</code> subclass must be
+effectively immutable. You should only use mutable state in your <code class="highlighter-rouge">Source</code>
+or <code class="highlighter-rouge">FileBasedSink</code> subclass if you are using lazy evaluation of expensive
+computations that you need to implement the source.</p>
+ </li>
+ <li>
+ <p><strong>Thread-Safety:</strong> Your code must be thread-safe. The Beam SDK for Python
+provides the <code class="highlighter-rouge">RangeTracker</code> class to make this easier.</p>
+ </li>
+ <li>
+ <p><strong>Testability:</strong> It is critical to exhaustively unit-test all of your
+<code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">FileBasedSink</code> subclasses. A minor implementation error can
+lead to data corruption or data loss (such as skipping or duplicating
+records) that can be hard to detect. You can use test harnesses and utility
+methods available in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/source_test_utils.py">source_test_utils module</a>
+to develop tests for your source.</p>
+ </li>
+</ol>
-<ul>
- <li>Dynamic splitting</li>
- <li>Progress estimation</li>
- <li>Size estimation</li>
- <li>Splitting into parts of particular size recommended by the service</li>
-</ul>
+<p>In addition, see the <a href="/contribute/ptransform-style-guide/">PTransform style guide</a>
+for Beam’s transform style guidance.</p>
-<p>For example, if you’d like to read from a new file format that contains many records per file, or if you’d like to read from a key-value store that supports read operations in sorted key order.</p>
+<h2 id="implementing-the-source-interface">Implementing the Source interface</h2>
<p>To create a new data source for your pipeline, you’ll need to provide the format-specific logic that tells the service how to read data from your input source, and how to split your data source into multiple parts so that multiple worker instances can read your data in parallel.</p>
-<p>You supply the logic for your new source by creating the following classes:</p>
+<p>Supply the logic for your new source by creating the following classes:</p>
<ul>
- <li>A subclass of <code class="highlighter-rouge">BoundedSource</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">BoundedSource</code> is a source that reads a finite amount of input records. The class describes the data you want to read, including the data’s location and parameters (such as how much data to read).</li>
- <li>A subclass of <code class="highlighter-rouge">RangeTracker</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">RangeTracker</code> is a thread-safe object used to manage a range for a given position type.</li>
+ <li>A subclass of <code class="highlighter-rouge">BoundedSource</code>. <code class="highlighter-rouge">BoundedSource</code> is a source that reads a
+finite amount of input records. The class describes the data you want to
+read, including the data’s location and parameters (such as how much data to
+read).</li>
+ <li>A subclass of <code class="highlighter-rouge">RangeTracker</code>. <code class="highlighter-rouge">RangeTracker</code> is a thread-safe object used to
+manage a range for a given position type.</li>
+ <li>One or more user-facing wrapper composite transforms (<code class="highlighter-rouge">PTransform</code>) that
+wrap read operations. <a href="#ptransform-wrappers">PTransform wrappers</a> discusses
+why you should avoid exposing your sources, and walks through how to create
+a wrapper.</li>
</ul>
-<h3 id="implementing-the-boundedsource-subclass">Implementing the BoundedSource Subclass</h3>
+<p>You can find these classes in the
+<a href="https://beam.apache.org/releases/pydoc/2.9.0/apache_beam.io.iobase.html">apache_beam.io.iobase module</a>.</p>
+
+<h3 id="implementing-the-boundedsource-subclass">Implementing the BoundedSource subclass</h3>
<p><code class="highlighter-rouge">BoundedSource</code> represents a finite data set from which the service reads, possibly in parallel. <code class="highlighter-rouge">BoundedSource</code> contains a set of methods that the service uses to split the data set for reading by multiple remote workers.</p>
@@ -366,7 +415,7 @@ limitations under the License.
</li>
</ul>
-<h3 id="implementing-the-rangetracker-subclass">Implementing the RangeTracker Subclass</h3>
+<h3 id="implementing-the-rangetracker-subclass">Implementing the RangeTracker subclass</h3>
<p>A <code class="highlighter-rouge">RangeTracker</code> is a thread-safe object used to manage the current range and current position of the reader of a <code class="highlighter-rouge">BoundedSource</code> and protect concurrent access to them.</p>
@@ -421,7 +470,7 @@ limitations under the License.
</li>
</ul>
-<h4 id="rangetracker-methods">RangeTracker Methods</h4>
+<h4 id="rangetracker-methods">RangeTracker methods</h4>
<p>To implement a <code class="highlighter-rouge">RangeTracker</code>, your subclass must override the following methods:</p>
@@ -456,7 +505,7 @@ limitations under the License.
<p><strong>Note:</strong> Methods of class <code class="highlighter-rouge">iobase.RangeTracker</code> may be invoked by multiple threads, hence this class must be made thread-safe, for example, by using a single lock object.</p>
-<h3 id="convenience-source-base-classes">Convenience Source Base Classes</h3>
+<h3 id="convenience-source-base-classes">Convenience Source base classes</h3>
<p>The Beam SDK for Python contains some convenient abstract base classes to help you easily create new sources.</p>
@@ -468,7 +517,7 @@ limitations under the License.
<p>See <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">AvroSource</a> for an example implementation of <code class="highlighter-rouge">FileBasedSource</code>.</p>
-<h2 id="reading-from-a-new-source">Reading from a New Source</h2>
+<h3 id="reading-from-a-new-source">Reading from a new Source</h3>
<p>The following example, <code class="highlighter-rouge">CountingSource</code>, demonstrates an implementation of <code class="highlighter-rouge">BoundedSource</code> and uses the SDK-provided <code class="highlighter-rouge">RangeTracker</code> called <code class="highlighter-rouge">OffsetRangeTracker</code>.</p>
@@ -521,83 +570,43 @@ limitations under the License.
</code></pre>
</div>
-<p><strong>Note:</strong> When you create a source that end-users are going to use, it’s recommended that you do not expose the code for the source itself as demonstrated in the example above, but rather use a wrapping <code class="highlighter-rouge">PTransform</code> instead. See <a href="#ptransform-wrappers">PTransform wrappers</a> to see how and why to avoid exposing your sources.</p>
-
-<h2 id="creating-a-new-sink">Creating a New Sink</h2>
-
-<p>You should create a new sink if you’d like to use the advanced features that the Sink API provides, such as global initialization and finalization that allow the write operation to appear “atomic” (i.e. either all data is written or none is).</p>
-
-<p>A sink represents a resource that can be written to using the <code class="highlighter-rouge">Write</code> transform. A parallel write to a sink consists of three phases:</p>
+<p><strong>Note:</strong> When you create a source that end-users are going to use, we
+recommended that you do not expose the code for the source itself as
+demonstrated in the example above. Use a wrapping <code class="highlighter-rouge">PTransform</code> instead.
+<a href="#ptransform-wrappers">PTransform wrappers</a> discusses why you should avoid
+exposing your sources, and walks through how to create a wrapper.</p>
-<ol>
- <li>A sequential initialization phase. For example, creating a temporary output directory.</li>
- <li>A parallel write phase where workers write bundles of records.</li>
- <li>A sequential finalization phase. For example, merging output files.</li>
-</ol>
-
-<p>For example, if you’d like to write to a new table in a database, you should use the Sink API. In this case, the initializer will create a temporary table, the writer will write rows to it, and the finalizer will rename the table to a final location.</p>
-
-<p>To create a new data sink for your pipeline, you’ll need to provide the format-specific logic that tells the sink how to write bounded data from your pipeline’s <code class="highlighter-rouge">PCollection</code>s to an output sink. The sink writes bundles of data in parallel using multiple workers.</p>
-
-<p>You supply the writing logic by creating the following classes:</p>
-
-<ul>
- <li>
- <p>A subclass of <code class="highlighter-rouge">Sink</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">Sink</code> describes the location or resource to write to. Depending on the type of sink, your <code class="highlighter-rouge">Sink</code> subclass may contain fields such as the path to an output directory on a filesystem or a database table name. <code [...]
- </li>
- <li>
- <p>A subclass of <code class="highlighter-rouge">Writer</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">Writer</code> writes a bundle of elements from an input <code class="highlighter-rouge">PCollection</code> to your designated data sink. <code class="highlighter-rouge">Writer</code> defines two methods: <code class="highlighter-rouge">write()</code>, wh [...]
- </li>
-</ul>
+<h2 id="using-the-filebasedsink-abstraction">Using the FileBasedSink abstraction</h2>
-<h3 id="implementing-the-sink-subclass">Implementing the Sink Subclass</h3>
+<p>If your data source uses files, you can implement the <a href="https://beam.apache.org/releases/pydoc/2.9.0/apache_beam.io.filebasedsink.html">FileBasedSink</a>
+abstraction to create a file-based sink. For other sinks, use <code class="highlighter-rouge">ParDo</code>,
+<code class="highlighter-rouge">GroupByKey</code>, and other transforms offered by the Beam SDK for Python. See the
+<a href="/documentation/io/developing-io-overview/">developing I/O connectors overview</a>
+for more details.</p>
-<p>Your <code class="highlighter-rouge">Sink</code> subclass describes the location or resource to which your pipeline writes its output. This might include a file system location, the name of a database table or dataset, etc.</p>
+<p>When using the <code class="highlighter-rouge">FileBasedSink</code> interface, you must provide the format-specific
+logic that tells the runner how to write bounded data from your pipeline’s
+<code class="highlighter-rouge">PCollection</code>s to an output sink. The runner writes bundles of data in parallel
+using multiple workers.</p>
-<p>To implement a <code class="highlighter-rouge">Sink</code>, your subclass must override the following methods:</p>
+<p>Supply the logic for your file-based sink by implementing the following classes:</p>
<ul>
<li>
- <p><code class="highlighter-rouge">initialize_write</code>: This method performs any necessary initialization before writing to the output location. Services call this method before writing begins. For example, you can use <code class="highlighter-rouge">initialize_write</code> to create a temporary output directory.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">open_writer</code>: This method enables writing a bundle of elements to the sink.</p>
+ <p>A subclass of the abstract base class <code class="highlighter-rouge">FileBasedSink</code>. <code class="highlighter-rouge">FileBasedSink</code>
+describes a location or resource that your pipeline can write to in
+parallel. To avoid exposing your sink to end-users, use the <code class="highlighter-rouge">_</code> prefix when
+creating your <code class="highlighter-rouge">FileBasedSink</code> subclass.</p>
</li>
<li>
- <p><code class="highlighter-rouge">finalize_write</code>:This method finalizes the sink after all data is written to it. Given the result of initialization and an iterable of results from bundle writes, <code class="highlighter-rouge">finalize_write</code> performs finalization after writing and closes the sink. This method is called after all bundle write operations are complete.</p>
+ <p>A user-facing wrapper <code class="highlighter-rouge">PTransform</code> that, as part of the logic, calls
+<code class="highlighter-rouge">Write</code> and passes your <code class="highlighter-rouge">FileBasedSink</code> as a parameter. A user should not
+need to call <code class="highlighter-rouge">Write</code> directly.</p>
</li>
</ul>
-<p><strong>Caution:</strong> <code class="highlighter-rouge">initialize_write</code> and <code class="highlighter-rouge">finalize_write</code> are conceptually called once: at the beginning and end of a <code class="highlighter-rouge">Write</code> transform. However, when you implement these methods, you must ensure that they are <strong>idempotent</strong>, as they may be called multiple times on different machines in the case of failure, retry, or for redundancy.</p>
-
-<h3 id="implementing-the-writer-subclass">Implementing the Writer Subclass</h3>
-
-<p>Your <code class="highlighter-rouge">Writer</code> subclass implements the logic for writing a bundle of elements from a <code class="highlighter-rouge">PCollection</code> to output location defined in your <code class="highlighter-rouge">Sink</code>. Services may instantiate multiple instances of your <code class="highlighter-rouge">Writer</code> in different threads on the same worker, so access to any static members or methods must be thread-safe.</p>
-
-<p>To implement a <code class="highlighter-rouge">Writer</code>, your subclass must override the following abstract methods:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">write</code>: This method writes a value to your <code class="highlighter-rouge">Sink</code> using the current writer.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">close</code>: This method closes the current writer.</p>
- </li>
-</ul>
-
-<h4 id="handling-bundle-ids">Handling Bundle IDs</h4>
-
-<p>When the service calls <code class="highlighter-rouge">Sink.open_writer</code>, it will pass a unique bundle ID for the records to be written. Your <code class="highlighter-rouge">Writer</code> must use this bundle ID to ensure that its output does not interfere with that of other <code class="highlighter-rouge">Writer</code> instances that might have been created in parallel. This is particularly important as the service may retry write operations multiple times in case of failure.</p>
-
-<p>For example, if your <code class="highlighter-rouge">Sink</code>’s output is file-based, your <code class="highlighter-rouge">Writer</code> class might use the bundle ID as a filename suffix to ensure that your <code class="highlighter-rouge">Writer</code> writes its records to a unique output file not used by other <code class="highlighter-rouge">Writer</code>s. You can then have your <code class="highlighter-rouge">Writer</code>’s <code class="highlighter-rouge">close</code> method [...]
-
-<h3 id="convenience-sink-and-writer-base-classes">Convenience Sink and Writer Base Classes</h3>
-
-<p>The Beam SDK for Python contains some convenient abstract base classes to help you create <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Reader</code> classes that work with common data storage formats, like files.</p>
-
-<h4 id="filesink">FileSink</h4>
-
-<p>If your data source uses files, you can derive your <code class="highlighter-rouge">Sink</code> and <code class="highlighter-rouge">Writer</code> classes from the <code class="highlighter-rouge">FileBasedSink</code> and <code class="highlighter-rouge">FileBasedSinkWriter</code> classes, which can be found in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filebasedsink.py">filebasedsink.py</a> module. These classes implement code common sinks that in [...]
+<p>The <code class="highlighter-rouge">FileBasedSink</code> abstract base class implements code that is common to Beam
+sinks that interact with files, including:</p>
<ul>
<li>Setting file headers and footers</li>
@@ -605,87 +614,37 @@ limitations under the License.
<li>Setting the output MIME type</li>
</ul>
-<h2 id="writing-to-a-new-sink">Writing to a New Sink</h2>
-
-<p>Consider a simple key-value storage that writes a given set of key-value pairs to a set of tables. The following is the key-value storage’s API:</p>
+<p><code class="highlighter-rouge">FileBasedSink</code> and its subclasses support writing files to any Beam-supported
+<code class="highlighter-rouge">FileSystem</code> implementations. See the following Beam-provided <code class="highlighter-rouge">FileBasedSink</code>
+implementation for an example:</p>
<ul>
- <li><code class="highlighter-rouge">connect(url)</code> Connects to the storage and returns an access token which can be used to perform further operations.</li>
- <li><code class="highlighter-rouge">open_table(access_token, table_name)</code> Creates a table named ‘table_name’. Returns a table object.</li>
- <li><code class="highlighter-rouge">write_to_table(access_token, table, key, value)</code> Writes a key, value pair to the given table.</li>
- <li><code class="highlighter-rouge">rename_table(access_token, old_name, new_name)</code> Renames the table named ‘old_name’ to ‘new_name’.</li>
+ <li><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py">TextSink</a></li>
</ul>
-<p>The following code demonstrates how to implement the <code class="highlighter-rouge">Sink</code> class for this key-value storage.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>class SimpleKVSink(iobase.Sink):
-
- def __init__(self, simplekv, url, final_table_name):
- self._simplekv = simplekv
- self._url = url
- self._final_table_name = final_table_name
-
- def initialize_write(self):
- access_token = self._simplekv.connect(self._url)
- return access_token
-
- def open_writer(self, access_token, uid):
- table_name = 'table' + uid
- return SimpleKVWriter(self._simplekv, access_token, table_name)
-
- def pre_finalize(self, init_result, writer_results):
- pass
-
- def finalize_write(self, access_token, table_names, pre_finalize_result):
- for i, table_name in enumerate(table_names):
- self._simplekv.rename_table(
- access_token, table_name, self._final_table_name + str(i))
-</code></pre>
-</div>
-
-<p>The following code demonstrates how to implement the <code class="highlighter-rouge">Writer</code> class for this key-value storage.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>class SimpleKVWriter(iobase.Writer):
-
- def __init__(self, simplekv, access_token, table_name):
- self._simplekv = simplekv
- self._access_token = access_token
- self._table_name = table_name
- self._table = self._simplekv.open_table(access_token, table_name)
-
- def write(self, record):
- key, value = record
-
- self._simplekv.write_to_table(self._access_token, self._table, key, value)
-
- def close(self):
- return self._table_name
-</code></pre>
-</div>
-
-<p>The following code demonstrates how to write to the sink using the <code class="highlighter-rouge">Write</code> transform.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>with beam.Pipeline(options=PipelineOptions()) as p:
- kvs = p | 'CreateKVs' >> beam.Create(KVs)
-
- kvs | 'WriteToSimpleKV' >> beam.io.Write(
- SimpleKVSink(simplekv, 'http://url_to_simple_kv/', final_table_name))
-</code></pre>
-</div>
-
-<p><strong>Note:</strong> When you create a sink that end-users are going to use, it’s recommended that you do not expose the code for the sink itself as demonstrated in the example above, but rather use a wrapping <code class="highlighter-rouge">PTransform</code> instead. See <a href="#ptransform-wrappers">PTransform wrappers</a> to see how and why to avoid exposing your sinks.</p>
-
-<h2 id="ptransform-wrappers">PTransform Wrappers</h2>
-
-<p>If you create a new source or sink for your own use, such as for learning purposes, you should create them as explained in the sections above and use them as demonstrated in the examples.</p>
-
-<p>However, when you create a source or sink that end-users are going to use, instead of exposing the source or sink itself, you should create a wrapper <code class="highlighter-rouge">PTransform</code>. Ideally, a source or sink should be exposed to users simply as “something that can be applied in a pipeline”, which is actually a <code class="highlighter-rouge">PTransform</code>. That way, its implementation can be hidden and arbitrarily complex or simple.</p>
-
-<p>The greatest benefit of not exposing the implementation details is that later on you will be able to add additional functionality without breaking the existing implementation for users. For example, if your users’ pipelines read from your source using <code class="highlighter-rouge">beam.io.Read(...)</code> and you want to insert a reshard into the pipeline, all of your users would need to add the reshard themselves (using the <code class="highlighter-rouge">GroupByKey</code> transfo [...]
-
-<p>To avoid exposing your sources and sinks to end-users, it’s recommended that you use the <code class="highlighter-rouge">_</code> prefix when creating your new source and sink classes. Then, create a wrapper <code class="highlighter-rouge">PTransform</code>.</p>
-
-<p>The following examples change the source and sink from the above sections so that they are not exposed to end-users. For the source, rename <code class="highlighter-rouge">CountingSource</code> to <code class="highlighter-rouge">_CountingSource</code>. Then, create the wrapper <code class="highlighter-rouge">PTransform</code>, called <code class="highlighter-rouge">ReadFromCountingSource</code>:</p>
+<h2 id="ptransform-wrappers">PTransform wrappers</h2>
+
+<p>When you create a source or sink that end-users will use, avoid exposing your
+source or sink code. To avoid exposing your sources and sinks to end-users, your
+new classes should use the <code class="highlighter-rouge">_</code> prefix. Then, implement a user-facing
+wrapper <code class="highlighter-rouge">PTransform</code>.`By exposing your source or sink as a transform, your
+implementation is hidden and can be arbitrarily complex or simple. The greatest
+benefit of not exposing implementation details is that later on, you can add
+additional functionality without breaking the existing implementation for users.</p>
+
+<p>For example, if your users’ pipelines read from your source using
+<code class="highlighter-rouge">beam.io.Read</code> and you want to insert a reshard into the pipeline, all
+users would need to add the reshard themselves (using the <code class="highlighter-rouge">GroupByKey</code>
+transform). To solve this, we recommended that you expose the source as a
+composite <code class="highlighter-rouge">PTransform</code> that performs both the read operation and the reshard.</p>
+
+<p>See Beam’s <a href="/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else">PTransform style guide</a>
+for additional information about wrapping with a <code class="highlighter-rouge">PTransform</code>.</p>
+
+<p>The following examples change the source and sink from the above sections so
+that they are not exposed to end-users. For the source, rename <code class="highlighter-rouge">CountingSource</code>
+to <code class="highlighter-rouge">_CountingSource</code>. Then, create the wrapper <code class="highlighter-rouge">PTransform</code>, called
+<code class="highlighter-rouge">ReadFromCountingSource</code>:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>class ReadFromCountingSource(PTransform):
diff --git a/website/generated-content/documentation/io/io-toc/index.html b/website/generated-content/documentation/io/io-toc/index.html
index e25783d..d3e173b 100644
--- a/website/generated-content/documentation/io/io-toc/index.html
+++ b/website/generated-content/documentation/io/io-toc/index.html
@@ -1,400 +1,10 @@
<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Pipeline I/O</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/io/io-toc/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">Languages</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/roadmap/">Roadmap</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- <li>
- <!--
- data-proofer-ignore disables link checking from website test automation.
- GitHub links will not resolve until the markdown source is available on the master branch.
- New pages would fail validation during development / PR test automation.
- -->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/io/io-toc.md" data-proofer-ignore>
- <i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
- </a>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Documentation</span></li>
-<li><a href="/documentation">Using the Documentation</a></li>
-<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline development lifecycle</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/pipelines/design-your-pipeline/">Design Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/create-your-pipeline/">Create Your Pipeline</a></li>
- <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Beam programming guide</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/">Overview</a></li>
- <li><a href="/documentation/programming-guide/#creating-a-pipeline">Pipelines</a></li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">PCollections</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pcollections">Creating a PCollection</a></li>
- <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li>
- <li>
- <span class="section-nav-list-title">Core Beam transforms</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li>
- <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li>
- <li><a href="/documentation/programming-guide/#combine">Combine</a></li>
- <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li>
- <li><a href="/documentation/programming-guide/#partition">Partition</a></li>
- </ul>
- </li>
-
- <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for user code</a></li>
- <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li>
- <li><a href="/documentation/programming-guide/#additional-outputs">Additional outputs</a></li>
- <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Pipeline I/O</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
- <li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Data encoding and type safety</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding basics</a></li>
- <li><a href="/documentation/programming-guide/#specifying-coders">Specifying coders</a></li>
- <li><a href="/documentation/programming-guide/#default-coders-and-the-coderregistry">Default coders and the CoderRegistry</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Windowing</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#windowing">Windowing basics</a></li>
- <li><a href="/documentation/programming-guide/#provided-windowing-functions">Provided windowing functions</a></li>
- <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Setting your PCollection’s windowing function</a></li>
- <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li>
- <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li>
- </ul>
- </li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Triggers</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/programming-guide/#triggers">Trigger basics</a></li>
- <li><a href="/documentation/programming-guide/#event-time-triggers">Event time triggers and the default trigger</a></li>
- <li><a href="/documentation/programming-guide/#processing-time-triggers">Processing time triggers</a></li>
- <li><a href="/documentation/programming-guide/#data-driven-triggers">Data-driven triggers</a></li>
- <li><a href="/documentation/programming-guide/#setting-a-trigger">Setting a trigger</a></li>
- <li><a href="/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
- </ul>
- </li>
- </ul>
-</li>
-<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Learning Resources</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
- <li><a href="/documentation/resources/learning-resources/#articles">Articles</a></li>
- <li><a href="/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
- <li><a href="/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
- <li><a href="/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
- <li><a href="/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
- <li><a href="/documentation/resources/learning-resources/#how-to-contribute">How to Contribute</a></li>
- <li><a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a></li>
- </ul>
-</li>
-<li><a href="https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam">Beam Wiki</a></li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#using-pipeline-io">Using Pipeline I/O</a></li>
- <li><a href="#authoring-read--write-io-transforms">Authoring Read & Write I/O Transforms</a></li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-<h1 id="pipeline-io">Pipeline I/O</h1>
-
-<h2 id="using-pipeline-io">Using Pipeline I/O</h2>
-<ul>
- <li><a href="/documentation/programming-guide#pipeline-io">Programming Guide: Using I/O Transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O Transforms</a></li>
-</ul>
-
-<h2 id="authoring-read--write-io-transforms">Authoring Read & Write I/O Transforms</h2>
-
-<blockquote>
- <p>Note: This guide is still in progress. There is an open issue to finish the guide: <a href="https://issues.apache.org/jira/browse/BEAM-1025">BEAM-1025</a>.</p>
-</blockquote>
-
-<ul>
- <li><a href="/documentation/io/authoring-overview/">Authoring I/O Transforms - Overview</a></li>
- <li><a href="/documentation/io/testing/">Testing I/O Transforms</a></li>
-</ul>
-
-<!-- TODO: commented out until this content is ready.
-* [Authoring I/O Transforms - Python](/documentation/io/authoring-python/)
-* [Authoring I/O Transforms - Java](/documentation/io/authoring-java/)
-
-* [Contributing I/O Transforms](/documentation/io/contributing/)
--->
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
+<html lang="en-US">
+<meta charset="utf-8">
+<title>Redirecting…</title>
+<link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-overview/">
+<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/io/developing-io-overview/">
+<h1>Redirecting…</h1>
+<a href="https://beam.apache.org/documentation/io/developing-io-overview/">Click here if you are not redirected.</a>
+<script>location="https://beam.apache.org/documentation/io/developing-io-overview/"</script>
</html>
diff --git a/website/generated-content/documentation/io/testing/index.html b/website/generated-content/documentation/io/testing/index.html
index ca62b73..05833ae 100644
--- a/website/generated-content/documentation/io/testing/index.html
+++ b/website/generated-content/documentation/io/testing/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
@@ -324,8 +334,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-<p><a href="/documentation/io/io-toc/">Pipeline I/O Table of Contents</a></p>
-
<h2 id="testing-io-transforms-in-apache-beam">Testing I/O Transforms in Apache Beam</h2>
<p><em>Examples and design patterns for testing Apache Beam I/O transforms</em></p>
diff --git a/website/generated-content/documentation/pipelines/create-your-pipeline/index.html b/website/generated-content/documentation/pipelines/create-your-pipeline/index.html
index a3db053..cc49afa 100644
--- a/website/generated-content/documentation/pipelines/create-your-pipeline/index.html
+++ b/website/generated-content/documentation/pipelines/create-your-pipeline/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/pipelines/design-your-pipeline/index.html b/website/generated-content/documentation/pipelines/design-your-pipeline/index.html
index 56c355f..2485058 100644
--- a/website/generated-content/documentation/pipelines/design-your-pipeline/index.html
+++ b/website/generated-content/documentation/pipelines/design-your-pipeline/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/pipelines/test-your-pipeline/index.html b/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
index 63a0550..ed3451e 100644
--- a/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
+++ b/website/generated-content/documentation/pipelines/test-your-pipeline/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index b7f909e..3f2b048 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
@@ -2347,7 +2357,7 @@ want your pipeline to output its result data to an external storage system.
Beam provides read and write transforms for a <a href="/documentation/io/built-in/">number of common data storage
types</a>. If you want your pipeline
to read from or write to a data storage format that isn’t supported by the
-built-in transforms, you can <a href="/documentation/io/io-toc/">implement your own read and write
+built-in transforms, you can <a href="/documentation/io/developing-io-overview/">implement your own read and write
transforms</a>.</p>
<h3 id="pipeline-io-reading-data">5.1. Reading input data</h3>
diff --git a/website/generated-content/documentation/resources/learning-resources/index.html b/website/generated-content/documentation/resources/learning-resources/index.html
index 84d11c7..02fb2fd 100644
--- a/website/generated-content/documentation/resources/learning-resources/index.html
+++ b/website/generated-content/documentation/resources/learning-resources/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/resources/videos-and-podcasts/index.html b/website/generated-content/documentation/resources/videos-and-podcasts/index.html
index b6491ba..e0d1bb3 100644
--- a/website/generated-content/documentation/resources/videos-and-podcasts/index.html
+++ b/website/generated-content/documentation/resources/videos-and-podcasts/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/runners/jstorm/index.html b/website/generated-content/documentation/runners/jstorm/index.html
index e166a5b..382c156 100644
--- a/website/generated-content/documentation/runners/jstorm/index.html
+++ b/website/generated-content/documentation/runners/jstorm/index.html
@@ -153,7 +153,7 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="/documentation">Using the Documentation</a></li>
<li><a href="/documentation/execution-model">Beam Execution Model</a></li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
<ul class="section-nav-list">
@@ -162,7 +162,7 @@
<li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li>
</ul>
</li>
-<li class="section-nav-item--collapsible">
+<li>
<span class="section-nav-list-title">Beam programming guide</span>
<ul class="section-nav-list">
@@ -205,8 +205,18 @@
<ul class="section-nav-list">
<li><a href="/documentation/programming-guide/#pipeline-io">Using I/O transforms</a></li>
- <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li>
- <li><a href="/documentation/io/authoring-overview/">Authoring new I/O transforms</a></li>
+ <li><a href="/documentation/io/built-in/">Built-in I/O connectors</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Developing new I/O connectors</span>
+
+ <ul class="section-nav-list">
+ <li><a href="/documentation/io/developing-io-overview/">Overview: Developing connectors</a></li>
+ <li><a href="/documentation/io/developing-io-java/">Developing connectors (Java)</a></li>
+ <li><a href="/documentation/io/developing-io-python/">Developing connectors (Python)</a></li>
+ </ul>
+ </li>
+
<li><a href="/documentation/io/testing/">Testing I/O transforms</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/feature-comparison/index.html b/website/generated-content/documentation/sdks/feature-comparison/index.html
index e8e6704..c783f05 100644
--- a/website/generated-content/documentation/sdks/feature-comparison/index.html
+++ b/website/generated-content/documentation/sdks/feature-comparison/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/go/index.html b/website/generated-content/documentation/sdks/go/index.html
index 05b1518..f34539b 100644
--- a/website/generated-content/documentation/sdks/go/index.html
+++ b/website/generated-content/documentation/sdks/go/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/java-dependencies/index.html b/website/generated-content/documentation/sdks/java-dependencies/index.html
index f71c021..fd2d188 100644
--- a/website/generated-content/documentation/sdks/java-dependencies/index.html
+++ b/website/generated-content/documentation/sdks/java-dependencies/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/java-extensions/index.html b/website/generated-content/documentation/sdks/java-extensions/index.html
index af43050..e97e2e3 100644
--- a/website/generated-content/documentation/sdks/java-extensions/index.html
+++ b/website/generated-content/documentation/sdks/java-extensions/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/java-thirdparty/index.html b/website/generated-content/documentation/sdks/java-thirdparty/index.html
index b56a970..eaf89ac 100644
--- a/website/generated-content/documentation/sdks/java-thirdparty/index.html
+++ b/website/generated-content/documentation/sdks/java-thirdparty/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/java/euphoria/index.html b/website/generated-content/documentation/sdks/java/euphoria/index.html
index 67ade47..c8d6987 100644
--- a/website/generated-content/documentation/sdks/java/euphoria/index.html
+++ b/website/generated-content/documentation/sdks/java/euphoria/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/java/index.html b/website/generated-content/documentation/sdks/java/index.html
index 33fa31e..615c95e 100644
--- a/website/generated-content/documentation/sdks/java/index.html
+++ b/website/generated-content/documentation/sdks/java/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/java/testing/nexmark/index.html b/website/generated-content/documentation/sdks/java/testing/nexmark/index.html
index b3f7098..9aef2cc 100644
--- a/website/generated-content/documentation/sdks/java/testing/nexmark/index.html
+++ b/website/generated-content/documentation/sdks/java/testing/nexmark/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/python-custom-io/index.html b/website/generated-content/documentation/sdks/python-custom-io/index.html
index 04d3a0e..e74d71d 100644
--- a/website/generated-content/documentation/sdks/python-custom-io/index.html
+++ b/website/generated-content/documentation/sdks/python-custom-io/index.html
@@ -1,803 +1,10 @@
<!DOCTYPE html>
-<!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<html lang="en">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<head>
- <meta charset="utf-8">
- <meta http-equiv="X-UA-Compatible" content="IE=edge">
- <meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Apache Beam: Creating New Sources and Sinks with the Python SDK</title>
- <meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
-">
- <link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
- <link rel="stylesheet" href="/css/site.css">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
- <script src="/js/bootstrap.min.js"></script>
- <script src="/js/language-switch.js"></script>
- <script src="/js/fix-menu.js"></script>
- <script src="/js/section-nav.js"></script>
- <script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/sdks/python-custom-io/" data-proofer-ignore>
- <link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
- <link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
- <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.4.1/css/all.css" integrity="sha384-5sAR7xN1Nv6T6+dT2mhtzEpVJvfS3NScPQTrOxhwjIuvcA67KV2R5Jz6kr4abQsz" crossorigin="anonymous">
- <script>
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-73650088-1', 'auto');
- ga('send', 'pageview');
- </script>
-</head>
-
- <body class="body" data-spy="scroll" data-target=".page-nav" data-offset="0">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<nav class="header navbar navbar-fixed-top">
- <div class="navbar-header">
- <button type="button" class="navbar-toggle" aria-expanded="false" aria-controls="navbar">
- <span class="sr-only">Toggle navigation</span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- <span class="icon-bar"></span>
- </button>
-
- <a href="/" class="navbar-brand" >
- <img alt="Brand" style="height: 25px" src="/images/beam_logo_navbar.png">
- </a>
- </div>
-
- <div class="navbar-mask closed"></div>
-
- <div id="navbar" class="navbar-container closed">
- <ul class="nav navbar-nav">
- <li>
- <a href="/get-started/beam-overview/">Get Started</a>
- </li>
- <li>
- <a href="/documentation/">Documentation</a>
- </li>
- <li>
- <a href="/documentation/sdks/java/">Languages</a>
- </li>
- <li>
- <a href="/documentation/runners/capability-matrix/">RUNNERS</a>
- </li>
- <li>
- <a href="/roadmap/">Roadmap</a>
- </li>
- <li>
- <a href="/contribute/">Contribute</a>
- </li>
- <li>
- <a href="/community/contact-us/">Community</a>
- </li>
- <li><a href="/blog">Blog</a></li>
- </ul>
- <ul class="nav navbar-nav navbar-right">
- <li class="dropdown">
- <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
- <ul class="dropdown-menu dropdown-menu-right">
- <li><a href="http://www.apache.org/">ASF Homepage</a></li>
- <li><a href="http://www.apache.org/licenses/">License</a></li>
- <li><a href="http://www.apache.org/security/">Security</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
- <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
- <li><a href="https://www.apache.org/foundation/policies/conduct">Code of Conduct</a></li>
- </ul>
- </li>
- <li>
- <!--
- data-proofer-ignore disables link checking from website test automation.
- GitHub links will not resolve until the markdown source is available on the master branch.
- New pages would fail validation during development / PR test automation.
- -->
- <a href="https://github.com/apache/beam/edit/master/website/src/documentation/sdks/python-custom-io.md" data-proofer-ignore>
- <i class="far fa-edit fa-lg" alt="Edit on GitHub" title="Edit on GitHub"></i>
- </a>
- </li>
- </ul>
- </div>
-</nav>
-
- <div class="clearfix container-main-content">
- <div class="section-nav closed" data-offset-top="90" data-offset-bottom="500">
- <span class="section-nav-back glyphicon glyphicon-menu-left"></span>
- <nav>
- <ul class="section-nav-list" data-section-nav>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<li><span class="section-nav-list-main-title">Languages</span></li>
-
-<li>
- <span class="section-nav-list-title">Java</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/sdks/java/">Java SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/javadoc/2.9.0/" target="_blank">Java SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
- </li>
- <li><a href="/documentation/sdks/java-dependencies/">Java SDK dependencies</a></li>
- <li><a href="/documentation/sdks/java-extensions/">Java SDK extensions</a></li>
- <li><a href="/documentation/sdks/java-thirdparty/">Java 3rd party extensions</a></li>
- <li><a href="/documentation/sdks/java/testing/nexmark/">Nexmark benchmark suite</a></li>
- </ul>
-</li>
-
-<li>
- <span class="section-nav-list-title">Python</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/sdks/python/">Python SDK overview</a></li>
- <li><a href="https://beam.apache.org/releases/pydoc/2.9.0/" target="_blank">Python SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
- </li>
- <li><a href="/documentation/sdks/python-dependencies/">Python SDK dependencies</a></li>
- <li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
- <li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
- <li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
- </ul>
-</li>
-
-<li>
- <span class="section-nav-list-title">Go</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/sdks/go/">Go SDK overview</a></li>
- <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam" target="_blank">Go SDK API reference <img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a>
- </li>
- </ul>
-</li>
-
-<li>
- <span class="section-nav-list-title">SQL</span>
- <ul class="section-nav-list">
- <li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
- <li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
- <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
- <li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">SQL Reference</span>
-
- <ul class="section-nav-list">
- <li><a href="/documentation/dsls/sql/data-types/">Data types</a></li>
- <li><a href="/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
- <li><a href="/documentation/dsls/sql/select/">SELECT</a></li>
- <li><a href="/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
- <li><a href="/documentation/dsls/sql/joins/">Joins</a></li>
- <li><a href="/documentation/dsls/sql/scalar-functions/">Scalar functions</a></li>
- <li><a href="/documentation/dsls/sql/aggregate-functions/">Aggregate functions</a></li>
- <li><a href="/documentation/dsls/sql/user-defined-functions/">User-defined functions</a></li>
- <li><a href="/documentation/dsls/sql/set/">SET Pipeline Options</a></li>
- </ul>
- </li>
- </ul>
-</li>
-
- </ul>
- </nav>
- </div>
-
- <nav class="page-nav clearfix" data-offset-top="90" data-offset-bottom="500">
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-
-
-<ul class="nav">
- <li><a href="#why-create-a-new-source-or-sink">Why Create a New Source or Sink</a></li>
- <li><a href="#basic-code-reqs">Basic Code Requirements for New Sources and Sinks</a>
- <ul>
- <li><a href="#serializability">Serializability</a></li>
- <li><a href="#immutability">Immutability</a></li>
- <li><a href="#thread-safety">Thread-Safety</a></li>
- <li><a href="#testability">Testability</a></li>
- </ul>
- </li>
- <li><a href="#creating-a-new-source">Creating a New Source</a>
- <ul>
- <li><a href="#implementing-the-boundedsource-subclass">Implementing the BoundedSource Subclass</a></li>
- <li><a href="#implementing-the-rangetracker-subclass">Implementing the RangeTracker Subclass</a></li>
- <li><a href="#convenience-source-base-classes">Convenience Source Base Classes</a></li>
- </ul>
- </li>
- <li><a href="#reading-from-a-new-source">Reading from a New Source</a></li>
- <li><a href="#creating-a-new-sink">Creating a New Sink</a>
- <ul>
- <li><a href="#implementing-the-sink-subclass">Implementing the Sink Subclass</a></li>
- <li><a href="#implementing-the-writer-subclass">Implementing the Writer Subclass</a></li>
- <li><a href="#convenience-sink-and-writer-base-classes">Convenience Sink and Writer Base Classes</a></li>
- </ul>
- </li>
- <li><a href="#writing-to-a-new-sink">Writing to a New Sink</a></li>
- <li><a href="#ptransform-wrappers">PTransform Wrappers</a></li>
-</ul>
-
-
- </nav>
-
- <div class="body__contained body__section-nav">
- <!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<h1 id="creating-new-sources-and-sinks-with-the-python-sdk">Creating New Sources and Sinks with the Python SDK</h1>
-
-<p>The Apache Beam SDK for Python provides an extensible API that you can use to create new data sources and sinks. This tutorial shows how to create new sources and sinks using <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">Beam’s Source and Sink API</a>.</p>
-
-<ul>
- <li>Create a new source by extending the <code class="highlighter-rouge">BoundedSource</code> and <code class="highlighter-rouge">RangeTracker</code> interfaces.</li>
- <li>Create a new sink by implementing the <code class="highlighter-rouge">Sink</code> and <code class="highlighter-rouge">Writer</code> classes.</li>
-</ul>
-
-<h2 id="why-create-a-new-source-or-sink">Why Create a New Source or Sink</h2>
-
-<p>You’ll need to create a new source or sink if you want your pipeline to read data from (or write data to) a storage system for which the Beam SDK for Python does not provide <a href="/documentation/programming-guide/#pipeline-io">native support</a>.</p>
-
-<p>In simple cases, you may not need to create a new source or sink. For example, if you need to read data from an SQL database using an arbitrary query, none of the advanced Source API features would benefit you. Likewise, if you’d like to write data to a third-party API via a protocol that lacks deduplication support, the Sink API wouldn’t benefit you. In such cases it makes more sense to use a <code class="highlighter-rouge">ParDo</code>.</p>
-
-<p>However, if you’d like to use advanced features such as dynamic splitting and size estimation, you should use Beam’s APIs and create a new source or sink.</p>
-
-<h2 id="basic-code-reqs">Basic Code Requirements for New Sources and Sinks</h2>
-
-<p>Services use the classes you provide to read and/or write data using multiple worker instances in parallel. As such, the code you provide for <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Sink</code> subclasses must meet some basic requirements:</p>
-
-<h3 id="serializability">Serializability</h3>
-
-<p>Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass must be serializable. The service may create multiple instances of your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass to be sent to multiple remote workers to facilitate reading or writing in parallel. Note that the <em>way</em> the source and sink objects are serialized is runner specific.</p>
-
-<h3 id="immutability">Immutability</h3>
-
-<p>Your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass must be effectively immutable. You should only use mutable state in your <code class="highlighter-rouge">Source</code> or <code class="highlighter-rouge">Sink</code> subclass if you are using lazy evaluation of expensive computations that you need to implement the source.</p>
-
-<h3 id="thread-safety">Thread-Safety</h3>
-
-<p>Your code must be thread-safe. The Beam SDK for Python provides the <code class="highlighter-rouge">RangeTracker</code> class to make this easier.</p>
-
-<h3 id="testability">Testability</h3>
-
-<p>It is critical to exhaustively unit-test all of your <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Sink</code> subclasses. A minor implementation error can lead to data corruption or data loss (such as skipping or duplicating records) that can be hard to detect.</p>
-
-<p>You can use test harnesses and utility methods available in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/source_test_utils.py">source_test_utils module</a> to develop tests for your source.</p>
-
-<h2 id="creating-a-new-source">Creating a New Source</h2>
-
-<p>You should create a new source if you’d like to use the advanced features that the Source API provides:</p>
-
-<ul>
- <li>Dynamic splitting</li>
- <li>Progress estimation</li>
- <li>Size estimation</li>
- <li>Splitting into parts of particular size recommended by the service</li>
-</ul>
-
-<p>For example, if you’d like to read from a new file format that contains many records per file, or if you’d like to read from a key-value store that supports read operations in sorted key order.</p>
-
-<p>To create a new data source for your pipeline, you’ll need to provide the format-specific logic that tells the service how to read data from your input source, and how to split your data source into multiple parts so that multiple worker instances can read your data in parallel.</p>
-
-<p>You supply the logic for your new source by creating the following classes:</p>
-
-<ul>
- <li>A subclass of <code class="highlighter-rouge">BoundedSource</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">BoundedSource</code> is a source that reads a finite amount of input records. The class describes the data you want to read, including the data’s location and parameters (such as how much data to read).</li>
- <li>A subclass of <code class="highlighter-rouge">RangeTracker</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">RangeTracker</code> is a thread-safe object used to manage a range for a given position type.</li>
-</ul>
-
-<h3 id="implementing-the-boundedsource-subclass">Implementing the BoundedSource Subclass</h3>
-
-<p><code class="highlighter-rouge">BoundedSource</code> represents a finite data set from which the service reads, possibly in parallel. <code class="highlighter-rouge">BoundedSource</code> contains a set of methods that the service uses to split the data set for reading by multiple remote workers.</p>
-
-<p>To implement a <code class="highlighter-rouge">BoundedSource</code>, your subclass must override the following methods:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">estimate_size</code>: Services use this method to estimate the <em>total size</em> of your data, in bytes. This estimate is in terms of external storage size, before performing decompression or other processing.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">split</code>: Service use this method to split your finite data into bundles of a given size.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">get_range_tracker</code>: Services use this method to get the <code class="highlighter-rouge">RangeTracker</code> for a given position range, and use the information to report progress and perform dynamic splitting of sources.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">read</code>: This method returns an iterator that reads data from the source, with respect to the boundaries defined by the given <code class="highlighter-rouge">RangeTracker</code> object.</p>
- </li>
-</ul>
-
-<h3 id="implementing-the-rangetracker-subclass">Implementing the RangeTracker Subclass</h3>
-
-<p>A <code class="highlighter-rouge">RangeTracker</code> is a thread-safe object used to manage the current range and current position of the reader of a <code class="highlighter-rouge">BoundedSource</code> and protect concurrent access to them.</p>
-
-<p>To implement a <code class="highlighter-rouge">RangeTracker</code>, you should first familiarize yourself with the following definitions:</p>
-
-<ul>
- <li>
- <p><strong>Position-based sources</strong> - A position-based source can be described by a range of positions of an ordered type, and the records read by the source can be described by positions of that type. For example, for a record within a file, the position can be the starting byte offset of the record. The position type for the record in this case is <code class="highlighter-rouge">long</code>.</p>
-
- <p>The main requirement for position-based sources is <strong>associativity</strong>: Reading records in position range ‘[A, B)’ and records in position range ‘[B, C)’ should give the same records as reading records in position range ‘[A, C)’, where ‘A’ <= ‘B’ <= ‘C’. This property ensures that no matter how many arbitrary sub-ranges a range of positions is split into, the total set of records they describe stays the same.</p>
-
- <p>The other important property is how the source’s range relates to positions of records in the source. In many sources each record can be identified by a unique starting position. In this case:</p>
-
- <ul>
- <li>All records returned by a source ‘[A, B)’ must have starting positions in this range.</li>
- <li>All but the last record should end within this range. The last record may or may not extend past the end of the range.</li>
- <li>Records must not overlap.</li>
- </ul>
-
- <p>Such sources should define “read ‘[A, B)’” as “read from the first record starting at or after ‘A’, up to but not including the first record starting at or after ‘B’”.</p>
-
- <p>Some examples of such sources include reading lines or CSV from a text file, reading keys and values from a database, etc.</p>
-
- <p>The concept of <em>split points</em> allows to extend the definitions for dealing with sources where some records cannot be identified by a unique starting position.</p>
- </li>
- <li>
- <p><strong>Split points</strong> - A split point describes a record that is the first one returned when reading the range from and including position <strong>A</strong> up to infinity (i.e. [A, infinity)).</p>
-
- <p>Some sources may have records that are not directly addressable. For example, imagine a file format consisting of a sequence of compressed blocks. Each block can be assigned an offset, but records within the block cannot be directly addressed without decompressing the block. Let us refer to this hypothetical format as <em>CBF (Compressed Blocks Format)</em>.</p>
-
- <p>Many such formats can still satisfy the associativity property. For example, in CBF, reading [A, B) can mean “read all the records in all blocks whose starting offset is in [A, B)”.</p>
-
- <p>To support such complex formats, Beam introduces the notion of <em>split points</em>. A record is a split point if there exists a position <strong>A</strong> such that the record is the first one to be returned when reading the range [A, infinity). In CBF, the only split points would be the first records in each block.</p>
-
- <p>Split points allow us to define the meaning of a record’s position and a source’s range in the following cases:</p>
-
- <ul>
- <li>For a record that is at a split point, its position is defined to be the largest <strong>A</strong> such that reading a source with the range [A, infinity) returns this record.</li>
- <li>Positions of other records are only required to be non-decreasing.</li>
- <li>Reading the source [A, B) must return records starting from the first split point at or after <strong>A</strong>, up to but not including the first split point at or after <strong>B</strong>. In particular, this means that the first record returned by a source MUST always be a split point.</li>
- <li>Positions of split points must be unique.</li>
- </ul>
-
- <p>As a result, for any decomposition of the full range of the source into position ranges, the total set of records will be the full set of records in the source, and each record will be read exactly once.</p>
- </li>
- <li>
- <p><strong>Consumed positions</strong> - Consumed positions refer to records that have been read.</p>
-
- <p>As the source is being read, and records read from it are being passed to the downstream transforms in the pipeline, we say that positions in the source are being <em>consumed</em>. When a reader has read a record (or promised to a caller that a record will be returned), positions up to and including the record’s start position are considered <em>consumed</em>.</p>
-
- <p>Dynamic splitting can happen only at <em>unconsumed</em> positions. If the reader just returned a record at offset 42 in a file, dynamic splitting can happen only at offset 43 or beyond. Otherwise, that record could be read twice (by the current reader and the reader of the new task).</p>
- </li>
-</ul>
-
-<h4 id="rangetracker-methods">RangeTracker Methods</h4>
-
-<p>To implement a <code class="highlighter-rouge">RangeTracker</code>, your subclass must override the following methods:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">start_position</code>: Returns the starting position of the current range, inclusive.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">stop_position</code>: Returns the ending position of the current range, exclusive.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">try_claim</code>: This method is used to determine if a record at a split point is within the range. This method should modify the internal state of the <code class="highlighter-rouge">RangeTracker</code> by updating the last-consumed position to the given starting <code class="highlighter-rouge">position</code> of the record being read by the source. The method returns true if the given position falls within the current range.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">set_current_position</code>: This method updates the last-consumed position to the given starting position of a record being read by a source. You can invoke this method for records that do not start at split points, and this should modify the internal state of the <code class="highlighter-rouge">RangeTracker</code>. If the record starts at a split point, you must invoke <code class="highlighter-rouge">try_claim</code> instead of this method.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">position_at_fraction</code>: Given a fraction within the range [0.0, 1.0), this method will return the position at the given fraction compared to the position range [<code class="highlighter-rouge">self.start_position</code>, <code class="highlighter-rouge">self.stop_position</code>).</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">try_split</code>: This method attempts to split the current range into two parts around a suggested position. It is allowed to split at a different position, but in most cases it will split at the suggested position.</p>
- </li>
-</ul>
-
-<p>This method splits the current range [<code class="highlighter-rouge">self.start_position</code>, <code class="highlighter-rouge">self.stop_position</code>) into a “primary” part [<code class="highlighter-rouge">self.start_position</code>, <code class="highlighter-rouge">split_position</code>), and a “residual” part [<code class="highlighter-rouge">split_position</code>, <code class="highlighter-rouge">self.stop_position</code>), assuming that <code class="highlighter-rouge">split_pos [...]
-
-<p>If <code class="highlighter-rouge">split_position</code> has already been consumed, the method returns <code class="highlighter-rouge">None</code>. Otherwise, it updates the current range to be the primary and returns a tuple (<code class="highlighter-rouge">split_position</code>, <code class="highlighter-rouge">split_fraction</code>). <code class="highlighter-rouge">split_fraction</code> should be the fraction of size of range [<code class="highlighter-rouge">self.start_position</c [...]
-
-<ul>
- <li><code class="highlighter-rouge">fraction_consumed</code>: Returns the approximate fraction of consumed positions in the source.</li>
-</ul>
-
-<p><strong>Note:</strong> Methods of class <code class="highlighter-rouge">iobase.RangeTracker</code> may be invoked by multiple threads, hence this class must be made thread-safe, for example, by using a single lock object.</p>
-
-<h3 id="convenience-source-base-classes">Convenience Source Base Classes</h3>
-
-<p>The Beam SDK for Python contains some convenient abstract base classes to help you easily create new sources.</p>
-
-<h4 id="filebasedsource">FileBasedSource</h4>
-
-<p><code class="highlighter-rouge">FileBasedSource</code> is a framework for developing sources for new file types. You can derive your <code class="highlighter-rouge">BoundedSource</code> class from the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filebasedsource.py">FileBasedSource</a> class.</p>
-
-<p>To create a source for a new file type, you need to create a sub-class of <code class="highlighter-rouge">FileBasedSource</code>. Sub-classes of <code class="highlighter-rouge">FileBasedSource</code> must implement the method <code class="highlighter-rouge">FileBasedSource.read_records()</code>.</p>
-
-<p>See <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">AvroSource</a> for an example implementation of <code class="highlighter-rouge">FileBasedSource</code>.</p>
-
-<h2 id="reading-from-a-new-source">Reading from a New Source</h2>
-
-<p>The following example, <code class="highlighter-rouge">CountingSource</code>, demonstrates an implementation of <code class="highlighter-rouge">BoundedSource</code> and uses the SDK-provided <code class="highlighter-rouge">RangeTracker</code> called <code class="highlighter-rouge">OffsetRangeTracker</code>.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>class CountingSource(iobase.BoundedSource):
-
- def __init__(self, count):
- self.records_read = Metrics.counter(self.__class__, 'recordsRead')
- self._count = count
-
- def estimate_size(self):
- return self._count
-
- def get_range_tracker(self, start_position, stop_position):
- if start_position is None:
- start_position = 0
- if stop_position is None:
- stop_position = self._count
-
- return OffsetRangeTracker(start_position, stop_position)
-
- def read(self, range_tracker):
- for i in range(self._count):
- if not range_tracker.try_claim(i):
- return
- self.records_read.inc()
- yield i
-
- def split(self, desired_bundle_size, start_position=None,
- stop_position=None):
- if start_position is None:
- start_position = 0
- if stop_position is None:
- stop_position = self._count
-
- bundle_start = start_position
- while bundle_start < self._count:
- bundle_stop = max(self._count, bundle_start + desired_bundle_size)
- yield iobase.SourceBundle(weight=(bundle_stop - bundle_start),
- source=self,
- start_position=bundle_start,
- stop_position=bundle_stop)
- bundle_start = bundle_stop
-</code></pre>
-</div>
-
-<p>To read data from the source in your pipeline, use the <code class="highlighter-rouge">Read</code> transform:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>with beam.Pipeline(options=PipelineOptions()) as p:
- numbers = p | 'ProduceNumbers' >> beam.io.Read(CountingSource(count))
-</code></pre>
-</div>
-
-<p><strong>Note:</strong> When you create a source that end-users are going to use, it’s recommended that you do not expose the code for the source itself as demonstrated in the example above, but rather use a wrapping <code class="highlighter-rouge">PTransform</code> instead. See <a href="#ptransform-wrappers">PTransform wrappers</a> to see how and why to avoid exposing your sources.</p>
-
-<h2 id="creating-a-new-sink">Creating a New Sink</h2>
-
-<p>You should create a new sink if you’d like to use the advanced features that the Sink API provides, such as global initialization and finalization that allow the write operation to appear “atomic” (i.e. either all data is written or none is).</p>
-
-<p>A sink represents a resource that can be written to using the <code class="highlighter-rouge">Write</code> transform. A parallel write to a sink consists of three phases:</p>
-
-<ol>
- <li>A sequential initialization phase. For example, creating a temporary output directory.</li>
- <li>A parallel write phase where workers write bundles of records.</li>
- <li>A sequential finalization phase. For example, merging output files.</li>
-</ol>
-
-<p>For example, if you’d like to write to a new table in a database, you should use the Sink API. In this case, the initializer will create a temporary table, the writer will write rows to it, and the finalizer will rename the table to a final location.</p>
-
-<p>To create a new data sink for your pipeline, you’ll need to provide the format-specific logic that tells the sink how to write bounded data from your pipeline’s <code class="highlighter-rouge">PCollection</code>s to an output sink. The sink writes bundles of data in parallel using multiple workers.</p>
-
-<p>You supply the writing logic by creating the following classes:</p>
-
-<ul>
- <li>
- <p>A subclass of <code class="highlighter-rouge">Sink</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">Sink</code> describes the location or resource to write to. Depending on the type of sink, your <code class="highlighter-rouge">Sink</code> subclass may contain fields such as the path to an output directory on a filesystem or a database table name. <code [...]
- </li>
- <li>
- <p>A subclass of <code class="highlighter-rouge">Writer</code>, which you can find in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">iobase.py</a> module. <code class="highlighter-rouge">Writer</code> writes a bundle of elements from an input <code class="highlighter-rouge">PCollection</code> to your designated data sink. <code class="highlighter-rouge">Writer</code> defines two methods: <code class="highlighter-rouge">write()</code>, wh [...]
- </li>
-</ul>
-
-<h3 id="implementing-the-sink-subclass">Implementing the Sink Subclass</h3>
-
-<p>Your <code class="highlighter-rouge">Sink</code> subclass describes the location or resource to which your pipeline writes its output. This might include a file system location, the name of a database table or dataset, etc.</p>
-
-<p>To implement a <code class="highlighter-rouge">Sink</code>, your subclass must override the following methods:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">initialize_write</code>: This method performs any necessary initialization before writing to the output location. Services call this method before writing begins. For example, you can use <code class="highlighter-rouge">initialize_write</code> to create a temporary output directory.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">open_writer</code>: This method enables writing a bundle of elements to the sink.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">finalize_write</code>:This method finalizes the sink after all data is written to it. Given the result of initialization and an iterable of results from bundle writes, <code class="highlighter-rouge">finalize_write</code> performs finalization after writing and closes the sink. This method is called after all bundle write operations are complete.</p>
- </li>
-</ul>
-
-<p><strong>Caution:</strong> <code class="highlighter-rouge">initialize_write</code> and <code class="highlighter-rouge">finalize_write</code> are conceptually called once: at the beginning and end of a <code class="highlighter-rouge">Write</code> transform. However, when you implement these methods, you must ensure that they are <strong>idempotent</strong>, as they may be called multiple times on different machines in the case of failure, retry, or for redundancy.</p>
-
-<h3 id="implementing-the-writer-subclass">Implementing the Writer Subclass</h3>
-
-<p>Your <code class="highlighter-rouge">Writer</code> subclass implements the logic for writing a bundle of elements from a <code class="highlighter-rouge">PCollection</code> to output location defined in your <code class="highlighter-rouge">Sink</code>. Services may instantiate multiple instances of your <code class="highlighter-rouge">Writer</code> in different threads on the same worker, so access to any static members or methods must be thread-safe.</p>
-
-<p>To implement a <code class="highlighter-rouge">Writer</code>, your subclass must override the following abstract methods:</p>
-
-<ul>
- <li>
- <p><code class="highlighter-rouge">write</code>: This method writes a value to your <code class="highlighter-rouge">Sink</code> using the current writer.</p>
- </li>
- <li>
- <p><code class="highlighter-rouge">close</code>: This method closes the current writer.</p>
- </li>
-</ul>
-
-<h4 id="handling-bundle-ids">Handling Bundle IDs</h4>
-
-<p>When the service calls <code class="highlighter-rouge">Sink.open_writer</code>, it will pass a unique bundle ID for the records to be written. Your <code class="highlighter-rouge">Writer</code> must use this bundle ID to ensure that its output does not interfere with that of other <code class="highlighter-rouge">Writer</code> instances that might have been created in parallel. This is particularly important as the service may retry write operations multiple times in case of failure.</p>
-
-<p>For example, if your <code class="highlighter-rouge">Sink</code>’s output is file-based, your <code class="highlighter-rouge">Writer</code> class might use the bundle ID as a filename suffix to ensure that your <code class="highlighter-rouge">Writer</code> writes its records to a unique output file not used by other <code class="highlighter-rouge">Writer</code>s. You can then have your <code class="highlighter-rouge">Writer</code>’s <code class="highlighter-rouge">close</code> method [...]
-
-<h3 id="convenience-sink-and-writer-base-classes">Convenience Sink and Writer Base Classes</h3>
-
-<p>The Beam SDK for Python contains some convenient abstract base classes to help you create <code class="highlighter-rouge">Source</code> and <code class="highlighter-rouge">Reader</code> classes that work with common data storage formats, like files.</p>
-
-<h4 id="filesink">FileSink</h4>
-
-<p>If your data source uses files, you can derive your <code class="highlighter-rouge">Sink</code> and <code class="highlighter-rouge">Writer</code> classes from the <code class="highlighter-rouge">FileBasedSink</code> and <code class="highlighter-rouge">FileBasedSinkWriter</code> classes, which can be found in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filebasedsink.py">filebasedsink.py</a> module. These classes implement code common sinks that in [...]
-
-<ul>
- <li>Setting file headers and footers</li>
- <li>Sequential record writing</li>
- <li>Setting the output MIME type</li>
-</ul>
-
-<h2 id="writing-to-a-new-sink">Writing to a New Sink</h2>
-
-<p>Consider a simple key-value storage that writes a given set of key-value pairs to a set of tables. The following is the key-value storage’s API:</p>
-
-<ul>
- <li><code class="highlighter-rouge">connect(url)</code> Connects to the storage and returns an access token which can be used to perform further operations.</li>
- <li><code class="highlighter-rouge">open_table(access_token, table_name)</code> Creates a table named ‘table_name’. Returns a table object.</li>
- <li><code class="highlighter-rouge">write_to_table(access_token, table, key, value)</code> Writes a key, value pair to the given table.</li>
- <li><code class="highlighter-rouge">rename_table(access_token, old_name, new_name)</code> Renames the table named ‘old_name’ to ‘new_name’.</li>
-</ul>
-
-<p>The following code demonstrates how to implement the <code class="highlighter-rouge">Sink</code> class for this key-value storage.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>class SimpleKVSink(iobase.Sink):
-
- def __init__(self, simplekv, url, final_table_name):
- self._simplekv = simplekv
- self._url = url
- self._final_table_name = final_table_name
-
- def initialize_write(self):
- access_token = self._simplekv.connect(self._url)
- return access_token
-
- def open_writer(self, access_token, uid):
- table_name = 'table' + uid
- return SimpleKVWriter(self._simplekv, access_token, table_name)
-
- def pre_finalize(self, init_result, writer_results):
- pass
-
- def finalize_write(self, access_token, table_names, pre_finalize_result):
- for i, table_name in enumerate(table_names):
- self._simplekv.rename_table(
- access_token, table_name, self._final_table_name + str(i))
-</code></pre>
-</div>
-
-<p>The following code demonstrates how to implement the <code class="highlighter-rouge">Writer</code> class for this key-value storage.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>class SimpleKVWriter(iobase.Writer):
-
- def __init__(self, simplekv, access_token, table_name):
- self._simplekv = simplekv
- self._access_token = access_token
- self._table_name = table_name
- self._table = self._simplekv.open_table(access_token, table_name)
-
- def write(self, record):
- key, value = record
-
- self._simplekv.write_to_table(self._access_token, self._table, key, value)
-
- def close(self):
- return self._table_name
-</code></pre>
-</div>
-
-<p>The following code demonstrates how to write to the sink using the <code class="highlighter-rouge">Write</code> transform.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>with beam.Pipeline(options=PipelineOptions()) as p:
- kvs = p | 'CreateKVs' >> beam.Create(KVs)
-
- kvs | 'WriteToSimpleKV' >> beam.io.Write(
- SimpleKVSink(simplekv, 'http://url_to_simple_kv/', final_table_name))
-</code></pre>
-</div>
-
-<p><strong>Note:</strong> When you create a sink that end-users are going to use, it’s recommended that you do not expose the code for the sink itself as demonstrated in the example above, but rather use a wrapping <code class="highlighter-rouge">PTransform</code> instead. See <a href="#ptransform-wrappers">PTransform wrappers</a> to see how and why to avoid exposing your sinks.</p>
-
-<h2 id="ptransform-wrappers">PTransform Wrappers</h2>
-
-<p>If you create a new source or sink for your own use, such as for learning purposes, you should create them as explained in the sections above and use them as demonstrated in the examples.</p>
-
-<p>However, when you create a source or sink that end-users are going to use, instead of exposing the source or sink itself, you should create a wrapper <code class="highlighter-rouge">PTransform</code>. Ideally, a source or sink should be exposed to users simply as “something that can be applied in a pipeline”, which is actually a <code class="highlighter-rouge">PTransform</code>. That way, its implementation can be hidden and arbitrarily complex or simple.</p>
-
-<p>The greatest benefit of not exposing the implementation details is that later on you will be able to add additional functionality without breaking the existing implementation for users. For example, if your users’ pipelines read from your source using <code class="highlighter-rouge">beam.io.Read(...)</code> and you want to insert a reshard into the pipeline, all of your users would need to add the reshard themselves (using the <code class="highlighter-rouge">GroupByKey</code> transfo [...]
-
-<p>To avoid exposing your sources and sinks to end-users, it’s recommended that you use the <code class="highlighter-rouge">_</code> prefix when creating your new source and sink classes. Then, create a wrapper <code class="highlighter-rouge">PTransform</code>.</p>
-
-<p>The following examples change the source and sink from the above sections so that they are not exposed to end-users. For the source, rename <code class="highlighter-rouge">CountingSource</code> to <code class="highlighter-rouge">_CountingSource</code>. Then, create the wrapper <code class="highlighter-rouge">PTransform</code>, called <code class="highlighter-rouge">ReadFromCountingSource</code>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>class ReadFromCountingSource(PTransform):
-
- def __init__(self, count):
- super(ReadFromCountingSource, self).__init__()
- self._count = count
-
- def expand(self, pcoll):
- return pcoll | iobase.Read(_CountingSource(self._count))
-</code></pre>
-</div>
-
-<p>Finally, read from the source:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>p = beam.Pipeline(options=PipelineOptions())
-numbers = p | 'ProduceNumbers' >> ReadFromCountingSource(count)
-</code></pre>
-</div>
-
-<p>For the sink, rename <code class="highlighter-rouge">SimpleKVSink</code> to <code class="highlighter-rouge">_SimpleKVSink</code>. Then, create the wrapper <code class="highlighter-rouge">PTransform</code>, called <code class="highlighter-rouge">WriteToKVSink</code>:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>class WriteToKVSink(PTransform):
-
- def __init__(self, simplekv, url, final_table_name, **kwargs):
- self._simplekv = simplekv
- super(WriteToKVSink, self).__init__(**kwargs)
- self._url = url
- self._final_table_name = final_table_name
-
- def expand(self, pcoll):
- return pcoll | iobase.Write(_SimpleKVSink(self._simplekv,
- self._url,
- self._final_table_name))
-</code></pre>
-</div>
-
-<p>Finally, write to the sink:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>with beam.Pipeline(options=PipelineOptions()) as p:
- kvs = p | 'CreateKVs' >> beam.core.Create(KVs)
- kvs | 'WriteToSimpleKV' >> WriteToKVSink(
- simplekv, 'http://url_to_simple_kv/', final_table_name)
-</code></pre>
-</div>
-
- </div>
- </div>
- <!--
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-<footer class="footer">
- <div class="footer__contained">
- <div class="footer__cols">
- <div class="footer__cols__col">
- <div class="footer__cols__col__logo">
- <img src="/images/beam_logo_circle.svg" class="footer__logo" alt="Beam logo">
- </div>
- <div class="footer__cols__col__logo">
- <img src="/images/apache_logo_circle.svg" class="footer__logo" alt="Apache logo">
- </div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Start</div>
- <div class="footer__cols__col__link"><a href="/get-started/beam-overview/">Overview</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-java/">Quickstart (Java)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-py/">Quickstart (Python)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/quickstart-go/">Quickstart (Go)</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/downloads/">Downloads</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Docs</div>
- <div class="footer__cols__col__link"><a href="/documentation/programming-guide/">Concepts</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/pipelines/design-your-pipeline/">Pipelines</a></div>
- <div class="footer__cols__col__link"><a href="/documentation/runners/capability-matrix/">Runners</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Community</div>
- <div class="footer__cols__col__link"><a href="/contribute/">Contribute</a></div>
- <div class="footer__cols__col__link"><a href="https://projects.apache.org/committee.html?beam" target="_blank">Team<img src="/images/external-link-icon.png"
- width="14" height="14"
- alt="External link."></a></div>
- <div class="footer__cols__col__link"><a href="/contribute/presentation-materials/">Media</a></div>
- </div>
- <div class="footer__cols__col footer__cols__col--md">
- <div class="footer__cols__col__title">Resources</div>
- <div class="footer__cols__col__link"><a href="/blog/">Blog</a></div>
- <div class="footer__cols__col__link"><a href="/get-started/support/">Support</a></div>
- <div class="footer__cols__col__link"><a href="https://github.com/apache/beam">GitHub</a></div>
- </div>
- </div>
- </div>
- <div class="footer__bottom">
- ©
- <a href="http://www.apache.org">The Apache Software Foundation</a>
- | <a href="/privacy_policy">Privacy Policy</a>
- | <a href="/feed.xml">RSS Feed</a>
- <br><br>
- Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are
- either registered trademarks or trademarks of The Apache Software
- Foundation. All other products or name brands are trademarks of their
- respective holders, including The Apache Software Foundation.
- </div>
-</footer>
-
- </body>
+<html lang="en-US">
+<meta charset="utf-8">
+<title>Redirecting…</title>
+<link rel="canonical" href="https://beam.apache.org/documentation/io/developing-io-python/">
+<meta http-equiv="refresh" content="0; url=https://beam.apache.org/documentation/io/developing-io-python/">
+<h1>Redirecting…</h1>
+<a href="https://beam.apache.org/documentation/io/developing-io-python/">Click here if you are not redirected.</a>
+<script>location="https://beam.apache.org/documentation/io/developing-io-python/"</script>
</html>
diff --git a/website/generated-content/documentation/sdks/python-dependencies/index.html b/website/generated-content/documentation/sdks/python-dependencies/index.html
index fb309c5..e18248a 100644
--- a/website/generated-content/documentation/sdks/python-dependencies/index.html
+++ b/website/generated-content/documentation/sdks/python-dependencies/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
index d19454c..20a59f5 100644
--- a/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/website/generated-content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/python-streaming/index.html b/website/generated-content/documentation/sdks/python-streaming/index.html
index d2ea7d2..bd70bd3 100644
--- a/website/generated-content/documentation/sdks/python-streaming/index.html
+++ b/website/generated-content/documentation/sdks/python-streaming/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/python-type-safety/index.html b/website/generated-content/documentation/sdks/python-type-safety/index.html
index f50d386..774ce3c 100644
--- a/website/generated-content/documentation/sdks/python-type-safety/index.html
+++ b/website/generated-content/documentation/sdks/python-type-safety/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
diff --git a/website/generated-content/documentation/sdks/python/index.html b/website/generated-content/documentation/sdks/python/index.html
index da7ac06..3b5e68b 100644
--- a/website/generated-content/documentation/sdks/python/index.html
+++ b/website/generated-content/documentation/sdks/python/index.html
@@ -179,7 +179,6 @@
<li><a href="/documentation/sdks/python-streaming/">Python streaming pipelines</a></li>
<li><a href="/documentation/sdks/python-type-safety/">Ensuring Python type safety</a></li>
<li><a href="/documentation/sdks/python-pipeline-dependencies/">Managing pipeline dependencies</a></li>
- <li><a href="/documentation/sdks/python-custom-io/">Creating new sources and sinks</a></li>
</ul>
</li>
@@ -243,7 +242,7 @@
<li><a href="#python-streaming-pipelines">Python streaming pipelines</a></li>
<li><a href="#python-type-safety">Python type safety</a></li>
<li><a href="#managing-python-pipeline-dependencies">Managing Python pipeline dependencies</a></li>
- <li><a href="#creating-new-sources-and-sinks">Creating new sources and Sinks</a></li>
+ <li><a href="#developing-new-io-connectors-for-python">Developing new I/O connectors for Python</a></li>
</ul>
@@ -287,9 +286,13 @@ starting with Beam SDK version 2.5.0.</p>
<p>When you run your pipeline locally, the packages that your pipeline depends on are available because they are installed on your local machine. However, when you want to run your pipeline remotely, you must make sure these dependencies are available on the remote machines. <a href="/documentation/sdks/python-pipeline-dependencies">Managing Python Pipeline Dependencies</a> shows you how to make your dependencies available to the remote workers.</p>
-<h2 id="creating-new-sources-and-sinks">Creating new sources and Sinks</h2>
+<h2 id="developing-new-io-connectors-for-python">Developing new I/O connectors for Python</h2>
+
+<p>The Beam SDK for Python provides an extensible API that you can use to create
+new I/O connectors. See the <a href="/documentation/io/developing-io-overview">Developing I/O connectors overview</a>
+for information about developing new I/O connectors and links to
+language-specific implementation guidance.</p>
-<p>The Beam SDK for Python provides an extensible API that you can use to create new data sources and sinks. <a href="/documentation/sdks/python-custom-io">Creating New Sources and Sinks with the Python SDK</a> shows how to create new sources and sinks using <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py">Beam’s Source and Sink API</a>.</p>
</div>
</div>
diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml
index c6211f3..d6b9b75 100644
--- a/website/generated-content/feed.xml
+++ b/website/generated-content/feed.xml
@@ -566,7 +566,7 @@ and <a href="https://spark.apache.org/docs/latest/api/java/org/apache/sp
</td>
<td>Custom receivers
</td>
- <td><a href="/documentation/io/authoring-overview/#read-transforms">Read Transforms</a>
+ <td><a href="/documentation/io/developing-io-overview/">Read Transforms</a>
</td>
<td><a href="https://spark.apache.org/docs/latest/streaming-custom-receivers.html">receiverStream</a>
</td>