You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/10/20 20:02:08 UTC

[GitHub] [beam] melap opened a new pull request #15763: [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

melap opened a new pull request #15763:
URL: https://github.com/apache/beam/pull/15763


   Update the Beam basics documentation page to add more info for Aggregation, Runner, UDF, and Schema, and add links to more information about them. This is part of the effort to create an introductory concepts guide.
   
   R: @ibzib 
   
   ------------------------
   
   `ValidatesRunner` compliance status (on master branch)
   --------------------------------------------------------
   
   <table>
     <thead>
       <tr>
         <th>Lang</th>
         <th>ULR</th>
         <th>Dataflow</th>
         <th>Flink</th>
         <th>Samza</th>
         <th>Spark</th>
         <th>Twister2</th>
       </tr>
     </thead>
     <tbody>
       <tr>
         <td>Go</td>
         <td>---</td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
           </a>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Samza/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>---</td>
       </tr>
       <tr>
         <td>Java</td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon?subject=V1+Streaming">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon?subject=V1+Java+11">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon?subject=V2+Streaming">
           </a><br>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon?subject=Java+8">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon?subject=Java+11">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon?subject=Portable">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon?subject=Portable+Streaming">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Samza/lastCompletedBuild/badge/icon?subject=Portable">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon?subject=Portable">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon?subject=Structured+Streaming">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
           </a>
         </td>
       </tr>
       <tr>
         <td>Python</td>
         <td>---</td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon?subject=V1">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon?subject=ValCont">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon?subject=Portable">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Samza/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>---</td>
       </tr>
       <tr>
         <td>XLang</td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Samza/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>---</td>
       </tr>
     </tbody>
   </table>
   
   Examples testing status on various runners
   --------------------------------------------------------
   
   <table>
     <thead>
       <tr>
         <th>Lang</th>
         <th>ULR</th>
         <th>Dataflow</th>
         <th>Flink</th>
         <th>Samza</th>
         <th>Spark</th>
         <th>Twister2</th>
       </tr>
     </thead>
     <tbody>
       <tr>
         <td>Go</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
       </tr>
       <tr>
         <td>Java</td>
         <td>---</td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon?subject=V1">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon?subject=V1+Java11">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon?subject=V2">
           </a><br>
         </td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
       </tr>
       <tr>
         <td>Python</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
       </tr>
       <tr>
         <td>XLang</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
       </tr>
     </tbody>
   </table>
   
   Post-Commit SDK/Transform Integration Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   <table>
     <thead>
       <tr>
         <th>Go</th>
         <th>Java</th>
         <th>Python</th>
       </tr>
     </thead>
     <tbody>
       <tr>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon?subject=3.6">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon?subject=3.7">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon?subject=3.8">
           </a>
         </td>
       </tr>
     </tbody>
   </table>
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   <table>
     <thead>
       <tr>
         <th>---</th>
         <th>Java</th>
         <th>Python</th>
         <th>Go</th>
         <th>Website</th>
         <th>Whitespace</th>
         <th>Typescript</th>
       </tr>
     </thead>
     <tbody>
       <tr>
         <td>Non-portable</td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
           </a><br>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon?subject=Tests">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon?subject=Lint">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon?subject=Docker">
           </a><br>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon?subject=Docs">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
           </a>
         </td>
       </tr>
       <tr>
         <td>Portable</td>
         <td>---</td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>
           <a href="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/">
             <img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_GoPortable_Cron/lastCompletedBuild/badge/icon">
           </a>
         </td>
         <td>---</td>
         <td>---</td>
         <td>---</td>
       </tr>
     </tbody>
   </table>
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] melap commented on a change in pull request #15763: [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

Posted by GitBox <gi...@apache.org>.

melap commented on a change in pull request #15763:
URL: https://github.com/apache/beam/pull/15763#discussion_r733976340



##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.

Review comment:
       Done

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These
+transforms allow selections and aggregations in terms of named schema fields.
+Another advantage of schemas is that they allow referencing of element fields by
+name. Beam provides a selection syntax for referencing fields, including nested
+and repeated fields.
+
+For more information about schemas, see the following pages:
+
+ * [Beam Programming Guide: Schemas](/documentation/programming-guide/#schemas)
+ * [Schema Patterns](/documentation/patterns/schema/)
 
 ### Runner
 
-The term "runner" is used for a couple of things. It generally refers to the
-software that takes a Beam pipeline and executes it somehow. Often, this is the
-translation code that you write. It usually also includes some customized
-operators for your data processing engine, and is sometimes used to refer to
-the full stack.
+A Beam runner runs a Beam pipeline on a specific platform. Most runners are
+translators or adapters to massively parallel big data processing systems, such
+as Apache Flink, Apache Spark, Google Cloud Dataflow, and more. The Direct
+Runner runs pipelines locally so you can test, debug, and validate that your
+pipeline adheres to the Apache Beam model as closely as possible.
+
+If you build a runner, you write translation code and often some customed

Review comment:
       Removed building runner paragraph and updated intro.
   
   

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These

Review comment:
       Done

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy

Review comment:
       Good idea, I added a sentence about stateful processing at the end of the paragraph.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure

Review comment:
       Done

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)

Review comment:
       To verify, do you mean remove the entire bullet item, or just the end part with the Read reference?
   

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)

Review comment:
       Added links for each, some we don't currently have a doc section for (ViewFn and WindowMappingFn) so I linked to a couple side input sections that were the closest match
   

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and

Review comment:
       Done

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These
+transforms allow selections and aggregations in terms of named schema fields.
+Another advantage of schemas is that they allow referencing of element fields by
+name. Beam provides a selection syntax for referencing fields, including nested
+and repeated fields.
+
+For more information about schemas, see the following pages:
+
+ * [Beam Programming Guide: Schemas](/documentation/programming-guide/#schemas)
+ * [Schema Patterns](/documentation/patterns/schema/)
 
 ### Runner
 
-The term "runner" is used for a couple of things. It generally refers to the
-software that takes a Beam pipeline and executes it somehow. Often, this is the
-translation code that you write. It usually also includes some customized
-operators for your data processing engine, and is sometimes used to refer to
-the full stack.
+A Beam runner runs a Beam pipeline on a specific platform. Most runners are
+translators or adapters to massively parallel big data processing systems, such
+as Apache Flink, Apache Spark, Google Cloud Dataflow, and more. The Direct
+Runner runs pipelines locally so you can test, debug, and validate that your
+pipeline adheres to the Apache Beam model as closely as possible.
+
+If you build a runner, you write translation code and often some customed
+operators for your data processing engine. Sometimes "runner" is used to refer
+to the full stack.
+
+A runner has just a single method: `run`. This method should be asynchronous and

Review comment:
       Removed

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -20,24 +20,34 @@ limitations under the License.
 Suppose you have a data processing engine that can pretty easily process graphs
 of operations. You want to integrate it with the Beam ecosystem to get access
 to other languages, great event time processing, and a library of connectors.
-You need to know the core vocabulary:
+You need to know the following core vocabulary:
 
  * [_Pipeline_](#pipeline) - A pipeline is a graph of transformations that a user constructs
    that defines the data processing they want to do.
  * [_PCollection_](#pcollections) - Data being processed in a pipeline is part of a PCollection.
  * [_PTransforms_](#ptransforms) - The operations executed within a pipeline. These are best
    thought of as operations on PCollections.
- * _SDK_ - A language-specific library for pipeline authors (we often call them
-   "users" even though we have many kinds of users) to build transforms,
-   construct their pipelines and submit them to a runner
- * _Runner_ - You are going to write a piece of software called a runner that
-   takes a Beam pipeline and executes it using the capabilities of your data
-   processing engine.
-
-These concepts may be very similar to your processing engine's concepts. Since
+ * [_Aggregation_](#aggregation) - Aggregation is computing a value from
+   multiple (1 or more) input elements.
+ * [_User-defined function (UDF)_](#user-defined-function-udf) - Some Beam
+   operations allow you to run user-defined code as a way of configuring the
+   transform.
+ * [_Schema_](#schema) - A schema is a language-independent type definition for
+   a `PCollection`. The schema for a `PCollection` defines elements of that
+   `PCollection` as an ordered list of named fields.
+ * [_SDK_](/documentation/sdks/java/) - A language-specific library for pipeline
+   authors (we often call them "users" even though we have many kinds of users)

Review comment:
       Agree, done. 

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These
+transforms allow selections and aggregations in terms of named schema fields.
+Another advantage of schemas is that they allow referencing of element fields by
+name. Beam provides a selection syntax for referencing fields, including nested
+and repeated fields.
+
+For more information about schemas, see the following pages:
+
+ * [Beam Programming Guide: Schemas](/documentation/programming-guide/#schemas)
+ * [Schema Patterns](/documentation/patterns/schema/)
 
 ### Runner
 
-The term "runner" is used for a couple of things. It generally refers to the
-software that takes a Beam pipeline and executes it somehow. Often, this is the
-translation code that you write. It usually also includes some customized
-operators for your data processing engine, and is sometimes used to refer to
-the full stack.
+A Beam runner runs a Beam pipeline on a specific platform. Most runners are
+translators or adapters to massively parallel big data processing systems, such

Review comment:
       Done

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements

Review comment:
       I had one request to put it here so I did initially, but I'm open to other options. I added this section because I couldn't find anywhere that talks about what aggregation is, so it'd have to be somewhere new. Hmm, what about adding 2 things... 1) something about it in the programming guide transforms intro, like
   
   "Aggregation is an important concept to understand before learning about Beam's transforms. The Basics of the Beam model page's Aggregation section contains an introduction to aggregation. "
   
   and 2) Moving these optimizations to the Combine section at https://beam.apache.org/documentation/programming-guide/#combine  ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ibzib merged pull request #15763: [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

Posted by GitBox <gi...@apache.org>.

ibzib merged pull request #15763:
URL: https://github.com/apache/beam/pull/15763


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] melap commented on a change in pull request #15763: [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

Posted by GitBox <gi...@apache.org>.

melap commented on a change in pull request #15763:
URL: https://github.com/apache/beam/pull/15763#discussion_r734121803



##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements

Review comment:
       Ok, moved the bullets to the programming guide and added the pointer to the aggregation page




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ibzib commented on a change in pull request #15763: [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

Posted by GitBox <gi...@apache.org>.

ibzib commented on a change in pull request #15763:
URL: https://github.com/apache/beam/pull/15763#discussion_r734111716



##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)

Review comment:
       The entire bullet item.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements

Review comment:
       Sounds good. I agree that the programming guide is a better place to explain optimizations.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] melap commented on pull request #15763: [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

Posted by GitBox <gi...@apache.org>.

melap commented on pull request #15763:
URL: https://github.com/apache/beam/pull/15763#issuecomment-949087847


   Ok, addressed the remaining comments. Let me know if you spot anything else you'd like updated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ibzib commented on a change in pull request #15763: [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema

Posted by GitBox <gi...@apache.org>.

ibzib commented on a change in pull request #15763:
URL: https://github.com/apache/beam/pull/15763#discussion_r733181464



##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -20,24 +20,34 @@ limitations under the License.
 Suppose you have a data processing engine that can pretty easily process graphs
 of operations. You want to integrate it with the Beam ecosystem to get access
 to other languages, great event time processing, and a library of connectors.
-You need to know the core vocabulary:
+You need to know the following core vocabulary:
 
  * [_Pipeline_](#pipeline) - A pipeline is a graph of transformations that a user constructs
    that defines the data processing they want to do.
  * [_PCollection_](#pcollections) - Data being processed in a pipeline is part of a PCollection.
  * [_PTransforms_](#ptransforms) - The operations executed within a pipeline. These are best
    thought of as operations on PCollections.
- * _SDK_ - A language-specific library for pipeline authors (we often call them
-   "users" even though we have many kinds of users) to build transforms,
-   construct their pipelines and submit them to a runner
- * _Runner_ - You are going to write a piece of software called a runner that
-   takes a Beam pipeline and executes it using the capabilities of your data
-   processing engine.
-
-These concepts may be very similar to your processing engine's concepts. Since
+ * [_Aggregation_](#aggregation) - Aggregation is computing a value from
+   multiple (1 or more) input elements.
+ * [_User-defined function (UDF)_](#user-defined-function-udf) - Some Beam
+   operations allow you to run user-defined code as a way of configuring the
+   transform.
+ * [_Schema_](#schema) - A schema is a language-independent type definition for
+   a `PCollection`. The schema for a `PCollection` defines elements of that
+   `PCollection` as an ordered list of named fields.
+ * [_SDK_](/documentation/sdks/java/) - A language-specific library for pipeline
+   authors (we often call them "users" even though we have many kinds of users)

Review comment:
       Nit: I don't think this detail is relevant. (I realize this was in there before, but this is a good chance to edit.)
   ```suggestion
      authors
   ```

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.

Review comment:
       This is a good place to link to cross-language documentation. https://beam.apache.org/documentation/patterns/cross-language/

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)

Review comment:
       While not actually deprecated yet, `Read` is on its way out (to be replaced with splittable DoFn) so we may want to erase this item.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These
+transforms allow selections and aggregations in terms of named schema fields.
+Another advantage of schemas is that they allow referencing of element fields by
+name. Beam provides a selection syntax for referencing fields, including nested
+and repeated fields.
+
+For more information about schemas, see the following pages:
+
+ * [Beam Programming Guide: Schemas](/documentation/programming-guide/#schemas)
+ * [Schema Patterns](/documentation/patterns/schema/)
 
 ### Runner
 
-The term "runner" is used for a couple of things. It generally refers to the
-software that takes a Beam pipeline and executes it somehow. Often, this is the
-translation code that you write. It usually also includes some customized
-operators for your data processing engine, and is sometimes used to refer to
-the full stack.
+A Beam runner runs a Beam pipeline on a specific platform. Most runners are
+translators or adapters to massively parallel big data processing systems, such
+as Apache Flink, Apache Spark, Google Cloud Dataflow, and more. The Direct
+Runner runs pipelines locally so you can test, debug, and validate that your
+pipeline adheres to the Apache Beam model as closely as possible.
+
+If you build a runner, you write translation code and often some customed

Review comment:
       "If you build a runner" is not something we need to be concerned with on this page; I would assume anyone who is seriously considering building a runner is already quite familiar with Beam basics.
   
   Side note: it seems like this page was originally intended to target runner authors. For example the top of the page says:
   
   ```
   Suppose you have a data processing engine that can pretty easily process graphs of operations. You want to integrate it with the Beam ecosystem to get access to other languages, great event time processing, and a library of connectors.
   ```
   
   We should remove such references so it's clear this page is targeting beginners.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure

Review comment:
       Let's move this sentence ("In many cases...") to the next paragraph.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy

Review comment:
       We should clarify that Beam does have a stateful processing API. https://beam.apache.org/blog/stateful-processing/

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)

Review comment:
       We should insert inline links to the programming guide for each of the items in the UDF list.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements

Review comment:
       These optimizations go beyond just the "basics." It may be better to move the advanced details to a dedicated page for aggregations (creating a new page if one doesn't currently exist).

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These
+transforms allow selections and aggregations in terms of named schema fields.
+Another advantage of schemas is that they allow referencing of element fields by
+name. Beam provides a selection syntax for referencing fields, including nested
+and repeated fields.
+
+For more information about schemas, see the following pages:
+
+ * [Beam Programming Guide: Schemas](/documentation/programming-guide/#schemas)
+ * [Schema Patterns](/documentation/patterns/schema/)
 
 ### Runner
 
-The term "runner" is used for a couple of things. It generally refers to the
-software that takes a Beam pipeline and executes it somehow. Often, this is the
-translation code that you write. It usually also includes some customized
-operators for your data processing engine, and is sometimes used to refer to
-the full stack.
+A Beam runner runs a Beam pipeline on a specific platform. Most runners are
+translators or adapters to massively parallel big data processing systems, such
+as Apache Flink, Apache Spark, Google Cloud Dataflow, and more. The Direct
+Runner runs pipelines locally so you can test, debug, and validate that your
+pipeline adheres to the Apache Beam model as closely as possible.
+
+If you build a runner, you write translation code and often some customed
+operators for your data processing engine. Sometimes "runner" is used to refer
+to the full stack.
+
+A runner has just a single method: `run`. This method should be asynchronous and

Review comment:
       The details in this paragraph are specific to the Java SDK, so I'm not sure they're relevant to this page either.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These
+transforms allow selections and aggregations in terms of named schema fields.
+Another advantage of schemas is that they allow referencing of element fields by
+name. Beam provides a selection syntax for referencing fields, including nested
+and repeated fields.
+
+For more information about schemas, see the following pages:
+
+ * [Beam Programming Guide: Schemas](/documentation/programming-guide/#schemas)
+ * [Schema Patterns](/documentation/patterns/schema/)
 
 ### Runner
 
-The term "runner" is used for a couple of things. It generally refers to the
-software that takes a Beam pipeline and executes it somehow. Often, this is the
-translation code that you write. It usually also includes some customized
-operators for your data processing engine, and is sometimes used to refer to
-the full stack.
+A Beam runner runs a Beam pipeline on a specific platform. Most runners are
+translators or adapters to massively parallel big data processing systems, such

Review comment:
       Let's provide more detail about what runners translate (for example, the Flink runner translates a Beam pipeline into a Flink job).

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and

Review comment:
       We should state explicitly that all these formats can be converted to Beam Schemas.

##########
File path: website/www/site/content/en/documentation/basics.md
##########
@@ -137,45 +147,158 @@ windowing strategy, and
 [GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
 which has behavior governed by the windowing strategy.
 
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
-   and GroupByKey)
+### Aggregation
+
+Aggregation is computing a value from multiple (1 or more) input elements. In
+Beam, the primary computational pattern for aggregation is to group all elements
+with a common key and window then combine each group of elements using an
+associative and commutative operation. This is similar to the "Reduce" operation
+in the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) model, though it is
+enhanced to work with unbounded input streams as well as bounded data sets.
+
+<img src="/images/aggregation.png" alt="Aggregation of elements." width="120px">
+
+*Figure 1: Aggregation of elements. Elements with the same color represent those
+with a common key and window.*
+
+Some simple aggregation transforms include `Count` (computes the count of all
+elements in the aggregation), `Max` (computes the maximum element in the
+aggregation), and `Sum` (computes the sum of all elements in the aggregation).
+
+When elements are grouped and emitted as a bag, the aggregation is known as
+`GroupByKey` (the associative/commutative operation is bag union). In this case,
+the output is no smaller than the input. Often, you will apply an operation such
+as summation, called a `CombineFn`, in which the output is significantly smaller
+than the input. In this case the aggregation is called `CombinePerKey`.
+
+The associativity and commutativity of a `CombineFn` allows runners to
+automatically apply some optimizations:
+
+ * _Combiner lifting_: This is the most significant optimization. Input elements
+   are combined per key and window before they are shuffled, so the volume of
+   data shuffled might be reduced by many orders of magnitude. Another term for
+   this optimization is "mapper-side combine."
+ * _Incremental combining_: When you have a `CombineFn` that reduces the data
+   size by a lot, it is useful to combine elements as they emerge from a
+   streaming shuffle. This spreads out the cost of doing combines over the time
+   that your streaming computation might be idle. Incremental combining also
+   reduces the storage of intermediate accumulators.
+
+In a real application, you might have millions of keys and/or windows; that is
+why this is still an "embarassingly parallel" computational pattern. In those
+cases where you have fewer keys, you can add parallelism by adding a
+supplementary key, splitting each of your problem's natural keys into many
+sub-keys. After these sub-keys are aggregated, the results can be further
+combined into a result for the original natural key for your problem. The
+associativity of your aggregation function ensures that this yields the same
+answer, but with more parallelism.
+
+When your input is unbounded, the computational pattern of grouping elements by
+key and window is roughly the same, but governing when and how to emit the
+results of aggregation involves three concepts:
+
+ * Windowing, which partitions your input into bounded subsets that can be
+   complete.
+ * Watermarks, which estimate the completeness of your input.
+ * Triggers, which govern when and how to emit aggregated results.
+
+For more information about available aggregation transforms, see the following
+pages:
+
+ * [Beam Programming Guide: Core Beam transforms](/documentation/programming-guide/#core-beam-transforms)
+ * Beam Transform catalog
+   ([Java](/documentation/transforms/java/overview/#aggregation),
+   [Python](/documentation/transforms/python/overview/#aggregation))
+
+### User-defined function (UDF)
+
+Some Beam operations allow you to run user-defined code as a way of configuring
+the transform. For example, when using `ParDo`, user-defined code specifies what
+operation to apply to every element. For `Combine`, it specifies how values
+should be combined. A Beam pipeline can contain UDFs written in a language other
+than your runner, or even multiple languages in the same pipeline so the
+definitions are language-independent.
+
+Beam has seven varieties of UDFs:
+
+ * _DoFn_ - per-element processing function (used in `ParDo`)
+ * _WindowFn_ - places elements in windows and merges windows (used in `Window`
+   and `GroupByKey`)
  * _Source_ - emits data read from external sources, including initial and
-   dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
-   in side inputs)
+   dynamic splitting for parallelism (used in `Read`)
+ * _ViewFn_ - adapts a materialized `PCollection` to a particular interface
+   (used in side inputs)
  * _WindowMappingFn_ - maps one element's window to another, and specifies
    bounds on how far in the past the result window will be (used in side
    inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ * _CombineFn_ - associative and commutative aggregation (used in `Combine` and
    state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the [_PTransforms_](#ptransforms) that use them.
+ * _Coder_ - encodes user data; some coders have standard formats and are not
+   really UDFs
+
+Each language SDK has its own idiomatic way of expressing the user-defined
+functions in Beam, but there are common requirements. When you build user code
+for a Beam transform, you should keep in mind the distributed nature of
+execution. For example, there might be many copies of your function running on a
+lot of different machines in parallel, and those copies function independently,
+without communicating or sharing state with any of the other copies. Each copy
+of your user code function might be retried or run multiple times, depending on
+the pipeline runner and the processing backend that you choose for your
+pipeline.
+
+For more information about user-defined functions, see the following pages:
+
+ * [Requirements for writing user code for Beam transforms](/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms)
+ * [Beam Programming Guide: ParDo](/documentation/programming-guide/#pardo)
+
+### Schema
+
+A schema is a language-independent type definition for a `PCollection`. The
+schema for a `PCollection` defines elements of that `PCollection` as an ordered
+list of named fields. Each field has a name, a type, and possibly a set of user
+options.  In many cases, the element type in a `PCollection` has a structure
+that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and
+database row objects.
+
+Even within a SDK pipeline, Simple Java POJOs (or equivalent structures in other
+languages) are often used as intermediate types, and these also have a clear
+structure that can be inferred by inspecting the class. By understanding the
+structure of a pipeline’s records, we can provide much more concise APIs for
+data processing.
+
+Beam provides a collection of transforms that operate natively on schemas. These

Review comment:
       It's worth mentioning Beam SQL as a common transform that operates on schemas.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org