You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by da...@apache.org on 2024/04/08 20:14:16 UTC
(beam) branch master updated: Duet AI Prompts - Documentation Lookup Without Links (#30873)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1c962a5ec7a Duet AI Prompts - Documentation Lookup Without Links (#30873)
1c962a5ec7a is described below
commit 1c962a5ec7ac80eb1165a7fe8fc9c362b94dcd0f
Author: Daria Bezkorovaina <99...@users.noreply.github.com>
AuthorDate: Mon Apr 8 21:14:09 2024 +0100
Duet AI Prompts - Documentation Lookup Without Links (#30873)
* Create 53_dataflow_runner.md
* Create 54_spark_runner.md
* Create 55_direct_runner.md
* Create 29_advanced_side_inputs.md
* Create 30_advanced_additional_outputs.md
* Create 31_advanced_batched_dofn.md
* Update 31_advanced_batched_dofn.md
* Update 30_advanced_additional_outputs.md
* Create 01_basic_learning_apache_beam.md
* Create 02_basic_pipelines.md
* Create 03_basic_configuring_pipelines.md
* Update 04_basic_pcollections.md
* Create 04_basic_pcollections.md
* Update 05_basic_ptransforms.md
* Create 05_basic_ptransforms.md
* Update 06_basic_schema.md
* Create 06_basic_schema.md
* Update 07_basic_runners.md
* Update 07_basic_runners.md
* Create 07_basic_runners.md
* Update 07_basic_runners.md
* Update 08_basic_windowing.md
* Create 08_basic_windowing.md
* Update 09_basic_triggers.md
* Create 09_basic_triggers.md
* Update 10_basic_metrics.md
* Update 10_basic_metrics.md
* Create 10_basic_metrics.md
* Update 11_basic_state.md
* Create 11_basic_state.md
* Update 12_basic_timers.md
* Update 12_basic_timers.md
* Create 12_basic_timers.md
* Update 13_advanced_splittable_dofn.md
* Update 13_advanced_splittable_dofn.md
* Create 13_advanced_splittable_dofn.md
* Update 13_advanced_splittable_dofn.md
* Update 14_advanced_pipeline_patterns.md
* Update 14_advanced_pipeline_patterns.md
* Create 14_advanced_pipeline_patterns.md
* Update 15_advanced_xlang.md
* Create 15_advanced_xlang.md
* Update 16_advanced_pipeline_lifecycle.md
* Update 16_advanced_pipeline_lifecycle.md
* Create 16_advanced_pipeline_lifecycle.md
* Update 17_advanced_ai_ml.md
* Update 17_advanced_ai_ml.md
* Update 17_advanced_ai_ml.md
* Create 17_advanced_ai_ml.md
* Update 18_io_kafka.md
* Create 18_io_kafka.md
* Update 19_io_pubsub.md
* Create 19_io_pubsub.md
* Update 20_io_biguery.md
* Create 20_io_biguery.md
* Update 21_io_bigtable.md
* Create 21_io_bigtable.md
* Update 22_io_spanner.md
* Create 22_io_spanner.md
* Update 23_io_gcs.md
* Create 23_io_gcs.md
* Update 24_io_json.md
* Create 24_io_json.md
* Update 25_io_csv.md
* Create 25_io_csv.md
* Update 26_io_avro.md
* Create 26_io_avro.md
* Update 27_io_parquet.md
* Create 27_io_parquet.md
* Update 28_io_jdbc.md
* Create 28_io_jdbc.md
* Update learning/prompts/documentation-lookup-nolinks/08_basic_windowing.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/09_basic_triggers.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/18_io_kafka.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/19_io_pubsub.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/20_io_biguery.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/20_io_biguery.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/21_io_bigtable.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/22_io_spanner.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/24_io_json.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/26_io_avro.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/27_io_parquet.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/25_io_csv.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/28_io_jdbc.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/30_advanced_additional_outputs.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/29_advanced_side_inputs.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/30_advanced_additional_outputs.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/30_advanced_additional_outputs.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/30_advanced_additional_outputs.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/53_dataflow_runner.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update learning/prompts/documentation-lookup-nolinks/54_spark_runner.md
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
* Update 53_dataflow_runner.md
* Update 07_basic_runners.md
* Update 53_dataflow_runner.md
---------
Co-authored-by: Andrey Devyatkin <an...@gmail.com>
---
.../01_basic_learning_apache_beam.md | 16 +++
.../02_basic_pipelines.md | 18 +++
.../03_basic_configuring_pipelines.md | 31 +++++
.../04_basic_pcollections.md | 8 +-
.../05_basic_ptransforms.md | 34 +++++
.../06_basic_schema.md | 25 ++++
.../07_basic_runners.md | 25 ++++
.../08_basic_windowing.md | 31 +++++
.../09_basic_triggers.md | 31 +++++
.../10_basic_metrics.md | 13 +-
.../documentation-lookup-nolinks/11_basic_state.md | 19 +++
.../12_basic_timers.md | 11 ++
.../13_advanced_splittable_dofn.md | 14 +-
.../14_advanced_pipeline_patterns.md | 14 ++
.../15_advanced_xlang.md | 11 ++
.../16_advanced_pipeline_lifecycle.md | 13 +-
.../17_advanced_ai_ml.md | 18 +++
.../documentation-lookup-nolinks/18_io_kafka.md | 29 ++++
.../documentation-lookup-nolinks/19_io_pubsub.md | 23 ++++
.../documentation-lookup-nolinks/20_io_biguery.md | 42 ++++++
.../documentation-lookup-nolinks/21_io_bigtable.md | 28 ++++
.../documentation-lookup-nolinks/22_io_spanner.md | 33 +++++
.../documentation-lookup-nolinks/23_io_gcs.md | 15 +++
.../documentation-lookup-nolinks/24_io_json.md | 29 ++++
.../documentation-lookup-nolinks/25_io_csv.md | 30 +++++
.../documentation-lookup-nolinks/26_io_avro.md | 25 ++++
.../documentation-lookup-nolinks/27_io_parquet.md | 21 +++
.../documentation-lookup-nolinks/28_io_jdbc.md | 42 ++++++
.../29_advanced_side_inputs.md | 36 +++++
.../30_advanced_additional_outputs.md | 100 ++++++++++++++
.../31_advanced_batched_dofn.md | 39 ++++++
.../53_dataflow_runner.md | 80 +++++++++++
.../54_spark_runner.md | 148 +++++++++++++++++++++
.../55_direct_runner.md | 68 ++++++++++
.../documentation-lookup/04_basic_pcollections.md | 6 +-
.../documentation-lookup/05_basic_ptransforms.md | 4 +-
.../documentation-lookup/06_basic_schema.md | 8 +-
.../documentation-lookup/07_basic_runners.md | 10 +-
.../documentation-lookup/08_basic_windowing.md | 14 +-
.../documentation-lookup/09_basic_triggers.md | 14 +-
.../documentation-lookup/10_basic_metrics.md | 9 +-
.../prompts/documentation-lookup/11_basic_state.md | 23 ++--
.../documentation-lookup/12_basic_timers.md | 7 +-
.../13_advanced_splittable_dofn.md | 12 +-
.../14_advanced_pipeline_patterns.md | 5 +-
.../documentation-lookup/15_advanced_xlang.md | 10 +-
.../16_advanced_pipeline_lifecycle.md | 9 +-
.../documentation-lookup/17_advanced_ai_ml.md | 5 +-
.../prompts/documentation-lookup/18_io_kafka.md | 11 +-
.../prompts/documentation-lookup/19_io_pubsub.md | 13 +-
.../prompts/documentation-lookup/20_io_biguery.md | 18 +--
.../prompts/documentation-lookup/21_io_bigtable.md | 9 +-
.../prompts/documentation-lookup/22_io_spanner.md | 6 +-
learning/prompts/documentation-lookup/23_io_gcs.md | 15 +--
.../prompts/documentation-lookup/24_io_json.md | 8 +-
learning/prompts/documentation-lookup/25_io_csv.md | 7 +-
.../prompts/documentation-lookup/26_io_avro.md | 8 +-
.../prompts/documentation-lookup/27_io_parquet.md | 6 +-
.../prompts/documentation-lookup/28_io_jdbc.md | 9 +-
.../30_advanced_additional_outputs.md | 12 +-
.../31_advanced_batched_dofn.md | 27 ++--
61 files changed, 1237 insertions(+), 168 deletions(-)
diff --git a/learning/prompts/documentation-lookup-nolinks/01_basic_learning_apache_beam.md b/learning/prompts/documentation-lookup-nolinks/01_basic_learning_apache_beam.md
new file mode 100644
index 00000000000..e28d44b1dfd
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/01_basic_learning_apache_beam.md
@@ -0,0 +1,16 @@
+Prompt:
+What is Apache Beam?
+
+Response:
+Apache Beam is an open-source unified programming model for data processing pipelines. It provides a set of language-specific SDKs that allow you to define and execute data processing workflows.
+Apache Beam allows you to write pipelines in one of the supported languages (Java, Python, Go, and Scala). The pipelines can be executed on one of the supported runners (Apache Flink, Apache Spark, and Google Cloud Dataflow).
+
+Learning Apache Beam is a valuable skill for data processing and analysis. Here are some resources to get you started:
+* The 'Get Started' section of the Apache Beam website is a perfect place to start. It provides an overview of Apache Beam and its key concepts.
+* For a hands-on introduction, you can start by exploring the WordCount example provided in the 'Get Started' section, which demonstrates the basic concepts of Apache Beam.
+* Tour of Beam is an interactive tour that teaches you core Beam concepts. It provides a sandbox environment to write and run pipelines while exploring different topics.
+* Beam Playground offers an interactive space to experiment with the Apache Beam transforms and examples without installing anything on your system.
+
+After you have a basic understanding of Apache Beam, you can start to build your own pipelines. The Apache Beam website has a number of resources that can help you get started, including Apache Beam Documentation and Apache Beam Community. For those interested in the source code, you can explore the Apache Beam repository on GitHub.
+
+By leveraging these resources, you can build a strong foundation in Apache Beam and start constructing your own data processing pipelines. Additionally, Apache Beam is an open-source project, and contributions are welcome. Whether you're fixing bugs, improving documentation, or adding new features, your work can benefit the entire Beam community. To learn more about how you can contribute, visit the 'Contribute' section of the Apache Beam website.
diff --git a/learning/prompts/documentation-lookup-nolinks/02_basic_pipelines.md b/learning/prompts/documentation-lookup-nolinks/02_basic_pipelines.md
new file mode 100644
index 00000000000..29620aad6fd
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/02_basic_pipelines.md
@@ -0,0 +1,18 @@
+Prompt:
+What is a pipeline in Apache Beam?
+
+Response:
+In Apache Beam, a pipeline encapsulates the entirety of a data processing task, including all the data and each step of the process. Essentially, it's a Directed Acyclic Graph of transformations (known as `PTransform`) applied to data collections (`PCollection`).
+
+The simplest pipelines in Apache Beam follow a linear flow of operations, typically adhering to a read-process-write pattern. However, pipelines can also be significantly more complex, featuring multiple input sources, output sinks, and operations (known as a `PTransform`) that can both read from and output to multiple `PCollection` objects. For more information about pipeline design and best practices, see the 'Common Pipeline Patterns' section in the Apache Beam documentation.
+
+To use Apache Beam, your driver program must first create an instance of the Beam SDK class `Pipeline` (typically in the `main()` function).
+
+```python
+import apache_beam as beam
+
+with beam.Pipeline() as pipeline:
+ pass # Build your pipeline here.
+```
+
+When you create your pipeline, you need to set configuration options. You can set your pipeline’s configuration options programmatically. You can also set the options ahead of time, or read them from the command line, and then pass them to the `Pipeline` object when you create the object. This option is often easier than setting the pipeline configuration options programmatically.
diff --git a/learning/prompts/documentation-lookup-nolinks/03_basic_configuring_pipelines.md b/learning/prompts/documentation-lookup-nolinks/03_basic_configuring_pipelines.md
new file mode 100644
index 00000000000..664b14df87d
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/03_basic_configuring_pipelines.md
@@ -0,0 +1,31 @@
+Prompt:
+How do I configure pipeline in Apache Beam?
+
+Response:
+Configuring pipeline options in Apache Beam is crucial for specifying the execution environment, managing resources, and tailoring the pipeline's behavior to meet specific data processing needs and performance requirements.
+
+You can set pipeline options programmatically or pass them in using the command line. These options include runner-specific settings, job name, project ID (for cloud runners), machine types, number of workers, and more.
+
+Apache Beam offers a variety of standard pipeline options that allow you to customize and optimize your data processing pipelines.
+
+Beam SDKs include a command-line parser that you can use to set pipeline options. Use command-line arguments in the format `--<option>=<value>`. For example, the following command sets the `--runner` option `DirectRunner` and the `--project` option `my-project-id`:
+
+```bash
+python my-pipeline.py --runner=DirectRunner --project=my-project-id
+```
+
+To set the pipeline options programmatically, use the `PipelineOptions` class. For example, the following code sets the `--runner` option to `DirectRunner` and the `--project` option to `my-project-id`:
+
+```python
+from apache_beam import Pipeline
+from apache_beam.options.pipeline_options import PipelineOptions
+
+options = PipelineOptions(
+ project='my-project-id',
+ runner='DirectRunner'
+)
+```
+
+In addition to the standard pipeline options, you can add custom pipeline options. For a common pattern for configuring pipeline options, see the 'Pipeline option patterns' section in the Apache Beam documentation.
+
+The WordCount example pipeline in the 'Get Started' section of the Apache Beam documentation demonstrates how to set pipeline options at runtime by using command-line options.
diff --git a/learning/prompts/documentation-lookup/04_basic_pcollections.md b/learning/prompts/documentation-lookup-nolinks/04_basic_pcollections.md
similarity index 70%
copy from learning/prompts/documentation-lookup/04_basic_pcollections.md
copy to learning/prompts/documentation-lookup-nolinks/04_basic_pcollections.md
index 5fefa5873f5..9aa1e49968c 100644
--- a/learning/prompts/documentation-lookup/04_basic_pcollections.md
+++ b/learning/prompts/documentation-lookup-nolinks/04_basic_pcollections.md
@@ -1,7 +1,7 @@
Prompt:
-What is a PCollection in Apache Beam?
-Response:
+What is a `PCollection` in Apache Beam?
+Response:
A `PCollection` in Apache Beam is an unordered bag of elements. Each `PCollection` is a potentially distributed, homogeneous data set or data stream, and is owned by the specific `Pipeline` object for which it is created. It’s the primary data structure used in Apache Beam pipelines to handle large-scale data processing in batch and streaming modes.
The following example shows how to create a `PCollection` using the `Create` transform:
@@ -13,7 +13,7 @@ with beam.Pipeline() as pipeline:
pcollection = pipeline | beam.Create([...]) # Create a PCollection
```
-A `PCollection` can either be bounded or unbounded, making it versatile for different [data source types](https://beam.apache.org/documentation/basics/#pcollection). Bounded `PCollections` represent a finite data set, such as files or databases, ideal for batch processing. Unbounded `PCollections`, on the other hand, represent data streams that continuously grow over time, such as real-time event logs, suitable for stream processing.
+A `PCollection` can either be bounded or unbounded, making it versatile for different data source types. Bounded `PCollection` objects represent a finite data set, such as files or databases, ideal for batch processing. Unbounded `PCollection` objects, on the other hand, represent data streams that continuously grow over time, such as real-time event logs, suitable for stream processing.
Beam’s computational patterns and transforms are focused on situations where distributed data-parallel computation is required. Therefore, a `PCollection` has the following key characteristics:
- All elements must be of the same type (with support of structured types).
@@ -22,4 +22,4 @@ Beam’s computational patterns and transforms are focused on situations where d
- Random access to individual elements of the collection isn't allowed.
- Beam encodes each element for distributed processing.
-For more information about `PCollection` objects in Apache Beam and usage patterns, see [Beam Programming Guide: PCollections](https://beam.apache.org/documentation/programming-guide/#pcollections).
+For more information about `PCollection` objects in Apache Beam and usage patterns, see the 'Beam Programming Guide: PCollections' section in the Apache Beam documentation.
diff --git a/learning/prompts/documentation-lookup-nolinks/05_basic_ptransforms.md b/learning/prompts/documentation-lookup-nolinks/05_basic_ptransforms.md
new file mode 100644
index 00000000000..7a64e6ed626
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/05_basic_ptransforms.md
@@ -0,0 +1,34 @@
+Prompt:
+What is a `PTransform` in Apache Beam?
+
+Response:
+A `PTransform` (or transform) represents a data processing operation, or a step, in an Apache Beam pipeline. A transform is applied to zero or more `PCollection` objects and produces zero or more `PCollection` objects.
+
+Transforms have the following key characteristics:
+1. Versatility: Able to execute a diverse range of operations on `PCollection` objects.
+2. Composability: Can be combined to form elaborate data processing pipelines.
+3. Parallel execution: Designed for distributed processing, allowing simultaneous execution across multiple workers.
+4. Scalability: Able to handle extensive data and suitable for both batch and streaming data.
+
+The Beam SDKs contain different transforms that you can apply to your pipeline’s `PCollection` objects. The following list includes common transform types:
+ - Source transforms such as `TextIO.Read` and `Create`. A source transform conceptually has no input.
+ - Processing and conversion operations such as `ParDo`, `GroupByKey`, `CoGroupByKey`, `Combine`, and `Count`.
+ - Outputting transforms such as `TextIO.Write`.
+ - User-defined, application-specific composite transforms.
+
+Transform processing logic is provided in the form of a function object, colloquially referred to as 'user code'. This code is applied to each element of the input `PCollection` (or more than one `PCollection`). The `PCollection` objects can be linked together to create complex data processing sequences.
+User code for transforms must satisfy the requirements of the Beam model.
+
+The following example shows how to apply custom user code to a `PCollection` using the `ParDo` transform:
+
+```python
+import apache_beam as beam
+
+def SomeUserCode(element):
+ # Do something with an element
+ return element
+
+with beam.Pipeline() as pipeline:
+ input_collection = pipeline | beam.Create([...])
+ output_collection = input_collection | beam.ParDo(SomeUserCode())
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/06_basic_schema.md b/learning/prompts/documentation-lookup-nolinks/06_basic_schema.md
new file mode 100644
index 00000000000..0f0c7921b0f
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/06_basic_schema.md
@@ -0,0 +1,25 @@
+Prompt:
+What are schemas in Apache Beam?
+
+Response:
+A Schema in Apache Beam is a language-independent type definition for a `PCollection`. The schema defines elements of that `PCollection` as an ordered list of named fields.
+
+In many cases, the element type in a `PCollection` has a structure that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and database row objects. All of these formats can be converted to Beam Schemas.
+
+To take advantage of schemas, your `PCollection` objects must have a schema attached. Often, the source itself attaches a schema to the `PCollection`.
+
+Although schemas are language-independent, they're designed to embed naturally into the Beam SDK programming languages. This feature allows you to continue using native types while leveraging the advantage of having Beam understand your element schemas.
+
+Apache Beam provides a collection of schema transforms that operate on schemas. For an example, see `SqlTransform` in the 'Beam SQL overview' section of the Apache Beam documentation.
+
+The following list includes key capabilities provided by schema transforms:
+* field selection
+* grouping and aggregation
+* join operations
+* filtering data
+* adding and removing fields
+* renaming fields
+* type conversion
+* enhanced ParDo functionality
+
+For more information about schemas and the Apache Beam best practices when working with structured data, see the 'Schema Usage Patterns' section in the Apache Beam documentation.
diff --git a/learning/prompts/documentation-lookup-nolinks/07_basic_runners.md b/learning/prompts/documentation-lookup-nolinks/07_basic_runners.md
new file mode 100644
index 00000000000..8ab719f0017
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/07_basic_runners.md
@@ -0,0 +1,25 @@
+Prompt:
+What is a runner in Apache Beam?
+
+Response:
+Apache Beam runners are the execution engines that run the pipelines. They translate or adapt the pipeline into a form that can be executed on a massively parallel big data processing system, such as Apache Flink, Apache Spark, Google Cloud Dataflow, and more.
+
+Choosing a runner is an important step in the pipeline development process. The runner you choose determines where and how your pipeline runs. For more information about available runners and their capabilities, see the Beam Capability Matrix.
+
+To specify a runner, when you execute the pipeline, use the `--runner` flag. For example, to run the WordCount pipeline on Google Cloud Dataflow, you would run the following command:
+
+```python
+python -m apache_beam.examples.wordcount \
+ --region DATAFLOW_REGION \
+ --input gs://dataflow-samples/shakespeare/kinglear.txt \
+ --output gs://STORAGE_BUCKET/results/outputs \
+ --runner DataflowRunner \
+ --project PROJECT_ID \
+ --temp_location gs://STORAGE_BUCKET/tmp/
+```
+
+For more details about the WordCount example, see the WordCount source code in the Apache Beam GitHub repository.
+
+The Apache Beam Direct Runner executes the pipeline on your local machine. It is helpful for testing and debugging purposes.
+
+For instructions about how to configure your development environment and runner, see the Apache Beam quickstarts for Java, Python, and Go.
diff --git a/learning/prompts/documentation-lookup-nolinks/08_basic_windowing.md b/learning/prompts/documentation-lookup-nolinks/08_basic_windowing.md
new file mode 100644
index 00000000000..7702d278c9f
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/08_basic_windowing.md
@@ -0,0 +1,31 @@
+Prompt:
+What is windowing in Apache Beam?
+
+Response:
+In stream processing, windowing allows you to divide data streams into logical units for efficient and correct parallel processing.
+
+With an unbounded data set, collecting all of the elements is impossible, because new elements are constantly being added. In the Apache Beam model, any `PCollection` (including unbounded `PCollection` objects) can be subdivided into logical windows. Grouping transforms then consider the elements of each `PCollection` on a per-window basis.
+
+Because Apache Beam's default windowing strategy is to assign each element to a single, global window, you must explicitly specify a windowing function for your pipeline.
+
+The following code snippet shows how to divide a `PCollection` into 60-second windows:
+
+```python
+from apache_beam import beam
+from apache_beam import window
+
+fixed_windowed_items = items | 'window' >> beam.WindowInto(window.FixedWindows(60))
+```
+
+Beam provides a number of built-in windowing functions that you can use to subdivide your data into different types of windows:
+- Fixed Time Windows
+- Sliding Time Windows
+- Per-Session Windows
+- Single Global Window
+- Calendar-based Windows (not supported by the Beam SDK for Python or Go)
+
+You can also create your own custom windowing function `WindowFn`.
+
+You need to specify a triggering strategy to determine when to emit the results of your pipeline’s windowed computations.
+
+You can adjust the windowing strategy to allow for late data, which is data that arrives after the watermark has passed the end of the window. You can also specify how to handle late data, such as discarding or adding it to the next window.
diff --git a/learning/prompts/documentation-lookup-nolinks/09_basic_triggers.md b/learning/prompts/documentation-lookup-nolinks/09_basic_triggers.md
new file mode 100644
index 00000000000..ede1300aa29
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/09_basic_triggers.md
@@ -0,0 +1,31 @@
+Prompt:
+What is a trigger in Apache Beam?
+
+Response:
+Apache Beam uses triggers to determine when to emit the aggregated results of each window, referred to as a pane.
+
+Triggers provide two additional capabilities compared to outputting at the end of a window:
+- Allowing early results to be output before the end of the window.
+- Allowing late data to be handled after the end of the window.
+
+This feature lets you control the flow of your data and balance between completeness, latency, and cost.
+
+You set the triggers for a `PCollection` by setting the `trigger` parameter of the `WindowInto` transform.
+
+```python
+pcollection | WindowInto(
+ FixedWindows(1 * 60),
+ trigger=AfterProcessingTime(1 * 60),
+ accumulation_mode=AccumulationMode.DISCARDING,
+)
+```
+
+When a trigger fires, it emits the current contents of the window as a pane. Because a trigger can fire multiple times, the accumulation mode determines whether the system accumulates the window panes as the trigger fires, or discards them. This behavior is controlled by the window accumulation mode parameter of the `WindowInto` transform.
+
+Beam provides several built-in triggers that you can use to determine when to emit the results of your pipeline's windowed computations:
+- Event time triggers
+- Processing time triggers
+- Data-driven triggers
+- Composite triggers
+
+One of the most useful trigger patterns is the `AfterWatermark` trigger, which fires a single time when Apache Beam estimates that all the data has arrived, such as when the watermark passes the end of the window.
diff --git a/learning/prompts/documentation-lookup/10_basic_metrics.md b/learning/prompts/documentation-lookup-nolinks/10_basic_metrics.md
similarity index 57%
copy from learning/prompts/documentation-lookup/10_basic_metrics.md
copy to learning/prompts/documentation-lookup-nolinks/10_basic_metrics.md
index 5899819acbf..ec1972a44d3 100644
--- a/learning/prompts/documentation-lookup/10_basic_metrics.md
+++ b/learning/prompts/documentation-lookup-nolinks/10_basic_metrics.md
@@ -1,16 +1,18 @@
Prompt:
-What are Metrics in Apache Beam?
+What are the metrics in Apache Beam?
+
Response:
-[Metrics](https://beam.apache.org/documentation/programming-guide/#metrics) in Apache Beam provides a way to get insight into the current state of your pipeline, including during pipeline execution.
+In the Apache Beam model, metrics provide insights into the current state of your pipeline, including during pipeline execution.
Metrics are named and scoped to a specific step in the pipeline. They can be created dynamically during pipeline execution. If a runner doesn't support some part of reporting metrics, the fallback behavior is to drop the metric updates rather than fail the pipeline.
-Beam provides a number of [built-in metric types](https://beam.apache.org/documentation/programming-guide/#types-of-metrics):
+Apache Beam provides a number of built-in metric types:
* Counters
* Distributions
* Gauges
To declare a metric, use the `beam.metrics.Metrics` class. For example:
+
```python
self.words_counter = Metrics.counter(self.__class__, 'words')
self.word_lengths_counter = Metrics.counter(self.__class__, 'word_lengths')
@@ -18,7 +20,6 @@ self.word_lengths_dist = Metrics.distribution(self.__class__, 'word_len_dist')
self.empty_line_counter = Metrics.counter(self.__class__, 'empty_lines')
```
-For implementation details, see the [WordCount example with metrics](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_with_metrics.py).
-
+For implementation details, see the WordCount example with metrics in the Apache Beam GitHub repository.
-You can export metrics to external sinks. Spark and Flink runners support REST HTTP and Graphite.
\ No newline at end of file
+You can export metrics to external sinks. Spark and Flink runners support REST HTTP and Graphite.
diff --git a/learning/prompts/documentation-lookup-nolinks/11_basic_state.md b/learning/prompts/documentation-lookup-nolinks/11_basic_state.md
new file mode 100644
index 00000000000..9cda394159a
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/11_basic_state.md
@@ -0,0 +1,19 @@
+Prompt:
+What is a state in Apache Beam?
+
+Response:
+State in Apache Beam is an API that allows for managing per-key state in element-wise operations like `ParDo` or `Map`. It enables fine-grained control over aggregations, particularly in situations where windowing and triggers are insufficient.
+
+State is scoped to the current processing key within a window, meaning each key has its own state. State is particularly useful for complex aggregations across multiple elements.
+
+Apache Beam provides several types of state that can be used for different use cases, including scalar and collection values:
+- `ValueState`
+- `BagState`
+- `SetState`
+- `MapState`
+- `OrderedListState`
+- `CombiningState`
+
+The per-key state needs to be garbage-collected because the increasing size of the state can negatively impact performance. For more information about common garbage collection strategies, see the 'Garbage collecting state' section in the Beam Programming Guide.
+
+For more information and additional examples of using state in Apache Beam, read the community blog post 'Stateful processing with Apache Beam' and refer to the 'State and Timers examples' section in the Beam Programming Guide.
diff --git a/learning/prompts/documentation-lookup-nolinks/12_basic_timers.md b/learning/prompts/documentation-lookup-nolinks/12_basic_timers.md
new file mode 100644
index 00000000000..49872bd4ef6
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/12_basic_timers.md
@@ -0,0 +1,11 @@
+Prompt:
+What is a timer in Apache Beam?
+
+Response:
+In Apache Beam, a timer is a per-key timer callback API that enables delayed processing of data stored using the State API.
+
+Apache Beam provides two types of timers: processing time timers and event time timers. Processing time timers are based on the system clock. Event time timers are based on the timestamps of the data elements.
+
+Apache Beam also supports dynamically setting a timer tag using `TimerMap` in the Java SDK. This feature lets you set multiple different timers in a `DoFn`. You can set the timer tags to be chosen dynamically, for example, based on data in the input elements.
+
+For more information and additional examples of using timers in Apache Beam, read the community blog post 'Timely (and Stateful) Processing with Apache Beam' and see the 'State and Timers examples' section in the Apache Beam documentation.
diff --git a/learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md b/learning/prompts/documentation-lookup-nolinks/13_advanced_splittable_dofn.md
similarity index 55%
copy from learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md
copy to learning/prompts/documentation-lookup-nolinks/13_advanced_splittable_dofn.md
index 41822a931fb..9ee65bcd5be 100644
--- a/learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md
+++ b/learning/prompts/documentation-lookup-nolinks/13_advanced_splittable_dofn.md
@@ -1,18 +1,18 @@
Prompt:
-What is Splittable DoFn in Apache Beam?
+What is a splittable `DoFn` in Apache Beam?
+
Response:
-A splittable `DoFn` (SDF) is a generalization of [`DoFn`](https://beam.apache.org/documentation/programming-guide/#pardo) that lets you process elements in a non-monolithic way. A splittable `DoFn` makes it easier to create complex, modular I/O connectors in Beam .
+A splittable `DoFn` (SDF) is a generalization of `DoFn` that lets you process elements in a non-monolithic way. A splittable `DoFn` makes it easier to create complex, modular I/O connectors in Apache Beam.
When you apply a splittable `DoFn` to an element, the runner can split the element’s processing into smaller tasks. You can checkpoint the processing of an element, and you can split the remaining work to yield additional parallelism.
At a high level, an SDF is responsible for processing element and restriction pairs. A restriction represents a subset of work that would have been necessary to have been done when processing the element.
-Executing a [Splittable `DoFn`](https://beam.apache.org/documentation/programming-guide/#splittable-dofns) uses the following steps:
-1. Each element is paired with a restriction. For example, filename is paired with offset range representing the whole file).
+Executing a splittable `DoFn` uses the following steps:
+1. Each element is paired with a restriction. For example, filename is paired with offset range representing the whole file.
2. Each element and restriction pair is split. For example, offset ranges are broken up into smaller pieces.
3. The runner redistributes the element and restriction pairs to several workers.
4. Element and restriction pairs are processed in parallel. For example, the file is read. Within this last step, the element and restriction pair can pause its own processing or be split into further element and restriction pairs.
-For an example, see the [Splittable DoFn module](https://tour.beam.apache.org/tour/python/splittable-dofn/splittable) in the Tour of Beam.
-
-For more information, see the [community blog post](https://beam.apache.org/blog/splittable-do-fn-is-available/).
+For an example, see the 'Splittable `DoFn`' module in the Tour of Beam.
+For more information, see the community blog post 'Splittable `DoFn` in Apache Beam is Ready to Use'.
diff --git a/learning/prompts/documentation-lookup-nolinks/14_advanced_pipeline_patterns.md b/learning/prompts/documentation-lookup-nolinks/14_advanced_pipeline_patterns.md
new file mode 100644
index 00000000000..f6f86ba4051
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/14_advanced_pipeline_patterns.md
@@ -0,0 +1,14 @@
+Prompt:
+What pipeline patterns exist in Apache Beam?
+
+Response:
+Beam pipeline patterns are a set of best practices for building Apache Beam pipelines. They are based on real-world Beam deployments and are designed to help you build resilient, flexible, and portable pipelines.
+
+The following list includes some of the most common pipeline patterns:
+- File processing patterns
+- Pipeline options patterns
+- Side input patterns
+- Windowing patterns
+- State and Timers patterns
+
+For more information, see the 'Pipeline Patterns' section of the Beam Programming Guide.
diff --git a/learning/prompts/documentation-lookup-nolinks/15_advanced_xlang.md b/learning/prompts/documentation-lookup-nolinks/15_advanced_xlang.md
new file mode 100644
index 00000000000..a04ecb51f91
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/15_advanced_xlang.md
@@ -0,0 +1,11 @@
+Prompt:
+What is a multi-language pipeline in Apache Beam?
+
+Response:
+Apache Beam lets you combine transforms written in any supported SDK language (currently, Java and Python) and use them in one multi-language pipeline. These pipelines can, for instance, extract data from a Python source, apply processing via a Java transform, and then deliver the results to a Python sink. Another example is leveraging the Apache Kafka connector and SQL transform from the Java SDK within Python pipelines.
+
+To learn how to create a multi-language pipeline, refer to the multi-language pipelines quickstarts for Java and Python on the Apache Beam website.
+
+Depending on the SDK language of the pipeline, you can use a high-level SDK wrapper class or a low-level transform class to access a cross-language transform. For more information, see the 'Using cross-language transforms' section in the Apache Beam documentation.
+
+To develop a cross-language transform, you need to define a Uniform Resource Name (URN) for registering the transform with an expansion service. For more information, see the 'Defining a URN' section in the Beam Programming Guide.
diff --git a/learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md b/learning/prompts/documentation-lookup-nolinks/16_advanced_pipeline_lifecycle.md
similarity index 59%
copy from learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md
copy to learning/prompts/documentation-lookup-nolinks/16_advanced_pipeline_lifecycle.md
index c59a177c252..f05c6c942a8 100644
--- a/learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md
+++ b/learning/prompts/documentation-lookup-nolinks/16_advanced_pipeline_lifecycle.md
@@ -1,9 +1,8 @@
Prompt:
What is a pipeline development lifecycle in Apache Beam?
-Response:
+Response:
The Apache Beam pipeline development lifecycle is an iterative process that usually involves the following steps:
-
- Design your pipeline.
- Develop your pipeline code.
- Test your pipeline.
@@ -17,9 +16,6 @@ To design a pipeline, you need answers to the following questions:
- What do you want to do with your data?
- What does your output data look like, and where should it go?
-Beam documentation has more information about [pipeline design](https://beam.apache.org/documentation/pipelines/design-your-pipeline/) and about common [pipeline patterns](https://beam.apache.org/documentation/patterns/overview/).
-
-
An Apache Beam program expresses a data processing pipeline, from start to finish. To construct a pipeline using the classes in the Apache Beam SDKs, your program needs to perform the following steps:
- Create a `Pipeline` object.
@@ -28,9 +24,8 @@ An Apache Beam program expresses a data processing pipeline, from start to finis
- Write or otherwise output the final, transformed `PCollection` objects.
- Run the pipeline.
-The Apache Beam documentation has more information about [developing](https://beam.apache.org/documentation/programming-guide/) and [executing](https://beam.apache.org/documentation/pipelines/create-your-pipeline/) pipelines.
+The Apache Beam documentation has more information about pipeline design, development, execution, and common pipeline patterns.
-Testing pipelines is a particularly important step in developing an effective data processing solution. The indirect nature of the Beam model, in which your user code constructs a pipeline graph to be executed remotely, can make debugging-failed runs difficult. For more information about pipeline testing strategies, see [Test Your Pipeline](https://beam.apache.org/documentation/pipelines/test-your-pipeline/).
+Testing pipelines is a particularly important step in developing an effective data processing solution. The indirect nature of the Beam model, in which your user code constructs a pipeline graph to be executed remotely, can make debugging failed runs difficult. For more information about pipeline testing strategies, see the 'Test Your Pipeline' section in the Apache Beam documentation.
-Choosing a [runner](https://beam.apache.org/documentation/#choosing-a-runner) is a crucial step in deploying your pipeline. The runner you choose determines where and how your pipeline executes.
-For more information about pipeline deployment, see [Container environments](https://beam.apache.org/documentation/runtime/environments/).
+Choosing a runner is a crucial step in deploying your pipeline. The runner you choose determines where and how your pipeline executes. For more information about pipeline deployment, see 'Container environments' on the Apache Beam website.
diff --git a/learning/prompts/documentation-lookup-nolinks/17_advanced_ai_ml.md b/learning/prompts/documentation-lookup-nolinks/17_advanced_ai_ml.md
new file mode 100644
index 00000000000..4682cd810fe
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/17_advanced_ai_ml.md
@@ -0,0 +1,18 @@
+Prompt:
+What are the AI and ML capabilities in Apache Beam?
+
+Response:
+Apache Beam has several built-in AI and ML capabilities that enable you to:
+- Process large datasets for both preprocessing and model inference.
+- Conduct exploratory data analysis and smoothly scale up data pipelines in production as part of your MLOps ecosystem.
+- Run your models in production with varying data loads, both in batch and streaming pipelines.
+
+For common AI platform integration patterns in Apache Beam, see the 'AI Platform integration patterns' section in the Apache Beam documentation.
+
+The recommended way to implement inference in Apache Beam is by using the RunInference API. For more information about how to use RunInference for PyTorch, scikit-learn, and TensorFlow, see the 'Use RunInference in Apache Beam' example in the Apache Beam GitHub repository.
+
+Using pre-trained models in Apache Beam is also supported with PyTorch, Scikit-learn, and TensorFlow. Running inference on custom models is also supported.
+
+Apache Beam also supports automatically updating the model being used with the `RunInference PTransform` in streaming pipelines without stopping the pipeline. The feature lets you avoid downtime downtime. For more information, see 'Automatic model refresh' on the Apache Beam website.
+
+For more information about using machine learning models with Apache Beam, read the blog post 'Running ML models now easier with new Dataflow ML innovations on Apache Beam'.
diff --git a/learning/prompts/documentation-lookup-nolinks/18_io_kafka.md b/learning/prompts/documentation-lookup-nolinks/18_io_kafka.md
new file mode 100644
index 00000000000..010226a9d34
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/18_io_kafka.md
@@ -0,0 +1,29 @@
+Prompt:
+Is Apache Kafka supported in Apache Beam?
+
+Response:
+Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications. It is horizontally scalable, fault-tolerant, and fast.
+
+Apache Beam has a built-in KafkaIO connector for Apache Kafka that enables Apache Beam pipelines to read data from and write data to Kafka. Apache Kafka is supported in both batch and streaming pipelines and can serve as both a source and a sink. For more information, refer to the KafkaIO connector documentation for your preferred programming language.
+
+To get started with Apache Kafka and Apache Beam, refer to the Dataflow Cookbook repository.
+
+Here is an example of Apache Beam pipeline code for reading data from a Kafka topic:
+
+```python
+with beam.Pipeline(options=options) as p:
+ output = (
+ p
+ | "Read from Kafka"
+ >> ReadFromKafka(
+ consumer_config={"bootstrap.servers": "localhost:9092"},
+ topics=["my_topic"],
+ with_metadata=False,
+ )
+ | "Log Data" >> Map(logging.info)
+ )
+```
+
+This code reads data from a Kafka topic `my_topic` on `localhost:9092` and logs the data to the console.
+
+For a detailed demonstration of using the KafkaIO connector, refer to the KafkaIO example in the Apache Beam GitHub repository.
diff --git a/learning/prompts/documentation-lookup-nolinks/19_io_pubsub.md b/learning/prompts/documentation-lookup-nolinks/19_io_pubsub.md
new file mode 100644
index 00000000000..13c0b05042b
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/19_io_pubsub.md
@@ -0,0 +1,23 @@
+Prompt:
+Is Pub/Sub supported in Apache Beam?
+
+Response:
+Pub/Sub is a Google Cloud service that provides a simple, reliable, scalable, and secure real-time messaging service for sending and receiving messages between independent applications. Apache Beam provides a PubSubIO connector that allows you to read and write messages from and to Pub/Sub. Pub/Sub is currently supported only in streaming pipelines.
+
+Pub/Sub is supported natively in the Beam SDKs for Java, Python, and Go. It is also available in Typescript through X Language.
+
+To get started with Pub/Sub and Apache Beam, the Dataflow Cookbook GitHub repository provides code examples for reading and writing data from and to Pub/Sub.
+
+Here is an example of the Apache Beam pipeline code for reading data from a Pub/Sub topic and writing it to another topic:
+
+```python
+from apache_beam.io import ReadFromPubSub
+from apache_beam.io import WriteToPubSub
+
+with beam.Pipeline(options=options) as p:
+ (
+ p
+ | "Read from Pub/Sub" >> ReadFromPubSub(topic="input_topic")
+ | "Write to Pub/Sub" >> WriteToPubSub(topic="output_topic")
+ )
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/20_io_biguery.md b/learning/prompts/documentation-lookup-nolinks/20_io_biguery.md
new file mode 100644
index 00000000000..427e25c43a1
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/20_io_biguery.md
@@ -0,0 +1,42 @@
+Prompt:
+Is BigQuery supported in Apache Beam?
+
+Response:
+Yes, Apache Beam supports BigQuery. BigQuery is a serverless and cost-effective enterprise data warehouse offered by Google Cloud. Apache Beam provides a BigQueryIO connector to read and write data from and to BigQuery. The BigQueryIO connector supports both batch and streaming pipelines.
+
+The following Apache Beam SDKs support the BigQueryIO connector:
+* Java (natively)
+* Python (natively)
+* Go (natively and through X Language)
+* Typescript (through X Language)
+
+To read data from BigQuery, use the `ReadFromBigQuery` function. Apache Beam can read data directly from a BigQuery table or using an SQL query. The default mode is to return table rows read from a BigQuery source as dictionaries. Built-in `TableRow` objects can also be returned.
+
+Here is an example of Apache Beam pipeline code for reading from BigQuery:
+
+```python
+from apache_beam.io.gcp.bigquery import ReadFromBigQuery
+
+with beam.Pipeline(options=options) as p:
+ # read from a table
+ lines_table = p | "Read" >> ReadFromBigQuery(table=table)
+ # read from a query
+ lines_query = p | "Read" >> ReadFromBigQuery(query="SELECT * FROM table")
+```
+
+Here is an example of Apache Beam pipeline code for writing to BigQuery:
+
+```python
+from apache_beam.io.gcp.bigquery import WriteToBigQuery
+
+with beam.Pipeline(options=options) as p:
+ # write to a table
+ p | "Write" >> beam.io.WriteToBigQuery(
+ table,
+ schema=TABLE_SCHEMA,
+ create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+ write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
+ )
+```
+
+To get started with BigQuery and Apache Beam, see the examples in the Dataflow Cookbook GitHub repository.
diff --git a/learning/prompts/documentation-lookup-nolinks/21_io_bigtable.md b/learning/prompts/documentation-lookup-nolinks/21_io_bigtable.md
new file mode 100644
index 00000000000..28931de2ee8
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/21_io_bigtable.md
@@ -0,0 +1,28 @@
+Prompt:
+Is Cloud Bigtable supported in Apache Beam?
+
+Response:
+Cloud Bigtable is a Google Cloud service that provides enterprise-grade NoSQL database service with single-digit millisecond latency, limitless scale, and 99.999% availability for large analytical and operational workloads.
+
+Apache Beam has a built-in connector for Cloud Bigtable that lets you read data from and write data to Bigtable. For more information, see the BigtableIO connector documentation for your preferred programming language.
+
+To get started with Cloud Bigtable and Apache Beam, refer to the Dataflow Cookbook GitHub repository.
+
+Here is an example of Apache Beam pipeline code for reading data from a Bigtable table:
+
+```python
+with beam.Pipeline(options=options) as p:
+ output = (
+ p
+ | "Read from Bigtable"
+ >> ReadFromBigtable(
+ project_id=project_id,
+ instance_id=instance_id,
+ table_id=table_id,
+ )
+ | "Extract cells" >> beam.Map(lambda row: row._cells)
+ | "Log Data" >> Map(logging.info)
+ )
+```
+
+For performance benchmarks of the Bigtable IO connector, see the 'Bigtable performance' section on the Apache Beam website.
diff --git a/learning/prompts/documentation-lookup-nolinks/22_io_spanner.md b/learning/prompts/documentation-lookup-nolinks/22_io_spanner.md
new file mode 100644
index 00000000000..bad9dc78bb7
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/22_io_spanner.md
@@ -0,0 +1,33 @@
+Prompt:
+Is Cloud Spanner supported in Apache Beam?
+
+Response:
+Cloud Spanner is a Google Cloud service that provides a fully managed, mission-critical, relational database service that offers transactional consistency at global scale, schemas, SQL (ANSI 2011 with extensions), and automatic, synchronous replication for high availability.
+
+Apache Beam has a built-in SpannerIO connector for Cloud Spanner that enables Apache Beam pipelines to read data from and write data to Cloud Spanner. Cloud Spanner is supported in both batch pipelines and streaming pipelines and can serve both as a source and a sink. For more information, see the SpannerIO connector documentation for Java, Python, or Go.
+
+To get started with Cloud Spanner and Apache Beam, refer to the Dataflow Cookbook GitHub repository.
+
+Here is an example of Apache Beam pipeline code for reading data from a Cloud Spanner table using an SQL query:
+
+```python
+class ExampleRow(NamedTuple):
+ id: int
+ name: str
+
+
+with beam.Pipeline(options=options) as p:
+ output = (
+ p
+ | "Read from table"
+ >> ReadFromSpanner(
+ project_id=options.project_id,
+ instance_id=options.instance_id,
+ database_id=options.database_id,
+ row_type=ExampleRow,
+ sql="SELECT * FROM example_row",
+ )
+ | "Map Data" >> Map(lambda row: f"Id = {row.id}, Name = {row.name}")
+ | "Log Data" >> Map(logging.info)
+ )
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/23_io_gcs.md b/learning/prompts/documentation-lookup-nolinks/23_io_gcs.md
new file mode 100644
index 00000000000..75ef3b85919
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/23_io_gcs.md
@@ -0,0 +1,15 @@
+Prompt:
+Is the Google Cloud Storage file system supported in Apache Beam?
+
+Response:
+Google Cloud Storage is an object storage service for storing and accessing data on Google Cloud. Apache Beam supports both working with the Google Cloud Storage file system and reading data from and writing data to Google Cloud Storage buckets.
+
+Apache Beam has a built-in TextIO connector for reading data from and writing data to Google Cloud Storage buckets. For more information, see the TextIO connector documentation for your programming language of choice.
+
+The Apache Beam AvroIO, XMLIO, TFRecordIO, and ParquetIO connectors also support reading data from and writing data to different file formats within Google Cloud Storage buckets.
+
+You can specify the Google Cloud Storage file path using the following format: `gs://<bucket>/<path>`. For example, `gs://my-bucket/my-file.txt`.
+
+You can use wildcards in read and write transforms to process data from multiple files or write data to multiple files simultaneously. For instance, you can specify `gs://my-bucket/my-files-*.txt` to read or write to files with names matching the specified pattern.
+
+Apache Beam also allows pipelines to interact directly with the Google Cloud Storage file system. For instance, you can use this file system to verify the existence of a file, retrieve a list of files in a directory, or delete a file. The Google Cloud Storage file system is supported in Java, Python, and Go.
diff --git a/learning/prompts/documentation-lookup-nolinks/24_io_json.md b/learning/prompts/documentation-lookup-nolinks/24_io_json.md
new file mode 100644
index 00000000000..9c2ca19538b
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/24_io_json.md
@@ -0,0 +1,29 @@
+Prompt:
+Is JSON supported in Apache Beam?
+
+Response:
+JSON is a data format for storing and exchanging data. Apache Beam supports reading data from and writing data to JSON files using the `ReadFromJson` and `WriteToJson` transforms within the `TextIO` module. For more information, see the TextIO connector documentation for your programming language of choice.
+
+To get started with JSON and Apache Beam, refer to the Dataflow Cookbook GitHub repository.
+
+Here is an example of Apache Beam pipeline code for reading data from a JSON file:
+
+```python
+class JsonOptions(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_argument(
+ "--file_path",
+ default="gs://your-bucket/your-file.json",
+ help="Json file path",
+ )
+
+
+options = JsonOptions()
+with beam.Pipeline(options=options) as p:
+ output = (
+ p
+ | "Read from Json file" >> ReadFromJson(path=options.file_path, lines=False)
+ | "Log Data" >> Map(logging.info)
+ )
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/25_io_csv.md b/learning/prompts/documentation-lookup-nolinks/25_io_csv.md
new file mode 100644
index 00000000000..89e87ee4e80
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/25_io_csv.md
@@ -0,0 +1,30 @@
+Prompt:
+Is CSV format supported in Apache Beam?
+
+Response:
+CSV is a data format for storing and exchanging data. Apache Beam supports reading data from and writing data to CSV files using the `ReadFromText` and `WriteToText` transforms in the `TextIO` module. For more information, see the TextIO connector documentation for your programming language of choice.
+
+To get started with CSV and Apache Beam, refer to the Dataflow Cookbook GitHub repository.
+
+Here is an example of Apache Beam pipeline code for reading data from a CSV file:
+
+```python
+class CsvOptions(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_argument(
+ "--file_path",
+ default="gs://your-bucket/your-file.csv",
+ help="Csv file path",
+ )
+
+
+options = CsvOptions()
+
+with beam.Pipeline(options=options) as p:
+ output = (
+ p
+ | "Read from Csv file" >> ReadFromCsv(path=options.file_path)
+ | "Log Data" >> Map(logging.info)
+ )
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/26_io_avro.md b/learning/prompts/documentation-lookup-nolinks/26_io_avro.md
new file mode 100644
index 00000000000..d20710b3470
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/26_io_avro.md
@@ -0,0 +1,25 @@
+Prompt:
+Is Apache Avro format supported in Apache Beam?
+
+Response:
+Apache Avro is a data format for storing and exchanging data. Apache Beam supports reading from and writing to Avro files using the `ReadFromAvro` and `WriteToAvro` transforms in the `AvroIO` module. For more information, see the AvroIO connector documentation for your programming language of choice.
+
+To get started with Avro and Apache Beam, refer to the Dataflow Cookbook GitHub repository.
+
+Here is an example of Apache Beam pipeline code for reading data from an Avro file:
+
+```python
+class ReadAvroOptions(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_argument(
+ "--path",
+ default="gs://cloud-samples-data/bigquery/us-states/*.avro",
+ help="GCS path to read from",
+ )
+
+ options = ReadAvroOptions()
+
+ with beam.Pipeline(options=options) as p:
+ (p | "Read from Avro" >> ReadFromAvro(options.path) | Map(logging.info))
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/27_io_parquet.md b/learning/prompts/documentation-lookup-nolinks/27_io_parquet.md
new file mode 100644
index 00000000000..a533bb830ee
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/27_io_parquet.md
@@ -0,0 +1,21 @@
+Prompt:
+Is Apache Parquet format supported in Apache Beam?
+
+Response:
+Apache Parquet is a data format for storing and exchanging data. Apache Beam supports reading from and writing to Parquet files using the `ReadFromParquet` and `WriteToParquet` transforms in the `ParquetIO` module. For more information, see the ParquetIO connector documentation for your programming language of choice.
+
+To get started with Avro and Apache Beam, refer to the Dataflow Cookbook GitHub repository.
+
+Here is an example of Apache Beam pipeline code for reading data from a Parquet file:
+
+```python
+class ReadParquetOptions(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_argument("--path", help="GCS path to parquet file")
+
+ options = ReadParquetOptions()
+
+ with beam.Pipeline(options=options) as p:
+ (p | "ReadParquet" >> ReadFromParquet(options.path) | Map(logging.info))
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/28_io_jdbc.md b/learning/prompts/documentation-lookup-nolinks/28_io_jdbc.md
new file mode 100644
index 00000000000..b98f992d067
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/28_io_jdbc.md
@@ -0,0 +1,42 @@
+Prompt:
+Is Oracle Java Database Connectivity (JDBC) supported in Apache Beam?
+
+Response:
+The Oracle Java Database Connectivity (JDBC) API is a Java API for connecting to relational databases. Apache Beam supports reading and writing data with JDBC using the `ReadFromJdbc` and `WriteToJdbc` transforms in the `JdbcIO` module. The JdbcIO connector is native to Java and available through cross-language transforms in Python and Go.
+
+To get started with JDBC and Apache Beam, refer to the Dataflow Cookbook GitHub repository.
+
+Here is an example of Apache Beam pipeline code for reading data with JDBC:
+
+```python
+class ExampleRow(NamedTuple):
+ id: int
+ name: str
+
+
+class ReadJdbcOptions(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_argument("--driverClassName", help="JDBC driver class name")
+ parser.add_argument("--url", help="JDBC URL")
+ parser.add_argument("--username", help="JDBC username")
+ parser.add_argument("--password", help="JDBC password")
+ parser.add_argument("--query", default="SELECT * FROM users", help="JDBC query")
+
+
+options = ReadJdbcOptions()
+
+with beam.Pipeline(options=options) as p:
+ (
+ p
+ | "Read from JDBC"
+ >> ReadFromJdbc(
+ driverClassName=options.driverClassName,
+ url=options.url,
+ username=options.username,
+ password=options.password,
+ query=options.query,
+ )
+ | Map(logging.info)
+ )
+```
diff --git a/learning/prompts/documentation-lookup-nolinks/29_advanced_side_inputs.md b/learning/prompts/documentation-lookup-nolinks/29_advanced_side_inputs.md
new file mode 100644
index 00000000000..d857380b820
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/29_advanced_side_inputs.md
@@ -0,0 +1,36 @@
+Prompt:
+What are side inputs in Apache Beam?
+
+Response:
+In Apache Beam, side inputs are additional inputs that can be provided to a `ParDo` transform, in addition to the main input `PCollection`. These side inputs can be used by the `ParDo` transform's `DoFn` to access and read additional data during the processing of each element in the main input `PCollection`.
+
+Side inputs are particularly useful when a Beam pipeline needs to ingest additional data dynamically at runtime when processing each element, rather than relying on preset or hard-coded values. By leveraging side inputs, Apache Beam can determine additional data based on the main `PCollection` data or even from a different branch of your pipeline. For example, you can leverage side inputs for your streaming analytics use cases, enriching streaming events with additional data retrieved fr [...]
+
+In the Apache Beam Python SDK, you can provide side inputs as extra arguments in the `DoFn`'s `process` method or the `Map`/`FlatMap` transforms. The Python SDK supports optional, positional, and keyword arguments.
+
+```python
+class MyDoFn(beam.DoFn):
+ def process(self, element, side_input):
+ ...
+```
+
+In the Apache Beam Java SDK, side inputs are provided to the `ParDo` transform by using the `.withSideInputs` method. To access a side input from within a `DoFn`, the `DoFn.ProcessContext.sideInput` method is invoked.
+
+```java
+PCollection<Integer> input = ...;
+PCollectionView<Integer> sideInput = ...;
+PCollection<Integer> output =
+ input.apply(ParDo.of(new DoFn<Integer, Integer>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ Integer sideInputValue = c.sideInput(sideInput);
+ ...
+ }
+ }).withSideInputs(sideInput));
+```
+
+Side inputs can also be used to process windowed data. Apache Beam uses the windows for the main input elements to look up the appropriate windows for the side input elements. Apache Beam projects the main input’s window to the side input's window set and uses the side input from the resulting window. Main and side inputs might have identical or different windowing strategies.
+
+For example, if the main input `PCollection` is windowed into ten-minute windows, and the side input is windowed into one-hour windows, Apache Beam projects the main input window set and selects the value from the corresponding one-hour side input window. For more information, refer to the Apache Beam documentation on side inputs and windowing.
+
+For common patterns and code samples to implement side inputs in Apache Beam pipelines, refer to the Apache Beam documentation on side input patterns.
diff --git a/learning/prompts/documentation-lookup-nolinks/30_advanced_additional_outputs.md b/learning/prompts/documentation-lookup-nolinks/30_advanced_additional_outputs.md
new file mode 100644
index 00000000000..9c3feeda22d
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/30_advanced_additional_outputs.md
@@ -0,0 +1,100 @@
+Prompt:
+What are additional outputs in Apache Beam?
+
+Response:
+In Apache Beam, additional outputs, also known as tagged or side outputs, refer to one or multiple extra `PCollection` objects produced by a single `ParDo` transform in addition to the main output `PCollection`. The `ParDo` transform has the flexibility to produce any number of additional output `PCollection` objects and return them bundled together with the main output `PCollection`.
+
+Additional outputs serve as a mechanism to implement pipeline branching. You can use them when there is a need to split the output of a single transform into several `PCollection` objects or produce outputs in different formats. Additional outputs become particularly beneficial when a transform’s computation per element is time-consuming because they enable transforms to process each element in the input `PCollection` just once.
+
+Producing additional outputs requires tagging each output `PCollection` with a unique identifier, which is then used to emit elements to the corresponding output.
+
+In the Apache Beam Java SDK, you can implement additional outputs by creating a `TupleTag` object to identify each collection produced by the `ParDo` transform. After specifying the `TupleTag` objects for each of the outputs, the tags are passed to the `ParDo` using the `.withOutputTags` method.
+
+The following Java code implements two additional output `PCollection` objects for string and integer values in addition to the main output `PCollection` of strings:
+
+```java
+// Input PCollection that contains strings.
+PCollection<String> input = ...;
+// Output tag for the main output PCollection of strings.
+final TupleTag<String> mainOutputTag = new TupleTag<String>() {};
+// Output tag for the additional output PCollection of strings.
+final TupleTag<String> additionalOutputTagString = new TupleTag<Integer>() {};
+// Output tag for the additional output PCollection of integers.
+final TupleTag<Integer> additionalOutputTagIntegers = new TupleTag<Integer>() {};
+
+PCollectionTuple results = input.apply(ParDo
+ .of(new DoFn<String, String>() {
+ // DoFn continues here.
+ ...
+ })
+ // Specify the tag for the main output.
+ .withOutputTags(mainOutputTag,
+ // Specify the tags for the two additional outputs as a TupleTagList.
+ TupleTagList.of(additionalOutputTagString)
+ .and(additionalOutputTagIntegers)));
+```
+
+The `processElement` method can emit elements to the main output or any additional output by invoking the output method on the `MultiOutputReceiver` object. The output method takes the tag of the output and the element to be emitted as arguments.
+
+```java
+public void processElement(@Element String word, MultiOutputReceiver out) {
+ if (condition
+ for main output) {
+ // Emit element to main output
+ out.get(mainOutputTag).output(word);
+ } else {
+ // Emit element to additional string output
+ out.get(additionalOutputTagString).output(word);
+ }
+ if (condition
+ for additional integer output) {
+ // Emit element to additional integer output
+ out.get(additionalOutputTagIntegers).output(word.length());
+ }
+}
+```
+
+In the Apache Beam Python SDK, you can implement additional outputs by invoking the `with_outputs()` method on the `ParDo` transform and specifying the expected tags for the multiple outputs.
+
+The following Python code demonstrates how to implement additional outputs for a `ParDo` transform that outputs two `PCollection` objects of strings and integers in addition to the main output `PCollection` of strings:
+
+```python
+class SplitLinesToWordsFn(beam.DoFn):
+ # These tags will be used to tag the outputs of this DoFn.
+ OUTPUT_TAG_SHORT_WORDS = "tag_short_words"
+ OUTPUT_TAG_CHARACTER_COUNT = "tag_character_count"
+
+ def process(self, element):
+ # yield a count (integer) to the OUTPUT_TAG_CHARACTER_COUNT tagged collection.
+ yield pvalue.TaggedOutput(self.OUTPUT_TAG_CHARACTER_COUNT, len(element))
+
+ words = re.findall(r"[A-Za-z\']+", element)
+ for word in words:
+ if len(word) <= 3:
+ # yield word as an output to the OUTPUT_TAG_SHORT_WORDS tagged collection.
+ yield pvalue.TaggedOutput(self.OUTPUT_TAG_SHORT_WORDS, word)
+ else:
+ # yield word to add it to the main collection.
+ yield word
+```
+
+The method returns a `DoOutputsTuple` object, with the specified tags serving as attributes that provide `ParDo` with access to the corresponding output `PCollection` objects.
+
+```python
+with beam.Pipeline(options=pipeline_options) as p:
+ lines = p | ReadFromText(known_args.input)
+
+ # with_outputs allows accessing the explicitly tagged outputs of a DoFn.
+ split_lines_result = lines | beam.ParDo(SplitLinesToWordsFn()).with_outputs(
+ SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
+ SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
+ main="words",
+ )
+
+ # split_lines_result is an object of type DoOutputsTuple
+ words, _, _ = split_lines_result
+ short_words = split_lines_result[SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS]
+ character_count = split_lines_result.tag_character_count
+```
+
+Refer to the multiple output `ParDo` example in the Apache Beam GitHub for the complete example code.
diff --git a/learning/prompts/documentation-lookup-nolinks/31_advanced_batched_dofn.md b/learning/prompts/documentation-lookup-nolinks/31_advanced_batched_dofn.md
new file mode 100644
index 00000000000..061ac084f8c
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/31_advanced_batched_dofn.md
@@ -0,0 +1,39 @@
+Prompt:
+What is a batched `DoFn` in Apache Beam?
+
+Response:
+In Apache Beam, a batched `DoFn` is a specialized type of `DoFn` that operates on batches of logical elements within a `PCollection`. Unlike regular `DoFn` functions, which process each element individually, batched `DoFn` functions enable the efficient processing of multiple elements together as batches.
+
+Batched `DoFn` functions are currently only available in the Apache Beam Python SDK. They leverage powerful Python vectorized libraries, such as NumPy, SciPy, and pandas, which are designed for analysis and computing at scale. Batched `DoFn` functions are particularly useful when dealing with large amounts of data, allowing for parallel and optimized processing. Additionally, you can chain multiple batched `DoFn` functions together to create a more efficient data processing pipeline.
+
+To implement a batched `DoFn`, you define a `process_batch` method instead of the typical `process` method used in a regular `DoFn`. The `process_batch` method takes a batch of elements as input and produces a batch of elements as an output. It is important to note that batches must have a single set of timing properties (event time, window, etc.) for every logical element within the batch; batches cannot span multiple timestamps. For some `DoFn` functions, you can provide both a batched [...]
+
+Here is a simple example of a flexible `DoFn` with both batched and element-wise implementations:
+
+```python
+class MultiplyByTen(beam.DoFn):
+ def process(self, element: np.int64) -> Iterator[np.int64]:
+ yield element * 10
+
+ def process_batch(self, batch: np.ndarray) -> Iterator[np.ndarray]:
+ yield batch * 10
+```
+
+In this example, when `MultiplyByTen` is applied to a `PCollection`, Apache Beam recognizes that `np.ndarray` is an acceptable batch type to use in conjunction with `np.int64` elements. The `np.int64` type represents the individual element. The `process` method multiplies an element by ten, yielding a single element. The `np.ndarray` type represents the batch. The `process_batch` method multiplies each element in the batch by ten, yielding a single batch. During pipeline execution, Apach [...]
+
+By default, Apache Beam implicitly buffers elements and creates batches on the input side, then explodes batches back into individual elements on the output side. However, if batched `DoFn` functions with equivalent types are chained together, this batch creation and explosion process is skipped, and the batches are passed through for more efficient processing.
+
+Here is an example with chained `DoFn` functions of equivalent types:
+
+```python
+(p | beam.Create([1, 2, 3, 4]).with_output_types(np.int64)
+ | beam.ParDo(MultiplyByTen()) # Implicit buffering and batch creation
+ | beam.ParDo(MultiplyByTen()) # Batches passed through
+ | beam.ParDo(MultiplyByTen()))
+```
+
+In this example, the `PTransform.with_output_types` method sets the element-wise typehint for the output. Thus, when the `MultiplyByTen` class is applied to a `PCollection`, Apache Beam recognizes that `np.ndarray` is an acceptable batch type to use in conjunction with `np.int64` elements.
+
+The `process_batch` method is expected to produce batched outputs, while the `process` method should produce individual elements. You can customize this production logic with the `@beam.DoFn.yields_elements` and `@beam.DoFn.yields_batches` decorators.
+
+For more details, including advanced usage and code samples, you can refer to the Apache Beam documentation on batched `DoFn` types.
diff --git a/learning/prompts/documentation-lookup-nolinks/53_dataflow_runner.md b/learning/prompts/documentation-lookup-nolinks/53_dataflow_runner.md
new file mode 100644
index 00000000000..7a9990c93b1
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/53_dataflow_runner.md
@@ -0,0 +1,80 @@
+Prompt:
+How to execute Apache Beam pipelines with Cloud Dataflow?
+
+Response:
+Apache Beam supports multiple runners for executing pipelines on various platforms. One such runner is the Cloud Dataflow Runner, tailored for running pipelines on the Google Cloud Dataflow service. Cloud Dataflow offers fully managed and unified stream and batch data processing, boasting dynamic work rebalancing and built-in autoscaling capabilities.
+
+When you execute a pipeline on Cloud Dataflow, the Runner uploads your code and dependencies to a Cloud Storage bucket and creates a Dataflow job, which then executes your pipeline on managed resources within the Google Cloud Platform.
+
+To execute Apache Beam pipelines using the Cloud Dataflow Runner, follow these steps:
+
+***1. Setup Your Cloud Project and Resources:***
+
+Complete the steps outlined in the 'Before You Begin' section of the Cloud Dataflow quickstart for your chosen programming language:
+1. Select or create a Google Cloud Platform Console project.
+2. Enable billing for your project.
+3. Enable the required Google Cloud APIs, including Cloud Dataflow, Compute Engine, Stackdriver Logging, Cloud Storage, Cloud Storage JSON, and Cloud Resource Manager. Additional APIs may be necessary depending on your pipeline code.
+4. Authenticate with Google Cloud Platform.
+5. Install the Google Cloud SDK.
+6. Create a Cloud Storage bucket.
+
+***2. Specify Dependencies (Java Only):***
+
+When using the Apache Beam Java SDK, specify your dependency on the Cloud Dataflow Runner in the `pom.xml` file of your Java project directory.
+
+```java
+<dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
+ <version>2.54.0</version>
+ <scope>runtime</scope>
+</dependency>
+```
+
+Ensure that you include all necessary dependencies to create a self-contained application. In some cases, such as when starting a pipeline using a scheduler, you'll need to package a self-executing JAR by explicitly adding a dependency in the Project section of your `pom.xml` file. For more details about running self-executing JARs on Cloud Dataflow, refer to the 'Self-executing JAR' section in the Apache Beam documentation on Cloud Dataflow Runner.
+
+***3. Configure Pipeline Options:***
+
+Configure the execution details, including the runner (set to `dataflow` or `DataflowRunner`), Cloud project ID, region, and streaming mode, using the `GoogleCloudOptions` interface for Python or the `DataflowPipelineOptions` interface for Java.
+
+You can utilize pipeline options to control various aspects of how Cloud Dataflow executes your job. For instance, you can specify whether your pipeline runs on worker virtual machines, on the Cloud Dataflow service backend, or locally. For additional pipeline configuration options, refer to the reference documentation for the respective interface.
+
+***4. Run Your Pipeline on Cloud Dataflow:***
+
+Execute your pipeline on Cloud Dataflow using the appropriate command for your SDK.
+
+The following example code, taken from the Cloud Dataflow quickstarts for Java and Python, shows how to run the WordCount example pipeline on Dataflow.
+
+For the Apache Beam Java SDK, in your terminal, run the following command (from your `word-count-beam` directory):
+
+```java
+mvn -Pdataflow-runner compile exec:java \
+ -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="--project=PROJECT_ID \
+ --gcpTempLocation=gs://BUCKET_NAME/temp/ \
+ --output=gs://BUCKET_NAME/output \
+ --runner=DataflowRunner \
+ --region=REGION
+```
+
+For the Apache Beam Python SDK, in your terminal, run the following command:
+
+```python
+python -m apache_beam.examples.wordcount \
+ --region DATAFLOW_REGION \
+ --input gs://dataflow-samples/shakespeare/kinglear.txt \
+ --output gs://STORAGE_BUCKET/results/outputs \
+ --runner DataflowRunner \
+ --project PROJECT_ID \
+ --temp_location gs://STORAGE_BUCKET/tmp/
+```
+
+Replace placeholders such as PROJECT_ID, BUCKET_NAME, and REGION with your Cloud project-specific details.
+
+To learn more about running pipelines on Cloud Dataflow, visit the Cloud Dataflow quickstart for your preferred programming language.
+
+***5. Monitor Your Cloud Dataflow Job:***
+
+Monitor the job's progress, view execution details, and receive updates on the pipeline's results using the Dataflow Monitoring Interface or the Dataflow Command-line Interface.
+
+For more information on the supported capabilities of the Cloud Dataflow Runner, refer to the Beam Capability Matrix in the Apache Beam documentation.
diff --git a/learning/prompts/documentation-lookup-nolinks/54_spark_runner.md b/learning/prompts/documentation-lookup-nolinks/54_spark_runner.md
new file mode 100644
index 00000000000..a692aa144ea
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/54_spark_runner.md
@@ -0,0 +1,148 @@
+Prompt:
+How to execute Apache Beam pipelines with Apache Spark?
+
+Response:
+Apache Beam supports multiple runners for executing pipelines on various platforms. One such runner is the Spark Runner, tailored for running pipelines using Apache Spark. The Spark Runner enables you to leverage the scalability and parallel processing capabilities of Apache Spark for your data processing tasks.
+
+Key features of the Spark Runner include:
+* Support for batch, streaming, and unified pipelines.
+* Offering the same fault-tolerance guarantees and security features as provided by Apache Spark.
+* Built-in metrics reporting using Apache Spark’s metrics system, which also reports Beam Aggregators.
+* Native support for Apache Beam side inputs via Apache Spark’s broadcast variables.
+
+There are three types of Spark Runners available:
+1. Legacy Spark Runner: supports Java (and other JVM-based languages) exclusively, based on Apache Spark’s RDD and DStream.
+2. Structured Streaming Spark Runner: supports Java (and other JVM-based languages) exclusively, based on Apache Spark's Datasets and Structured Streaming framework. Currently, it only supports batch mode with limited coverage of the Apache Beam model.
+3. Portable Spark Runner: supports Java, Python, and Go.
+
+For Java-based applications, consider using the Java-based runners, while for Python or Go pipelines, opt for the portable Runner.
+
+The Spark Runner can execute Spark pipelines similar to a native Spark application, allowing deployment as a self-contained application for local mode, running on Spark Standalone Resource Manager (RM), or using YARN or Mesos.
+
+To execute your Apache Beam pipeline on a Spark Standalone RM, follow these steps:
+
+***Java-based Non-portable Spark Runners (Java Only)***
+
+***1. Specify Dependencies:***
+
+In the `pom.xml` file of your Java project directory, specify your dependency on the latest version of the Spark Runner:
+
+```java
+<dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-spark-3</artifactId>
+ <version>2.54.0</version>
+</dependency>
+```
+
+***2. Deploy Spark with Your Application:***
+
+When running pipelines in a Spark Standalone mode, ensure that your self-contained application includes Spark dependencies explicitly in your `pom.xml` file:
+
+```java
+<dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-core_2.12</artifactId>
+ <version>${spark.version}</version>
+</dependency>
+
+<dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-streaming_2.12</artifactId>
+ <version>${spark.version}</version>
+</dependency>
+```
+
+Shade the application JAR using the Maven shade plugin and make sure the shaded JAR file is visible in the target directory by running `is target`.
+
+To run pipelines in a Spark Standalone mode using the legacy RDD/DStream-based Spark Runner, use the following command:
+
+```java
+spark-submit --class com.beam.examples.BeamPipeline --master spark://HOST:PORT target/beam-examples-1.0.0-shaded.jar --runner=SparkRunner
+```
+
+To run pipelines in a Spark Standalone mode using the Structured Streaming Spark Runner, run the following command:
+
+```java
+spark-submit --class com.beam.examples.BeamPipeline --master spark://HOST:PORT target/beam-examples-1.0.0-shaded.jar --runner=SparkStructuredStreamingRunner
+```
+
+***3. Configure Pipeline Options:***
+
+Set the runner option in your pipeline options to specify that you want to use the Spark Runner. In Java, you can do this as follows:
+
+```java
+SparkPipelineOptions options = PipelineOptionsFactory.as(SparkPipelineOptions.class);
+options.setRunner(SparkRunner.class);
+```
+
+For additional pipeline configuration options, refer to the Spark Runner documentation.
+
+***4. Run Your Pipeline:***
+
+In Java, you can use the `PipelineRunner` to run your pipeline:
+
+```java
+Pipeline p = Pipeline.create(options);
+// Add transforms to your pipeline
+p.run();
+```
+
+***5. Monitor Your Job:***
+
+Monitor the execution of your pipeline using the Apache Spark Web Interfaces, which provides information about tasks, stages, and overall progress. Access the Spark UI by navigating to the appropriate URL (usually `localhost:4040`). Metrics are also accessible via the Apache Beam REST API. Apache Spark offers a metrics system for reporting metrics to various sinks.
+
+***Portable Spark Runner (Python)***
+
+***1. Deploy Spark with Your Application:***
+
+You will need Docker installed in your execution environment. Pre-built Spark Job Service Docker images are available on Docker Hub.
+
+Start the JobService endpoint:
+
+```python
+docker run --net=host apache/beam_spark_job_server:latest
+```
+A Beam JobService is a central instance where you submit your Apache Beam pipeline. It needs to be provided with the Spark master address to create a job for execution on your Spark cluster.
+
+Submit the Python pipeline to this endpoint, providing Beam JobService with the Spark master address to execute the job on a Spark cluster:
+
+```python
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+
+options = PipelineOptions([
+ "--runner=PortableRunner",
+ "--job_endpoint=localhost:8099", # localhost:8099 is the default address of the JobService
+ "--environment_type=LOOPBACK"
+])
+with beam.Pipeline(options) as p:
+ ...
+```
+
+***2. Configure Pipeline Options:***
+
+Set the runner option in your pipeline options to specify that you want to use the Spark Runner. In Python, you can do this as follows:
+
+```python
+from apache_beam.options.pipeline_options import PipelineOptions
+options = PipelineOptions()
+options.view_as(SparkRunnerOptions).runner = 'SparkRunner'
+```
+
+For additional pipeline configuration options, refer to the Spark Runner documentation.
+
+***3. Run Your Pipeline:***
+
+In Python, you can use the `run()` method of your pipeline object to execute the pipeline:
+
+```python
+# Run your pipeline
+p.run()
+```
+
+***4. Monitor Your Job:***
+
+You can monitor the execution of your pipeline using the Apache Spark Web Interfaces, which provides information about tasks, stages, and overall progress. Access the Spark UI by navigating to the appropriate URL (usually `localhost:4040`) in your web browser. Spark metrics are not yet supported on the portable runner.
+
+For more details and alternative methods of deployment, check the Apache Beam documentation on Spark Runner capabilities.
diff --git a/learning/prompts/documentation-lookup-nolinks/55_direct_runner.md b/learning/prompts/documentation-lookup-nolinks/55_direct_runner.md
new file mode 100644
index 00000000000..8a28c543249
--- /dev/null
+++ b/learning/prompts/documentation-lookup-nolinks/55_direct_runner.md
@@ -0,0 +1,68 @@
+Prompt:
+How to run Apache Beam pipelines locally?
+
+Response:
+You can run Apache Beam pipelines locally on your machine using the Apache Beam Direct Runner. It is designed for testing and development, ensuring that pipelines adhere closely to the Apache Beam model and are robust across different Beam runners. This option is suitable for testing and debugging pipelines, allowing quick iteration on code with immediate result visualization and unit testing using preferred local debugging tools.
+
+The Direct Runner performs additional checks to ensure users do not rely on semantics not guaranteed by the model. These checks include:
+* Enforcing immutability of elements.
+* Enforcing encodability of elements.
+* Ensuring elements are processed in an arbitrary order at all points.
+* Ensuring serialization of user functions (`DoFn`, `CombineFn`, etc.).
+
+To execute your Apache Beam pipeline locally using the Direct Runner, follow these steps:
+
+***1. Specify Dependencies (Java Only):***
+
+When using the Apache Beam Java SDK, specify your dependency on the Direct Runner in the `pom.xml` file of your Java project directory:
+
+```java
+<dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-direct-java</artifactId>
+ <version>2.54.0</version>
+ <scope>runtime</scope>
+</dependency>
+```
+
+Make sure to include all necessary dependencies to create a self-contained application and compile your Java code into a JAR file.
+
+***2. Configure Pipeline Options:***
+
+While you can configure your pipeline by creating a `PipelineOptions` object and setting fields directly, the Beam SDKs include a command-line parser for setting fields using command-line arguments.
+
+To read options from the command line, construct your `PipelineOptions` object as shown:
+
+Java SDK:
+
+```java
+PipelineOptions options =
+ PipelineOptionsFactory.fromArgs(args).withValidation().create();
+```
+
+Python SDK:
+
+```python
+from apache_beam.options.pipeline_options import PipelineOptions
+beam_options = PipelineOptions()
+```
+
+This method enables specifying options as command-line arguments in the format `--<option>=<value>`. Set the `runner` to `direct` or `DirectRunner`. Default values for other pipeline options are usually sufficient. For additional options, refer to the reference documentation for the `DirectOptions` interface for Python and Java.
+
+***3. Run the Pipeline:***
+
+Execute your pipeline on the Direct Runner from the command line.
+
+For Python SDK, use the `python -m` syntax to invoke the Python module containing your pipeline code:
+
+```python
+python -m your_module_name
+```
+
+For Java SDK, run your Java JAR file using the `java -jar` command:
+
+```java
+java -jar your-project.jar
+```
+
+For additional information and considerations, refer to the Apache Beam documentation on the Direct Runner.
diff --git a/learning/prompts/documentation-lookup/04_basic_pcollections.md b/learning/prompts/documentation-lookup/04_basic_pcollections.md
index 5fefa5873f5..e2431c8e3e5 100644
--- a/learning/prompts/documentation-lookup/04_basic_pcollections.md
+++ b/learning/prompts/documentation-lookup/04_basic_pcollections.md
@@ -1,7 +1,7 @@
Prompt:
-What is a PCollection in Apache Beam?
-Response:
+What is a `PCollection` in Apache Beam?
+Response:
A `PCollection` in Apache Beam is an unordered bag of elements. Each `PCollection` is a potentially distributed, homogeneous data set or data stream, and is owned by the specific `Pipeline` object for which it is created. It’s the primary data structure used in Apache Beam pipelines to handle large-scale data processing in batch and streaming modes.
The following example shows how to create a `PCollection` using the `Create` transform:
@@ -13,7 +13,7 @@ with beam.Pipeline() as pipeline:
pcollection = pipeline | beam.Create([...]) # Create a PCollection
```
-A `PCollection` can either be bounded or unbounded, making it versatile for different [data source types](https://beam.apache.org/documentation/basics/#pcollection). Bounded `PCollections` represent a finite data set, such as files or databases, ideal for batch processing. Unbounded `PCollections`, on the other hand, represent data streams that continuously grow over time, such as real-time event logs, suitable for stream processing.
+A `PCollection` can either be bounded or unbounded, making it versatile for different [data source types](https://beam.apache.org/documentation/basics/#pcollection). Bounded `PCollection` objects represent a finite data set, such as files or databases, ideal for batch processing. Unbounded `PCollection` objects, on the other hand, represent data streams that continuously grow over time, such as real-time event logs, suitable for stream processing.
Beam’s computational patterns and transforms are focused on situations where distributed data-parallel computation is required. Therefore, a `PCollection` has the following key characteristics:
- All elements must be of the same type (with support of structured types).
diff --git a/learning/prompts/documentation-lookup/05_basic_ptransforms.md b/learning/prompts/documentation-lookup/05_basic_ptransforms.md
index fcdc6481d69..44f17e76280 100644
--- a/learning/prompts/documentation-lookup/05_basic_ptransforms.md
+++ b/learning/prompts/documentation-lookup/05_basic_ptransforms.md
@@ -1,7 +1,7 @@
Prompt:
What is a PTransform in Apache Beam?
-Response:
+Response:
A [`PTransform`](https://beam.apache.org/documentation/programming-guide/#transforms) (or transform) represents a data processing operation, or a step, in a Beam pipeline. A transform is applied to zero or more `PCollection` objects and produces zero or more `PCollection` objects.
Transforms have the following key characteristics:
@@ -31,4 +31,4 @@ def SomeUserCode(element):
with beam.Pipeline() as pipeline:
input_collection = pipeline | beam.Create([...])
output_collection = input_collection | beam.ParDo(SomeUserCode())
-```
\ No newline at end of file
+```
diff --git a/learning/prompts/documentation-lookup/06_basic_schema.md b/learning/prompts/documentation-lookup/06_basic_schema.md
index cecbba17457..db53c04b838 100644
--- a/learning/prompts/documentation-lookup/06_basic_schema.md
+++ b/learning/prompts/documentation-lookup/06_basic_schema.md
@@ -1,8 +1,8 @@
Prompt:
-What are schemas in Apache Beam
-Response:
+What are schemas in Apache Beam?
- A [Schema in Apache Beam](https://beam.apache.org/documentation/programming-guide/#schemas) is a language-independent type definition for a `PCollection`. The schema defines elements of that `PCollection` as an ordered list of named fields.
+Response:
+A [Schema in Apache Beam](https://beam.apache.org/documentation/programming-guide/#schemas) is a language-independent type definition for a `PCollection`. The schema defines elements of that `PCollection` as an ordered list of named fields.
In many cases, the element type in a `PCollection` has a structure that can be introspected. Some examples are JSON, Protocol Buffer, Avro, and database row objects. All of these formats can be converted to Beam Schemas.
@@ -22,4 +22,4 @@ The following list includes key capabilities provided by schema transforms:
* type conversion
* enhanced ParDo functionality
-For more information about schemas and for Beam best practices when working with structured data, see [Schema Usage Patterns](https://beam.apache.org/documentation/patterns/schema/).
\ No newline at end of file
+For more information about schemas and the Apache Beam best practices when working with structured data, see [Schema Usage Patterns](https://beam.apache.org/documentation/patterns/schema/).
diff --git a/learning/prompts/documentation-lookup/07_basic_runners.md b/learning/prompts/documentation-lookup/07_basic_runners.md
index 982c06a4415..7345eac102f 100644
--- a/learning/prompts/documentation-lookup/07_basic_runners.md
+++ b/learning/prompts/documentation-lookup/07_basic_runners.md
@@ -1,11 +1,12 @@
Prompt:
What is a Runner in Apache Beam?
+
Response:
-Apache Beam Runners are the execution engines that run the pipelines. They translate or adapt the pipeline into a form that can be executed on a massively parallel big data processing system, such as Apache Flink, Apache Spark, Google Cloud Dataflow, and more.
+Apache Beam runners are the execution engines that run the pipelines. They translate or adapt the pipeline into a form that can be executed on a massively parallel big data processing system, such as Apache Flink, Apache Spark, Google Cloud Dataflow, and more.
-Choosing a runner is an important step in the pipeline development process. The runner you choose determines where and how your pipeline runs. For more information about available runners and their capabilities, see the [capabilities matrix](https://beam.apache.org/documentation/runners/capability-matrix/).
+Choosing a runner is an important step in the pipeline development process. The runner you choose determines where and how your pipeline runs. For more information about available runners and their capabilities, see the [Beam Capability Matrix](https://beam.apache.org/documentation/runners/capability-matrix/).
-To specify a runner, when you execute the pipeline, use the `--runner` flag . For example, to run the WordCount pipeline on Google Cloud Dataflow, you would run the following command:
+To specify a runner, when you execute the pipeline, use the `--runner` flag. For example, to run the WordCount pipeline on Google Cloud Dataflow, you would run the following command:
```python
python -m apache_beam.examples.wordcount \
@@ -16,8 +17,9 @@ python -m apache_beam.examples.wordcount \
--project PROJECT_ID \
--temp_location gs://STORAGE_BUCKET/tmp/
```
+
For more details about the WordCount example, see the [WordCount source code in GitHub](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount.py).
The [Direct Runner](https://beam.apache.org/documentation/runners/direct/) executes the pipeline on your local machine. It is helpful for testing and debugging purposes.
-For instructions about how to configure your development environment and runner, see the quickstarts for [Java](https://beam.apache.org/get-started/quickstart-java), [Python](https://beam.apache.org/get-started/quickstart-py) and [Go](https://beam.apache.org/get-started/quickstart-go).
\ No newline at end of file
+For instructions about how to configure your development environment and runner, see the quickstarts for [Java](https://beam.apache.org/get-started/quickstart-java), [Python](https://beam.apache.org/get-started/quickstart-py), and [Go](https://beam.apache.org/get-started/quickstart-go).
diff --git a/learning/prompts/documentation-lookup/08_basic_windowing.md b/learning/prompts/documentation-lookup/08_basic_windowing.md
index 7a9d7a100d2..8e116b2114c 100644
--- a/learning/prompts/documentation-lookup/08_basic_windowing.md
+++ b/learning/prompts/documentation-lookup/08_basic_windowing.md
@@ -1,13 +1,15 @@
Prompt:
-What is Windowing in Apache Beam?
+What is windowing in Apache Beam?
+
Response:
-In stream processing, windowing lets you to divide data streams into logical units for efficient and correct parallel processing.
+In stream processing, windowing allows you to divide data streams into logical units for efficient and correct parallel processing.
With an unbounded data set, collecting all of the elements is impossible, because new elements are constantly being added. In the Beam model, any `PCollection` (including unbounded `PCollection` objects) can be subdivided into [logical windows](https://beam.apache.org/documentation/programming-guide/#windowing-basics). Grouping transforms then consider the elements of each `PCollection` on a per-window basis.
-Because Beam's default windowing strategy is to assign each element to a single, global window, you must explicitly specify a [windowing function](https://beam.apache.org/documentation/programming-guide/#setting-your-pcollections-windowing-function) for your pipeline.
+Because Apache Beam's default windowing strategy is to assign each element to a single, global window, you must explicitly specify a [windowing function](https://beam.apache.org/documentation/programming-guide/#setting-your-pcollections-windowing-function) for your pipeline.
+
+The following code snippet shows how to divide a `PCollection` into 60-second windows:
-The following code snippet shows how to divide a `PCollection` into 60-second windows:
```python
from apache_beam import beam
from apache_beam import window
@@ -22,8 +24,8 @@ Beam provides a number of [built-in windowing functions](https://beam.apache.org
- Single Global Window
- Calendar-based Windows (not supported by the Beam SDK for Python or Go)
-You can also create your own custom windowing function [WindowFn](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/window.py).
+You can also create your own custom windowing function [`WindowFn`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/window.py).
-You also need to specify a [triggering strategy](https://beam.apache.org/documentation/programming-guide/#triggers) to determine when to emit the results of your pipeline’s windowed computations.
+You need to specify a [triggering strategy](https://beam.apache.org/documentation/programming-guide/#triggers) to determine when to emit the results of your pipeline’s windowed computations.
You can adjust the windowing strategy to allow for [late data](https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data), which is data that arrives after the watermark has passed the end of the window. You can also specify how to handle late data, such as discarding or adding it to the next window.
diff --git a/learning/prompts/documentation-lookup/09_basic_triggers.md b/learning/prompts/documentation-lookup/09_basic_triggers.md
index 69d7b7bd914..587effafa06 100644
--- a/learning/prompts/documentation-lookup/09_basic_triggers.md
+++ b/learning/prompts/documentation-lookup/09_basic_triggers.md
@@ -1,12 +1,12 @@
Prompt:
-What are Triggers in Apache Beam?
+What is a trigger in Apache Beam?
+
Response:
-Beam uses triggers to determine when to emit the aggregated results of each [window](https://beam.apache.org/documentation/programming-guide/#windowing), referred to as a pane.
+Apache Beam uses triggers to determine when to emit the aggregated results of each [window](https://beam.apache.org/documentation/programming-guide/#windowing), referred to as a pane.
Triggers provide two additional capabilities compared to [outputting at the end of a window](https://beam.apache.org/documentation/programming-guide/#default-trigger):
-
-- Allowing early results to be output before the end of the window
-- Allowing late data to be handled after the end of the window
+- Allowing early results to be output before the end of the window.
+- Allowing late data to be handled after the end of the window.
This feature lets you control the flow of your data and balance between completeness, latency, and cost.
@@ -21,12 +21,10 @@ You set the triggers for a `PCollection` by setting the `trigger` parameter of t
When a trigger fires, it emits the current contents of the window as a pane. Because a trigger can fire multiple times, the accumulation mode determines whether the system accumulates the window panes as the trigger fires, or discards them. This behavior is controlled by the [window accumulation mode](https://beam.apache.org/documentation/programming-guide/#window-accumulation-modes) parameter of the `WindowInto` transform.
-
Beam provides several [built-in triggers](https://beam.apache.org/documentation/programming-guide/#triggers) that you can use to determine when to emit the results of your pipeline's windowed computations:
- Event time triggers
- Processing time triggers
- Data-driven triggers
- Composite triggers
-One of the most useful trigger patterns is the [AfterWatermark](https://beam.apache.org/documentation/programming-guide/#composite-afterwatermark) trigger, which fires a single time when Beam estimates that all the data has arrived, such as when the watermark passes the end of the window.
-
+One of the most useful trigger patterns is the [`AfterWatermark`](https://beam.apache.org/documentation/programming-guide/#composite-afterwatermark) trigger, which fires a single time when Apache Beam estimates that all the data has arrived, such as when the watermark passes the end of the window.
diff --git a/learning/prompts/documentation-lookup/10_basic_metrics.md b/learning/prompts/documentation-lookup/10_basic_metrics.md
index 5899819acbf..9b33569c484 100644
--- a/learning/prompts/documentation-lookup/10_basic_metrics.md
+++ b/learning/prompts/documentation-lookup/10_basic_metrics.md
@@ -1,7 +1,8 @@
Prompt:
-What are Metrics in Apache Beam?
+What are the metrics in Apache Beam?
+
Response:
-[Metrics](https://beam.apache.org/documentation/programming-guide/#metrics) in Apache Beam provides a way to get insight into the current state of your pipeline, including during pipeline execution.
+In the Apache Beam model, [metrics](https://beam.apache.org/documentation/programming-guide/#metrics) provide insights into the current state of your pipeline, including during pipeline execution.
Metrics are named and scoped to a specific step in the pipeline. They can be created dynamically during pipeline execution. If a runner doesn't support some part of reporting metrics, the fallback behavior is to drop the metric updates rather than fail the pipeline.
@@ -11,6 +12,7 @@ Beam provides a number of [built-in metric types](https://beam.apache.org/docume
* Gauges
To declare a metric, use the `beam.metrics.Metrics` class. For example:
+
```python
self.words_counter = Metrics.counter(self.__class__, 'words')
self.word_lengths_counter = Metrics.counter(self.__class__, 'word_lengths')
@@ -20,5 +22,4 @@ self.empty_line_counter = Metrics.counter(self.__class__, 'empty_lines')
For implementation details, see the [WordCount example with metrics](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_with_metrics.py).
-
-You can export metrics to external sinks. Spark and Flink runners support REST HTTP and Graphite.
\ No newline at end of file
+You can export metrics to external sinks. Spark and Flink runners support REST HTTP and Graphite.
diff --git a/learning/prompts/documentation-lookup/11_basic_state.md b/learning/prompts/documentation-lookup/11_basic_state.md
index a4bf757a11b..c5235da2e17 100644
--- a/learning/prompts/documentation-lookup/11_basic_state.md
+++ b/learning/prompts/documentation-lookup/11_basic_state.md
@@ -1,20 +1,21 @@
Prompt:
-What is a State in Apache Beam?
+What is a state in Apache Beam?
+
Response:
-[State](https://beam.apache.org/documentation/basics/#state-and-timers) in Apache Beam is an API that allows for managing per-key state in element-wise operations like [ParDo](https://beam.apache.org/documentation/programming-guide/#pardo) or Map. It enables fine-grained control over aggregations, particularly in situations where [windowing](https://beam.apache.org/documentation/programming-guide/#windowing) and [triggers](https://beam.apache.org/documentation/programming-guide/#triggers [...]
+[State](https://beam.apache.org/documentation/basics/#state-and-timers) in Apache Beam is an API that allows for managing per-key state in element-wise operations like [`ParDo`](https://beam.apache.org/documentation/programming-guide/#pardo) or `Map`. It enables fine-grained control over aggregations, particularly in situations where [windowing](https://beam.apache.org/documentation/programming-guide/#windowing) and [triggers](https://beam.apache.org/documentation/programming-guide/#trig [...]
State is [scoped](https://beam.apache.org/documentation/programming-guide/#state-and-timers) to the current processing key within a window, meaning each key has its own state. State is particularly useful for complex aggregations across multiple elements.
-Beam provides several [types of state](https://beam.apache.org/documentation/programming-guide/#types-of-state) that can be used for different use cases, including scalar and collection values:
-- ValueState
-- BagState
-- SetState
-- MapState
-- OrderedListState
-- CombiningState
+Apache Beam provides several [types of state](https://beam.apache.org/documentation/programming-guide/#types-of-state) that can be used for different use cases, including scalar and collection values:
+- `ValueState`
+- `BagState`
+- `SetState`
+- `MapState`
+- `OrderedListState`
+- `CombiningState`
-The per-key state needs to be garbage collected, because the increasing size of the state can negatively impact performance. For more information about common garbage collection strategies, see [Garbage collecting state](https://beam.apache.org/documentation/programming-guide/#garbage-collecting-state) .
+The per-key state needs to be garbage-collected because the increasing size of the state can negatively impact performance. For more information about common garbage collection strategies, see [Garbage collecting state](https://beam.apache.org/documentation/programming-guide/#garbage-collecting-state).
For more information, read the community blog post about [Stateful Processing](https://beam.apache.org/blog/stateful-processing/).
-For additional examples of using State in Apache Beam see [State and Time examples](https://beam.apache.org/documentation/programming-guide/#state-timers-examples).
\ No newline at end of file
+For additional examples of using state in Apache Beam, see [State and Timers examples](https://beam.apache.org/documentation/programming-guide/#state-timers-examples).
diff --git a/learning/prompts/documentation-lookup/12_basic_timers.md b/learning/prompts/documentation-lookup/12_basic_timers.md
index 7a4e85bdf59..cd59f603138 100644
--- a/learning/prompts/documentation-lookup/12_basic_timers.md
+++ b/learning/prompts/documentation-lookup/12_basic_timers.md
@@ -1,11 +1,12 @@
Prompt:
-What is a Timer in Apache Beam?
+What is a timer in Apache Beam?
+
Response:
-In Apache Beam, a [Timer](https://beam.apache.org/documentation/basics/#state-and-timers) is a per-key timer callback API that enables delayed processing of data stored using the [State API](https://beam.apache.org/documentation/programming-guide/#state-and-timers)
+In Apache Beam, a [timer](https://beam.apache.org/documentation/basics/#state-and-timers) is a per-key timer callback API that enables delayed processing of data stored using the [State API](https://beam.apache.org/documentation/programming-guide/#state-and-timers).
Apache Beam provides two [types of timers](https://beam.apache.org/documentation/programming-guide/#timers): processing time timers and event time timers. Processing time timers are based on the system clock. Event time timers are based on the timestamps of the data elements.
-Beam also supports dynamically setting a timer tag using `TimerMap` in the Java SDK. This feature lets you set multiple different timers in a `DoFn`. You can set the timer tags to be chosen dynamically, for example, based on data in the input elements.
+Apache Beam also supports dynamically setting a timer tag using `TimerMap` in the Java SDK. This feature lets you set multiple different timers in a `DoFn`. You can set the timer tags to be chosen dynamically, for example, based on data in the input elements.
For more information, see the community blog post about [Timely Processing](https://beam.apache.org/blog/timely-processing/).
diff --git a/learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md b/learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md
index 41822a931fb..228127bf812 100644
--- a/learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md
+++ b/learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md
@@ -1,18 +1,18 @@
Prompt:
-What is Splittable DoFn in Apache Beam?
+What is a splittable `DoFn` in Apache Beam?
+
Response:
-A splittable `DoFn` (SDF) is a generalization of [`DoFn`](https://beam.apache.org/documentation/programming-guide/#pardo) that lets you process elements in a non-monolithic way. A splittable `DoFn` makes it easier to create complex, modular I/O connectors in Beam .
+A splittable `DoFn` (SDF) is a generalization of [`DoFn`](https://beam.apache.org/documentation/programming-guide/#pardo) that lets you process elements in a non-monolithic way. A splittable `DoFn` makes it easier to create complex, modular I/O connectors in Apache Beam.
When you apply a splittable `DoFn` to an element, the runner can split the element’s processing into smaller tasks. You can checkpoint the processing of an element, and you can split the remaining work to yield additional parallelism.
At a high level, an SDF is responsible for processing element and restriction pairs. A restriction represents a subset of work that would have been necessary to have been done when processing the element.
-Executing a [Splittable `DoFn`](https://beam.apache.org/documentation/programming-guide/#splittable-dofns) uses the following steps:
-1. Each element is paired with a restriction. For example, filename is paired with offset range representing the whole file).
+Executing a [splittable `DoFn`](https://beam.apache.org/documentation/programming-guide/#splittable-dofns) uses the following steps:
+1. Each element is paired with a restriction. For example, filename is paired with offset range representing the whole file.
2. Each element and restriction pair is split. For example, offset ranges are broken up into smaller pieces.
3. The runner redistributes the element and restriction pairs to several workers.
4. Element and restriction pairs are processed in parallel. For example, the file is read. Within this last step, the element and restriction pair can pause its own processing or be split into further element and restriction pairs.
-For an example, see the [Splittable DoFn module](https://tour.beam.apache.org/tour/python/splittable-dofn/splittable) in the Tour of Beam.
+For an example, see the [Splittable `DoFn` module](https://tour.beam.apache.org/tour/python/splittable-dofn/splittable) in the Tour of Beam.
For more information, see the [community blog post](https://beam.apache.org/blog/splittable-do-fn-is-available/).
-
diff --git a/learning/prompts/documentation-lookup/14_advanced_pipeline_patterns.md b/learning/prompts/documentation-lookup/14_advanced_pipeline_patterns.md
index f8cfaa86acc..b347b965761 100644
--- a/learning/prompts/documentation-lookup/14_advanced_pipeline_patterns.md
+++ b/learning/prompts/documentation-lookup/14_advanced_pipeline_patterns.md
@@ -1,7 +1,8 @@
Prompt:
What pipeline patterns exist in Apache Beam?
+
Response:
-Beam pipeline patterns are a set of best practices for building Beam pipelines. They are based on real-world Beam deployments and are designed to help you build resilient, flexible, and portable Beam pipelines.
+Beam pipeline patterns are a set of best practices for building Apache Beam pipelines. They are based on real-world Beam deployments and are designed to help you build resilient, flexible, and portable pipelines.
The following list includes some of the most common pipeline patterns:
- [File processing patterns](https://beam.apache.org/documentation/patterns/file-processing/)
@@ -10,4 +11,4 @@ The following list includes some of the most common pipeline patterns:
- [Windowing patterns](https://beam.apache.org/documentation/patterns/custom-windows/)
- [State and Timers patterns](https://beam.apache.org/documentation/patterns/grouping-elements-for-efficient-external-service-calls/)
-For more information, see the [Pipeline Patterns](https://beam.apache.org/documentation/patterns/overview/) section of the Beam programming guide.
+For more information, see the [Pipeline Patterns](https://beam.apache.org/documentation/patterns/overview/) section of the Beam Programming Guide.
diff --git a/learning/prompts/documentation-lookup/15_advanced_xlang.md b/learning/prompts/documentation-lookup/15_advanced_xlang.md
index f7df6785fab..6f35a43d320 100644
--- a/learning/prompts/documentation-lookup/15_advanced_xlang.md
+++ b/learning/prompts/documentation-lookup/15_advanced_xlang.md
@@ -1,15 +1,13 @@
Prompt:
What is a multi-language pipeline in Apache Beam?
+
Response:
-Beam lets you combine transforms written in any supported SDK language (currently, [Java](https://beam.apache.org/documentation/programming-guide/#1311-creating-cross-language-java-transforms) and [Python](https://beam.apache.org/documentation/programming-guide/#1312-creating-cross-language-python-transforms)) and use them in one multi-language pipeline. For example, a multi-language pipeline might read from a Python source, process the data using a Java transform, and write the data to [...]
+Apache Beam lets you combine transforms written in any supported SDK language (currently, [Java](https://beam.apache.org/documentation/programming-guide/#1311-creating-cross-language-java-transforms) and [Python](https://beam.apache.org/documentation/programming-guide/#1312-creating-cross-language-python-transforms)) and use them in one multi-language pipeline. For example, a multi-language pipeline might read from a Python source, process the data using a Java transform, and write the d [...]
For example, you can use the [Apache Kafka connector](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py) and [SQL transform](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/sql.py) from the Java SDK in Python pipelines.
To learn how to create a multi-language pipeline, see the quickstart examples for [Java](https://beam.apache.org/documentation/sdks/java-multi-language-pipelines) and [Python](https://beam.apache.org/documentation/sdks/python-multi-language-pipelines).
-Depending on the SDK language of the pipeline, you can use a high-level SDK-wrapper class or a low-level transform class to access a cross-language transform. For more information, see [Using cross-language transforms](https://beam.apache.org/documentation/programming-guide/#use-x-lang-transforms).
-
-To develop a cross-language transform, you need to define a Uniform Resourse Name(URN) for registering the transform with an expansion service. For more information, see [Defining a URN](https://beam.apache.org/documentation/programming-guide/#1314-defining-a-urn).
-
-
+Depending on the SDK language of the pipeline, you can use a high-level SDK wrapper class or a low-level transform class to access a cross-language transform. For more information, see [Using cross-language transforms](https://beam.apache.org/documentation/programming-guide/#use-x-lang-transforms).
+To develop a cross-language transform, you need to define a Uniform Resource Name(URN) for registering the transform with an expansion service. For more information, see [Defining a URN](https://beam.apache.org/documentation/programming-guide/#1314-defining-a-urn).
diff --git a/learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md b/learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md
index c59a177c252..3189483739a 100644
--- a/learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md
+++ b/learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md
@@ -1,9 +1,8 @@
Prompt:
What is a pipeline development lifecycle in Apache Beam?
-Response:
+Response:
The Apache Beam pipeline development lifecycle is an iterative process that usually involves the following steps:
-
- Design your pipeline.
- Develop your pipeline code.
- Test your pipeline.
@@ -19,7 +18,6 @@ To design a pipeline, you need answers to the following questions:
Beam documentation has more information about [pipeline design](https://beam.apache.org/documentation/pipelines/design-your-pipeline/) and about common [pipeline patterns](https://beam.apache.org/documentation/patterns/overview/).
-
An Apache Beam program expresses a data processing pipeline, from start to finish. To construct a pipeline using the classes in the Apache Beam SDKs, your program needs to perform the following steps:
- Create a `Pipeline` object.
@@ -30,7 +28,6 @@ An Apache Beam program expresses a data processing pipeline, from start to finis
The Apache Beam documentation has more information about [developing](https://beam.apache.org/documentation/programming-guide/) and [executing](https://beam.apache.org/documentation/pipelines/create-your-pipeline/) pipelines.
-Testing pipelines is a particularly important step in developing an effective data processing solution. The indirect nature of the Beam model, in which your user code constructs a pipeline graph to be executed remotely, can make debugging-failed runs difficult. For more information about pipeline testing strategies, see [Test Your Pipeline](https://beam.apache.org/documentation/pipelines/test-your-pipeline/).
+Testing pipelines is a particularly important step in developing an effective data processing solution. The indirect nature of the Beam model, in which your user code constructs a pipeline graph to be executed remotely, can make debugging failed runs difficult. For more information about pipeline testing strategies, see [Test Your Pipeline](https://beam.apache.org/documentation/pipelines/test-your-pipeline/).
-Choosing a [runner](https://beam.apache.org/documentation/#choosing-a-runner) is a crucial step in deploying your pipeline. The runner you choose determines where and how your pipeline executes.
-For more information about pipeline deployment, see [Container environments](https://beam.apache.org/documentation/runtime/environments/).
+Choosing a [runner](https://beam.apache.org/documentation/#choosing-a-runner) is a crucial step in deploying your pipeline. The runner you choose determines where and how your pipeline executes. For more information about pipeline deployment, see [Container environments](https://beam.apache.org/documentation/runtime/environments/).
diff --git a/learning/prompts/documentation-lookup/17_advanced_ai_ml.md b/learning/prompts/documentation-lookup/17_advanced_ai_ml.md
index f758c7e5c27..09475ad796d 100644
--- a/learning/prompts/documentation-lookup/17_advanced_ai_ml.md
+++ b/learning/prompts/documentation-lookup/17_advanced_ai_ml.md
@@ -1,5 +1,6 @@
Prompt:
-What are AI and ML capabilities in Apache Beam?
+What are the AI and ML capabilities in Apache Beam?
+
Response:
Apache Beam has several built-in [AI and ML capabilities](https://beam.apache.org/documentation/ml/overview/) that enable you to:
- Process large datasets for both preprocessing and model inference.
@@ -10,7 +11,7 @@ For common AI platform integration patterns in Apache Beam, see [AI Platform int
The recommended way to implement inference in Apache Beam is by using the [RunInference API](https://beam.apache.org/documentation/sdks/python-machine-learning/). For more information about how to use RunInference for PyTorch, scikit-learn, and TensorFlow, see the [Use RunInference in Apache Beam](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb) example in GitHub.
-Using pre-trained models in Apache Beam is also supported with [PyTorch](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch.ipynb), [Scikit-learn](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_sklearn.ipynb), and [Tensorflow](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_tensorflow.ipynb). Running inference on [custom models](https://beam.apache.org/documentation/ml/about [...]
+Using pre-trained models in Apache Beam is also supported with [PyTorch](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch.ipynb), [scikit-learn](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_sklearn.ipynb), and [TensorFlow](https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_tensorflow.ipynb). Running inference on [custom models](https://beam.apache.org/documentation/ml/about [...]
Apache Beam also supports automatically updating the model being used with the `RunInference PTransform` in streaming pipelines without stopping the pipeline. The feature lets you avoid downtime downtime. For more information, see [Automatic model refresh](https://beam.apache.org/documentation/ml/about-ml/#automatic-model-refresh).
For more information about using machine learning models with Apache Beam, see [Running ML models now easier with new Dataflow ML innovations on Apache Beam](https://cloud.google.com/blog/products/ai-machine-learning/dataflow-ml-innovations-on-apache-beam/).
diff --git a/learning/prompts/documentation-lookup/18_io_kafka.md b/learning/prompts/documentation-lookup/18_io_kafka.md
index 28c22508109..b5727fa8c6f 100644
--- a/learning/prompts/documentation-lookup/18_io_kafka.md
+++ b/learning/prompts/documentation-lookup/18_io_kafka.md
@@ -3,15 +3,16 @@ Is Apache Kafka supported in Apache Beam?
Response:
[Apache Kafka](https://kafka.apache.org/) is a distributed streaming platform that is used for building real-time data pipelines and streaming applications. It is horizontally scalable, fault-tolerant, and fast.
-Apache Beam has a built-in KafkaIO connector for Apache Kafka that lets you read data from and write data to Kafka. Apache Kafka is supported in both batch pipelines and streaming pipelines, and as a source and a sink. For more information, see the KafkaIO connector documentation:
+
+Apache Beam has a built-in KafkaIO connector for Apache Kafka that enables Apache Beam pipelines to read data from and write data to Kafka. Apache Kafka is supported in both batch and streaming pipelines and can serve as both a source and a sink. For more information, refer to the KafkaIO connector documentation in the following languages:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html)
* [Python (using cross-language transforms)](https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html)
* [Go (using cross-language transforms)](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/kafkaio)
* [Typescript (using cross-language transforms)](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/kafka.ts)
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with Apache Kafka and Apache Beam.
+To get started with Apache Kafka and Apache Beam, refer to the [Dataflow Cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
-Apache Beam pipeline code for reading data from a Kafka topic might look like the following example:
+Here is an example of Apache Beam pipeline code for reading data from a Kafka topic:
```python
with beam.Pipeline(options=options) as p:
@@ -21,7 +22,7 @@ with beam.Pipeline(options=options) as p:
with_metadata=False)
| "Log Data" >> Map(logging.info))
```
-This code reads data from a Kafka topic `my_topic` on `localhost:9092` and logs the data to the console.
-The Apache Beam repository has a [KafkaIO example](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/kafkataxi/kafka_taxi.py) that demonstrates how to use the KafkaIO connector.
+This code reads data from a Kafka topic `my_topic` on `localhost:9092` and logs the data to the console.
+For a detailed demonstration of using the KafkaIO connector, refer to the [KafkaIO example](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/kafkataxi/kafka_taxi.py) in the Apache Beam GitHub repository.
diff --git a/learning/prompts/documentation-lookup/19_io_pubsub.md b/learning/prompts/documentation-lookup/19_io_pubsub.md
index c1447459a83..e6f2d99d8cf 100644
--- a/learning/prompts/documentation-lookup/19_io_pubsub.md
+++ b/learning/prompts/documentation-lookup/19_io_pubsub.md
@@ -1,19 +1,18 @@
Prompt:
Is Pub/Sub supported in Apache Beam?
-Response:
-[Pub/Sub](https://cloud.google.com/pubsub) is a [Google Cloud](https://cloud.google.com/) service that provides a simple, reliable, scalable, and secure real-time messaging service for sending and receiving messages between independent applications. Apache Beam provides a PubSubIO connector that allows you to read and write messages from and to Pub/Sub.
-Pub/Sub is currently supported only in streaming pipelines.
+Response:
+[Pub/Sub](https://cloud.google.com/pubsub) is a [Google Cloud](https://cloud.google.com/) service that provides a simple, reliable, scalable, and secure real-time messaging service for sending and receiving messages between independent applications. Apache Beam provides a PubSubIO connector that allows you to read and write messages from and to Pub/Sub. Pub/Sub is currently supported only in streaming pipelines.
Pub/Sub is supported in the following Beam SDKs:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html)
* [Python](https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html)
* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/pubsubio)
-* [Typescript](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/pubsub.ts) through X Language.
+* [Typescript](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/pubsub.ts) (through X Language)
-The [Dataflow Cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) will help you to get started with Pub/Sub and Apache Beam. See here for [read](https://github.com/GoogleCloudPlatform/dataflow-cookbook/blob/main/Python/pubsub/read_pubsub_multiple.py) and [write](https://github.com/GoogleCloudPlatform/dataflow-cookbook/blob/main/Python/pubsub/write_pubsub.py) examples in Python.
+To get started with Pub/Sub and Apache Beam, refer to the [Dataflow Cookbook GitHub repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook). The repository provides Python code examples that [read](https://github.com/GoogleCloudPlatform/dataflow-cookbook/blob/main/Python/pubsub/read_pubsub_multiple.py) and [write](https://github.com/GoogleCloudPlatform/dataflow-cookbook/blob/main/Python/pubsub/write_pubsub.py) data from and to Pub/Sub.
-Apache Beam pipeline code for reading data from Pub/Sub topic and writing it back to another topic might look like the following example:
+Here is an example of the Apache Beam pipeline code for reading data from a Pub/Sub topic and writing it to another topic:
```python
from apache_beam.io import ReadFromPubSub
@@ -23,5 +22,3 @@ from apache_beam.io import WriteToPubSub
(p | "Read from Pub/Sub" >> ReadFromPubSub(topic="input_topic")
| "Write to Pub/Sub" >> WriteToPubSub(topic="output_topic"))
```
-
-
diff --git a/learning/prompts/documentation-lookup/20_io_biguery.md b/learning/prompts/documentation-lookup/20_io_biguery.md
index 5e29f17590b..a9df1ddb846 100644
--- a/learning/prompts/documentation-lookup/20_io_biguery.md
+++ b/learning/prompts/documentation-lookup/20_io_biguery.md
@@ -1,17 +1,18 @@
Prompt:
Is BigQuery supported in Apache Beam?
+
Response:
-Yes, Apache Beam supports BigQuery. [BigQuery](https://cloud.google.com/bigquery) is a serverless and cost-effective enterprise data warehouse offered by [Google Cloud](https://cloud.google.com/). Apache Beam provides a `BigQueryIO` connector to read and write data to and from BigQuery. The `BigQueryIO` connector supports both batch and streaming pipelines.
+Yes, Apache Beam supports BigQuery. [BigQuery](https://cloud.google.com/bigquery) is a serverless and cost-effective enterprise data warehouse offered by [Google Cloud](https://cloud.google.com/). Apache Beam provides a BigQueryIO connector to read and write data from and to BigQuery. The BigQueryIO connector supports both batch and streaming pipelines.
-The following Apache Beam SDKs support the `BigQueryIO` connector:
+The following Apache Beam SDKs support the BigQueryIO connector:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html)
* [Python](https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.bigquery.html)
-* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/bigqueryio) native and though [X Language](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/bigqueryio)
-* [Typescript](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/bigqueryio.ts) through X Language
+* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/bigqueryio) (native and through [X Language](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/bigqueryio))
+* [Typescript](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/bigqueryio.ts) (through X Language)
-To read data from BigQuery, use `ReadFromBigQuery`. Apache Beam can read data directly from a BigQuery table or using a SQL query. The default mode is to return table rows read from a BigQuery source as dictionaries. Built-in `TableRow` objects can also be returned.
+To read data from BigQuery, use the `ReadFromBigQuery` function. Apache Beam can read data directly from a BigQuery table or using an SQL query. The default mode is to return table rows read from a BigQuery source as dictionaries. Built-in `TableRow` objects can also be returned.
-Apache Beam pipeline code for reading from BigQuery might look like the following example:
+Here is an example of Apache Beam pipeline code for reading from BigQuery:
```python
from apache_beam.io.gcp.bigquery import ReadFromBigQuery
@@ -21,9 +22,9 @@ with beam.Pipeline(options=options) as p:
lines_table = p | 'Read' >> ReadFromBigQuery(table=table)
# read from a query
lines_query = p | 'Read' >> ReadFromBigQuery(query="SELECT * FROM table")
-
```
-Apache Beam pipeline code for writing from BigQuery might look like the following example:
+
+Here is an example of Apache Beam pipeline code for writing to BigQuery:
```python
from apache_beam.io.gcp.bigquery import WriteToBigQuery
@@ -36,4 +37,5 @@ with beam.Pipeline(options=options) as p:
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)
```
+
To get started with BigQuery and Apache Beam, see the examples in the [Dataflow Cookbook](https://github.com/GoogleCloudPlatform/dataflow-cookbook) GitHub repository. For Python read examples, see the [read_table_bigquery.py](https://github.com/GoogleCloudPlatform/dataflow-cookbook/blob/main/Python/bigquery/read_table_bigquery.py) example. For Python write examples, see the [write_bigquery.py](https://github.com/GoogleCloudPlatform/dataflow-cookbook/blob/main/Python/bigquery/write_bigque [...]
diff --git a/learning/prompts/documentation-lookup/21_io_bigtable.md b/learning/prompts/documentation-lookup/21_io_bigtable.md
index 60f564256f4..29c5def98ef 100644
--- a/learning/prompts/documentation-lookup/21_io_bigtable.md
+++ b/learning/prompts/documentation-lookup/21_io_bigtable.md
@@ -1,5 +1,5 @@
Prompt:
-Is Cloud BigTable supported in Apache Beam?
+Is Cloud Bigtable supported in Apache Beam?
Response:
[Cloud Bigtable](https://cloud.google.com/bigtable) is a [Google Cloud](https://cloud.google.com/) service that provides enterprise-grade NoSQL database service with single-digit millisecond latency, limitless scale, and 99.999% availability for large analytical and operational workloads.
@@ -11,9 +11,9 @@ Apache Beam has a built-in connector for Cloud Bigtable that lets you read data
* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/bigtableio)
* [Go (using cross-language transforms)](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/bigtableio)
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with Cloud Bigtable and Apache Beam.
+To get started with Cloud Bigtable and Apache Beam, refer to the [Dataflow Cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
-Apache Beam pipeline code for reading data from a Bigtable table might look like the following example:
+Here is an example of Apache Beam pipeline code for reading data from a Bigtable table:
```python
with beam.Pipeline(options=options) as p:
@@ -25,4 +25,5 @@ output = (p | "Read from Bigtable" >> ReadFromBigtable(
| "Extract cells" >> beam.Map(lambda row: row._cells)
| "Log Data" >> Map(logging.info))
```
-For performance benchmarks of the Bigtable IO connector, see [Bigtable performance](https://beam.apache.org/performance/bigtable/).
\ No newline at end of file
+
+For performance benchmarks of the Bigtable IO connector, see [Bigtable performance](https://beam.apache.org/performance/bigtable/).
diff --git a/learning/prompts/documentation-lookup/22_io_spanner.md b/learning/prompts/documentation-lookup/22_io_spanner.md
index b7a4cfb18aa..3b6ec417d84 100644
--- a/learning/prompts/documentation-lookup/22_io_spanner.md
+++ b/learning/prompts/documentation-lookup/22_io_spanner.md
@@ -4,14 +4,14 @@ Is Cloud Spanner supported in Apache Beam?
Response:
[Cloud Spanner](https://cloud.google.com/spanner) is a [Google Cloud](https://cloud.google.com/) service that provides a fully managed, mission-critical, relational database service that offers transactional consistency at global scale, schemas, SQL (ANSI 2011 with extensions), and automatic, synchronous replication for high availability.
-Apache Beam has a built-in SpannerIO connector for Cloud Spanner that lets you read data from and write data to Cloud Spanner. Cloud Spanner is supported in both batch pipelines and streaming pipelines, and as a source and a sink. For more information, see the SpannerIO connector documentation:
+Apache Beam has a built-in SpannerIO connector for Cloud Spanner that enables Apache Beam pipelines to read data from and write data to Cloud Spanner. Cloud Spanner is supported in both batch pipelines and streaming pipelines and can serve both as a source and a sink. For more information, see the SpannerIO connector documentation:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.html)
* [Python (using cross-language transforms)](https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.spanner.html)
* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/spannerio)
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with Cloud Spanner and Apache Beam.
+To get started with Cloud Spanner and Apache Beam, refer to the [Dataflow Cookbook GitHub repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
-Apache Beam pipeline code for reading data from a Cloud Spanner table using a SQL query might look like the following example:
+Here is an example of Apache Beam pipeline code for reading data from a Cloud Spanner table using an SQL query:
```python
class ExampleRow(NamedTuple):
diff --git a/learning/prompts/documentation-lookup/23_io_gcs.md b/learning/prompts/documentation-lookup/23_io_gcs.md
index e67d6fb02cc..c9e7dde6241 100644
--- a/learning/prompts/documentation-lookup/23_io_gcs.md
+++ b/learning/prompts/documentation-lookup/23_io_gcs.md
@@ -1,23 +1,22 @@
Prompt:
-Is Google Cloud Storage file system supported in Apache Beam?
+Is the Google Cloud Storage file system supported in Apache Beam?
Response:
[Google Cloud Storage](https://cloud.google.com/storage) is an object storage service for storing and accessing data on Google Cloud. Apache Beam supports both working with the Google Cloud Storage file system and reading data from and writing data to Google Cloud Storage buckets.
-Apache Beam has a built-in TextIO connector for reading data from and writing data to Google Cloud Storage buckets. For more information, see the TextIO connector documentation:
+Apache Beam has a built-in TextIO connector for reading data from and writing data to Google Cloud Storage buckets. For more information, see the TextIO connector documentation for your programming language of choice:
* [Java connector](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html)
* [Python connector](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html)
* [Go connector](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio)
* [Typescript connector (using cross-language transforms)](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/textio.ts)
-Reading and writing data in different file formats in Google Cloud Storage buckets is also supported in the AvroIO, XMLIO, TFRecordIO, and ParquetIO connectors.
+The Apache Beam AvroIO, XMLIO, TFRecordIO, and ParquetIO connectors also support reading data from and writing data to different file formats within Google Cloud Storage buckets.
-Specify the Google Cloud Storage path using the following format: `gs://<bucket>/<path>`. For example, `gs://my-bucket/my-file.txt`.
-You can use wildcards in read transforms and write transforms to read data from and write data to multiple files. For example, `gs://my-bucket/my-files-*.txt`.
+You can specify the Google Cloud Storage file path using the following format: `gs://<bucket>/<path>`. For example, `gs://my-bucket/my-file.txt`.
-Apache Beam also has support for the Google Cloud Storage file system that lets you work with the Google Cloud Storage file system directly. For example, you can use the Google Cloud Storage file system to check whether a file exists, list files in a directory, or delete a file. The Google Cloud Storage file system is supported in the following languages:
+You can use wildcards in read and write transforms to process data from multiple files or write data to multiple files simultaneously. For instance, you can specify `gs://my-bucket/my-files-*.txt` to read or write to files with names matching the specified pattern.
+
+Apache Beam also allows pipelines to interact directly with the Google Cloud Storage file system. For instance, you can use this file system to verify the existence of a file, retrieve a list of files in a directory, or delete a file. The Google Cloud Storage file system is supported in the following languages:
* [Java file system](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html)
* [Python file system](https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html)
* [Go file system](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/gcs)
-
-
diff --git a/learning/prompts/documentation-lookup/24_io_json.md b/learning/prompts/documentation-lookup/24_io_json.md
index b1c2d89d742..49f5628b599 100644
--- a/learning/prompts/documentation-lookup/24_io_json.md
+++ b/learning/prompts/documentation-lookup/24_io_json.md
@@ -2,15 +2,15 @@ Prompt:
Is JSON supported in Apache Beam?
Response:
-[JSON](https://www.json.org/) is a data format for storing and exchanging data. Apache Beam supports reading and writing JSON files using the `ReadFromJson` and `WriteToJson` transforms in the `TextIO` module. For more information, see the TextIO connector documentation:
+[JSON](https://www.json.org/) is a data format for storing and exchanging data. Apache Beam supports reading data from and writing data to JSON files using the `ReadFromJson` and `WriteToJson` transforms within the `TextIO` module. For more information, see the TextIO connector documentation in the following languages:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html)
* [Python](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html)
* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio)
* [Typescript](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/textio.ts)
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with JSON and Apache Beam.
+To get started with JSON and Apache Beam, refer to the [Dataflow Cookbook GitHub repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
-Apache Beam pipeline code for reading data from a JSON file might look like the following example:
+Here is an example of Apache Beam pipeline code for reading data from a JSON file:
```python
class JsonOptions(PipelineOptions):
@@ -29,4 +29,4 @@ with beam.Pipeline(options=options) as p:
lines=False
)
| "Log Data" >> Map(logging.info))
-```
\ No newline at end of file
+```
diff --git a/learning/prompts/documentation-lookup/25_io_csv.md b/learning/prompts/documentation-lookup/25_io_csv.md
index 8803fa28c79..3ce5ddd2b5b 100644
--- a/learning/prompts/documentation-lookup/25_io_csv.md
+++ b/learning/prompts/documentation-lookup/25_io_csv.md
@@ -8,9 +8,10 @@ Response:
* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio)
* [Typescript](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/textio.ts)
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with CSV and Apache Beam.
+To get started with CSV and Apache Beam, refer to the [Dataflow Cookbook GitHub repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
+
+Here's an example of Apache Beam pipeline code for reading data from a CSV file:
-Apache Beam pipeline code for reading data from a CSV file might look like the following example:
```python
class CsvOptions(PipelineOptions):
@classmethod
@@ -29,4 +30,4 @@ with beam.Pipeline(options=options) as p:
)
| "Log Data" >> Map(logging.info))
-```
\ No newline at end of file
+```
diff --git a/learning/prompts/documentation-lookup/26_io_avro.md b/learning/prompts/documentation-lookup/26_io_avro.md
index 4161c47710c..a5199eadb38 100644
--- a/learning/prompts/documentation-lookup/26_io_avro.md
+++ b/learning/prompts/documentation-lookup/26_io_avro.md
@@ -2,17 +2,17 @@ Prompt:
Is Apache Avro format supported in Apache Beam?
Response:
-[Apache Avro](https://avro.apache.org/) is a data format for storing and exchanging data. Apache Beam supports reading from and writing to Avro files using the `ReadFromAvro` and `WriteToAvro` transforms in the `AvroIO` module. For more information, see the AvroIO connector documentation:
+[Apache Avro](https://avro.apache.org/) is a data format for storing and exchanging data. Apache Beam supports reading from and writing to Avro files using the `ReadFromAvro` and `WriteToAvro` transforms in the `AvroIO` module. For more information, see the AvroIO connector documentation for the following languages:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/avro/io/AvroIO.html)
* [Python](https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html)
* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/avroio)
* [Typescript (using cross-language transforms)](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/avroio.ts)
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with Avro and Apache Beam.
+To get started with Avro and Apache Beam, refer to the [Dataflow Cookbook GitHub repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
-Apache Beam pipeline code for reading data from an Avro file might look like the following example:
-```python
+Here is an example of Apache Beam pipeline code for reading data from an Avro file:
+```python
class ReadAvroOptions(PipelineOptions):
@classmethod
diff --git a/learning/prompts/documentation-lookup/27_io_parquet.md b/learning/prompts/documentation-lookup/27_io_parquet.md
index 34ec6892421..543569b110f 100644
--- a/learning/prompts/documentation-lookup/27_io_parquet.md
+++ b/learning/prompts/documentation-lookup/27_io_parquet.md
@@ -2,16 +2,16 @@ Prompt:
Is Apache Parquet format supported in Apache Beam?
Response:
-[Apache Parquet](https://parquet.apache.org/) is a data format for storing and exchanging data. Apache Beam supports reading from and writing to Parquet files using the `ReadFromParquet` and `WriteToParquet` transforms in the `ParquetIO` module. For more information, see the ParquetIO connector documentation:
+[Apache Parquet](https://parquet.apache.org/) is a data format for storing and exchanging data. Apache Beam supports reading from and writing to Parquet files using the `ReadFromParquet` and `WriteToParquet` transforms in the `ParquetIO` module. For more information, see the ParquetIO connector documentation for the following languages:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html)
* [Python](https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html)
* [Go](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/parquetio)
* [Typescript (using cross-language transforms)](https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/io/parquetio.ts)
+To get started with Avro and Apache Beam, refer to the [Dataflow Cookbook GitHub repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with Avro and Apache Beam.
+Here is an example of Apache Beam pipeline code for reading data from a Parquet file:
-Apache Beam pipeline code for reading data from a Parquet file might look like the following example:
```python
class ReadParquetOptions(PipelineOptions):
diff --git a/learning/prompts/documentation-lookup/28_io_jdbc.md b/learning/prompts/documentation-lookup/28_io_jdbc.md
index 13d6502c5bc..0a67d144fc8 100644
--- a/learning/prompts/documentation-lookup/28_io_jdbc.md
+++ b/learning/prompts/documentation-lookup/28_io_jdbc.md
@@ -2,20 +2,20 @@ Prompt:
Is Oracle Java Database Connectivity (JDBC) supported in Apache Beam?
Response:
-The [Oracle Java Database Connectivity (JDBC) API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/) is a Java API for connecting to relational databases. Apache Beam supports reading and writing data with JDBC using the `ReadFromJdbc` and `WriteToJdbc` transforms in the `JdbcIO` module. For more information, see the JdbcIO connector documentation:
+The [Oracle Java Database Connectivity (JDBC) API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/) is a Java API for connecting to relational databases. Apache Beam supports reading and writing data with JDBC using the `ReadFromJdbc` and `WriteToJdbc` transforms in the `JdbcIO` module. For more information, see the JdbcIO connector documentation for the following languages:
* [Java](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jdbc/JdbcIO.html)
* [Python (using cross-language transforms)](https://beam.apache.org/releases/pydoc/current/apache_beam.io.jdbc.html)
* [Go (using cross-language transforms)](https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/jdbcio)
-The [Dataflow cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook) can help you to get started with JDBC and Apache Beam.
+To get started with JDBC and Apache Beam, refer to the [Dataflow Cookbook repository](https://github.com/GoogleCloudPlatform/dataflow-cookbook).
+
+Here is an example of Apache Beam pipeline code for reading data with JDBC:
-Apache Beam pipeline code for reading data with JDBC might look like the following example:
```python
class ExampleRow(NamedTuple):
id: int
name: str
-
class ReadJdbcOptions(PipelineOptions):
@classmethod
@@ -49,4 +49,3 @@ with beam.Pipeline(options=options) as p:
query=options.query)
| Map(logging.info))
```
-
diff --git a/learning/prompts/documentation-lookup/30_advanced_additional_outputs.md b/learning/prompts/documentation-lookup/30_advanced_additional_outputs.md
index 9ca5a9e6410..840286553dd 100644
--- a/learning/prompts/documentation-lookup/30_advanced_additional_outputs.md
+++ b/learning/prompts/documentation-lookup/30_advanced_additional_outputs.md
@@ -2,15 +2,15 @@ Prompt:
What are additional outputs in Apache Beam?
Response:
-In Apache Beam, additional outputs, also known as tagged or side outputs, refer to one or multiple extra [`PCollection`s](https://beam.apache.org/documentation/basics/#pcollection) produced by a single [`ParDo`](https://beam.apache.org/documentation/programming-guide/#pardo) transform in addition to the main output `PCollection`. The `ParDo` transform has the flexibility to produce any number of additional output `PCollection`s and return them bundled together with the main output `PColl [...]
+In Apache Beam, additional outputs, also known as tagged or side outputs, refer to one or multiple extra [`PCollection`](https://beam.apache.org/documentation/basics/#pcollection) objects produced by a single [`ParDo`](https://beam.apache.org/documentation/programming-guide/#pardo) transform in addition to the main output `PCollection`. The `ParDo` transform has the flexibility to produce any number of additional output `PCollection` objects and return them bundled together with the main [...]
-Additional outputs serve as a mechanism to implement [pipeline branching](https://beam.apache.org/documentation/pipelines/design-your-pipeline/#branching-pcollections). You can use them when there is a need to split the output of a single transform into several `PCollection`s or produce outputs in different formats. Additional outputs become particularly beneficial when a transform’s computation per element is time-consuming because they enable transforms to process each element in the i [...]
+Additional outputs serve as a mechanism to implement [pipeline branching](https://beam.apache.org/documentation/pipelines/design-your-pipeline/#branching-pcollections). You can use them when there is a need to split the output of a single transform into several `PCollection` objects or produce outputs in different formats. Additional outputs become particularly beneficial when a transform’s computation per element is time-consuming because they enable transforms to process each element i [...]
Producing additional outputs requires [tagging](https://beam.apache.org/documentation/programming-guide/#output-tags) each output `PCollection` with a unique identifier, which is then used to [emit](https://beam.apache.org/documentation/programming-guide/#multiple-outputs-dofn) elements to the corresponding output.
-In the Apache Beam Java SDK, you can implement additional outputs by creating a `TupleTag` object to identify each collection produced by the `ParDo` transform. After specifying the `TupleTag`s for each of the outputs, the tags are passed to the `ParDo` using the `.withOutputTags` method. You can find a sample Apache Beam Java pipeline that applies one transform to output two `PCollection`s in the [Branching `PCollection`s](https://beam.apache.org/documentation/pipelines/design-your-pipe [...]
+In the Apache Beam Java SDK, you can implement additional outputs by creating a `TupleTag` object to identify each collection produced by the `ParDo` transform. After specifying the `TupleTag` objects for each of the outputs, the tags are passed to the `ParDo` using the `.withOutputTags` method. You can find a sample Apache Beam Java pipeline that applies one transform to output two `PCollection` objects in the [Branching `PCollection`s](https://beam.apache.org/documentation/pipelines/de [...]
-The following Java code implements two additional output `PCollection`s for string and integer values in addition to the main output `PCollection` of strings:
+The following Java code implements two additional output `PCollection` objects for string and integer values in addition to the main output `PCollection` of strings:
```java
// Input PCollection that contains strings.
@@ -55,7 +55,7 @@ public void processElement(@Element String word, MultiOutputReceiver out) {
In the Apache Beam Python SDK, you can implement additional outputs by invoking the `with_outputs()` method on the `ParDo` and specifying the expected tags for the multiple outputs.
-The following Python code demonstrates how to implement additional outputs for a `ParDo` transform that outputs two `PCollection`s of strings and integers in addition to the main output `PCollection` of strings:
+The following Python code demonstrates how to implement additional outputs for a `ParDo` transform that outputs two `PCollection` objects of strings and integers in addition to the main output `PCollection` of strings:
```python
class SplitLinesToWordsFn(beam.DoFn):
@@ -78,7 +78,7 @@ class SplitLinesToWordsFn(beam.DoFn):
yield word
```
-The method returns a `DoOutputsTuple` object, with the specified tags serving as attributes that provide `ParDo` with access to the corresponding output `PCollection`s.
+The method returns a `DoOutputsTuple` object, with the specified tags serving as attributes that provide `ParDo` with access to the corresponding output `PCollection` objects.
```python
with beam.Pipeline(options=pipeline_options) as p:
diff --git a/learning/prompts/documentation-lookup/31_advanced_batched_dofn.md b/learning/prompts/documentation-lookup/31_advanced_batched_dofn.md
index 0828096815f..73b697a7206 100644
--- a/learning/prompts/documentation-lookup/31_advanced_batched_dofn.md
+++ b/learning/prompts/documentation-lookup/31_advanced_batched_dofn.md
@@ -1,18 +1,17 @@
Prompt:
-What is a batched 'DoFn' in Apache Beam?
+What is a batched `DoFn` in Apache Beam?
Response:
-In Apache Beam, a [batched ‘DoFn’](https://beam.apache.org/documentation/programming-guide/#batched-dofns) is a specialized type of [‘DoFn’](https://beam.apache.org/documentation/programming-guide/#pardo) that operates on batches of logical elements within a [‘PCollection’](https://beam.apache.org/documentation/basics/#pcollection). Unlike regular ‘DoFn’s, which process each element individually, batched ‘DoFn’s enable the efficient processing of multiple elements together as batches.
+In Apache Beam, a [batched `DoFn`](https://beam.apache.org/documentation/programming-guide/#batched-dofns) is a specialized type of [`DoFn`](https://beam.apache.org/documentation/programming-guide/#pardo) that operates on batches of logical elements within a [`PCollection`](https://beam.apache.org/documentation/basics/#pcollection). Unlike regular `DoFn` functions, which process each element individually, batched `DoFn` functions enable the efficient processing of multiple elements toget [...]
-Batched `DoFns` are currently only available in the Apache Beam Python SDK. They leverage powerful Python vectorized libraries, such as [NumPy](https://numpy.org/), [SciPy](https://scipy.org/), and [pandas](https://pandas.pydata.org/), which are designed for analysis and computing at scale.
-Batched 'DoFn's are particularly useful when dealing with large amounts of data, allowing for parallel and optimized processing. Additionally, you can chain multiple batched ‘DoFn’s together to create a more efficient data processing pipeline.
+Batched `DoFn` functions are currently only available in the Apache Beam Python SDK. They leverage powerful Python vectorized libraries, such as [NumPy](https://numpy.org/), [SciPy](https://scipy.org/), and [pandas](https://pandas.pydata.org/), which are designed for analysis and computing at scale.
-To implement a batched ‘DoFn’, you define a [‘process_batch’](https://beam.apache.org/releases/pydoc/current/_modules/apache_beam/transforms/core.html#DoFn.process_batch) method instead of the typical [‘process’](https://beam.apache.org/releases/pydoc/current/_modules/apache_beam/transforms/core.html#DoFn.process) method used in regular ‘DoFn’s. The ‘process_batch’ method takes a batch of elements as input and produces a batch of elements as an output. It is important to note that batche [...]
-For some ‘DoFn’s, you can provide both a batched and an element-wise implementation of your desired logic by defining both ‘process’ and ‘process_batch’ methods.
+Batched `DoFn` functions are particularly useful when dealing with large amounts of data, allowing for parallel and optimized processing. Additionally, you can chain multiple batched `DoFn` functions together to create a more efficient data processing pipeline.
-Here is a simple example of a flexible ‘DoFn’ with both batched and element-wise implementations:
+To implement a batched `DoFn`, you define a [`process_batch`](https://beam.apache.org/releases/pydoc/current/_modules/apache_beam/transforms/core.html#DoFn.process_batch) method instead of the typical [`process`](https://beam.apache.org/releases/pydoc/current/_modules/apache_beam/transforms/core.html#DoFn.process) method used in a regular `DoFn`. The `process_batch` method takes a batch of elements as input and produces a batch of elements as an output. It is important to note that batch [...]
+For some `DoFn` functions, you can provide both a batched and an element-wise implementation of your desired logic by defining both `process` and `process_batch` methods.
-In example below, when `MultiplyByTen` is applied to a `PCollection`, Beam recognizes that `np.ndarray` is an acceptable batch type to use in conjunction with `np.int64` elements.
+Here is a simple example of a flexible `DoFn` with both batched and element-wise implementations:
```python
class MultiplyByTen(beam.DoFn):
@@ -23,11 +22,11 @@ class MultiplyByTen(beam.DoFn):
yield batch * 10
```
-In this example, the ‘np.int64’ type represents the individual element. The 'process' method multiplies an element by ten, yielding a single element. The ‘np.ndarray’ type represents the batch. The 'process_batch' method multiplies each element in the batch by ten, yielding a single batch. During pipeline execution, Apache Beam will automatically select the best implementation based on the context.
+In this example, when `MultiplyByTen` is applied to a `PCollection`, Apache Beam recognizes that `np.ndarray` is an acceptable batch type to use in conjunction with `np.int64` elements. The `np.int64` type represents the individual element. The `process` method multiplies an element by ten, yielding a single element. The `np.ndarray` type represents the batch. The `process_batch` method multiplies each element in the batch by ten, yielding a single batch. During pipeline execution, Apach [...]
-By default, Apache Beam implicitly buffers elements and creates batches on the input side, then explodes batches back into individual elements on the output side. However, if batched 'DoFn's with equivalent types are chained together, this batch creation and explosion process is skipped, and the batches are passed through for more efficient processing.
+By default, Apache Beam implicitly buffers elements and creates batches on the input side, then explodes batches back into individual elements on the output side. However, if batched `DoFn` functions with equivalent types are chained together, this batch creation and explosion process is skipped, and the batches are passed through for more efficient processing.
-Here’s an example with chained ‘DoFn’s of equivalent types:
+Here is an example with chained `DoFn` functions of equivalent types:
```python
(p | beam.Create([1, 2, 3, 4]).with_output_types(np.int64)
@@ -36,8 +35,8 @@ Here’s an example with chained ‘DoFn’s of equivalent types:
| beam.ParDo(MultiplyByTen()))
```
-In this example, the ‘PTransform.with_output_types’ method sets the element-wise typehint for the output. Thus, when the `MultiplyByTen` class is applied to a `PCollection`, Apache Beam recognizes that `np.ndarray` is an acceptable batch type to use in conjunction with `np.int64` elements.
+In this example, the `PTransform.with_output_types` method sets the element-wise typehint for the output. Thus, when the `MultiplyByTen` class is applied to a `PCollection`, Apache Beam recognizes that `np.ndarray` is an acceptable batch type to use in conjunction with `np.int64` elements.
-The ‘process_batch’ method is expected to produce batched outputs, while the ‘process’ method should produce individual elements. You can customize this production logic with the ‘@beam.DoFn.yields_elements’ and ‘@beam.DoFn.yields_batches` decorators.
+The `process_batch` method is expected to produce batched outputs, while the `process` method should produce individual elements. You can customize this production logic with the `@beam.DoFn.yields_elements` and `@beam.DoFn.yields_batches` decorators.
-For more details, including advanced usage and code samples, you can refer to the [Apache Beam documentation on batched ‘DoFn’s](https://beam.apache.org/documentation/programming-guide/#batched-dofn-types).
\ No newline at end of file
+For more details, including advanced usage and code samples, you can refer to the [Apache Beam documentation on batched `DoFn` type](https://beam.apache.org/documentation/programming-guide/#batched-dofn-types).