Re: [PR] [Docs] Document UnboundedSource in the Python I/O connector guide [beam]

["Abacn (via GitHub)" <gi...@apache.org>]

Abacn commented on code in PR #39529:
URL: https://github.com/apache/beam/pull/39529#discussion_r3707392608


##########
website/www/site/content/en/documentation/io/developing-io-python.md:
##########
@@ -69,23 +69,39 @@ for Beam's transform style guidance.
 
 To create a new data source for your pipeline, you'll need to provide the format-specific logic that tells the service how to read data from your input source, and how to split your data source into multiple parts so that multiple worker instances can read your data in parallel.
 
+If you're creating a data source that reads unbounded data, you must also
+provide the logic for managing your source's watermark and checkpointing.
+
 Supply the logic for your new source by creating the following classes:
 
-  * A subclass of `BoundedSource`. `BoundedSource` is a source that reads a
-    finite amount of input records. The class describes the data you want to
-    read, including the data's location and parameters (such as how much data to
+  * A subclass of `BoundedSource` if you want to read a finite (batch) data set,
+    or a subclass of `UnboundedSource` if you want to read an infinite
+    (streaming) data set. The class describes the data you want to read,
+    including the data's location and parameters (such as how much data to
     read).
-  * A subclass of `RangeTracker`. `RangeTracker` is a thread-safe object used to
-    manage a range for a given position type.
+  * For a `BoundedSource`, a subclass of `RangeTracker`. `RangeTracker` is a
+    thread-safe object used to manage a range for a given position type.
+  * For an `UnboundedSource`, a subclass of `UnboundedReader`, which holds the
+    state involved in reading the stream, and a subclass of `CheckpointMark`,
+    which records the position that a reader resumes from.
   * One or more user-facing wrapper composite transforms (`PTransform`) that
     wrap read operations. [PTransform wrappers](#ptransform-wrappers) discusses
     why you should avoid exposing your sources, and walks through how to create
     a wrapper.
 
-You can find these classes in the
-[apache_beam.io.iobase module](https://beam.apache.org/releases/pydoc/{{< param release_latest >}}/apache_beam.io.iobase.html).
+You can find `BoundedSource` and `RangeTracker` in the
+[apache_beam.io.iobase module](https://beam.apache.org/releases/pydoc/{{< param release_latest >}}/apache_beam.io.iobase.html),
+and the unbounded classes in the
+[apache_beam.io.unbounded_source module](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/unbounded_source.py).

Review Comment:
   Let's use https://beam.apache.org/releases/pydoc/ website. This PR won't be merged until Beam 2.76.0 released, at that time pydoc website is updated to have unbounded_source



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org