[GitHub] [spark] HeartSaVioR commented on a change in pull request #34333: [SPARK-37062][SS] Introduce a new data source for providing consistent set of rows per microbatch

[GitBox <gi...@apache.org>]

HeartSaVioR commented on a change in pull request #34333:
URL: https://github.com/apache/spark/pull/34333#discussion_r732395419



##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -517,6 +517,8 @@ There are a few built-in sources.
 
   - **Rate source (for testing)** - Generates data at the specified number of rows per second, each output row contains a `timestamp` and `value`. Where `timestamp` is a `Timestamp` type containing the time of message dispatch, and `value` is of `Long` type containing the message count, starting from 0 as the first row. This source is intended for testing and benchmarking.

Review comment:
       The set of available options between rate source and rate-epoch source are different, like, `rampUpTime` is only valid for rate source, whereas `startTimestamp` and `advanceMillisPerEpoch` are only valid for rate-epoch source. 
   
   So if we have a unified data source (mix up `rate-epoch` into current `rate`), then there will be two sets of options, which defines the behavior, which IMHO is confusing.
   
   Could we please enumerate the upside of adding this as different mode of rate source?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org