Re: [PR] [SPARK-55450][SS][PYTHON][DOCS] Document admission control in PySpark streaming data sources [spark]

["jiteshsoni (via GitHub)" <gi...@apache.org>]

jiteshsoni commented on PR #54807:
URL: https://github.com/apache/spark/pull/54807#issuecomment-4185848309

   ## ✅ Testing Verification - Simplified Admission Control Example
   
   Tested the updated example on Databricks (DBR 17.3+ / Spark 4.0+). The admission control is working correctly.
   
   ### Test Configuration
   - **CHAIN_HEIGHT:** 10,000 blocks
   - **Batch size:** 20 blocks per batch (via `getDefaultReadLimit()`)
   - **Run duration:** ~30 seconds
   
   ### Results
   - **Completed batches:** 79 batches in 28 seconds
   - **Input rows per batch:** Consistently **20 rows** (admission control working)
   - **Processing rate:** ~60 records/second
   - **Batch duration:** ~342ms average
   
   ### Key Evidence from Streaming UI
   
   ```json
   {
     "batchId": 78,
     "numInputRows": 20,
     "sources": [{
       "description": "PythonMicroBatchStreamWithAdmissionControl",
       "startOffset": {"block_number": 1560},
       "endOffset": {"block_number": 1580},
       "numInputRows": 20
     }]
   }
   ```
   
   ### Verified Functionality
   - ✅ `getDefaultReadLimit()` returns `ReadMaxRows(20)` - limits batch size correctly
   - ✅ `latestOffset(start, limit)` respects the `ReadLimit` parameter
   - ✅ Each micro-batch processes exactly 20 blocks as expected
   - ✅ Streaming query runs continuously with controlled ingestion rate
   
   ---
   _Tested on Databricks Dogfood Staging with DBR 17.3 (Spark 4.0)_
   
   ---
   _This comment was generated with [GitHub MCP](http://go/mcps)._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org