You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anishshri-db (via GitHub)" <gi...@apache.org> on 2024/08/06 05:47:36 UTC
[PR] [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState [spark]
anishshri-db opened a new pull request, #47625:
URL: https://github.com/apache/spark/pull/47625
### What changes were proposed in this pull request?
Add docs for state metadata source for operators using schema format v2 such as transformWithState
### Why are the changes needed?
Adding documentation for state metadata source and its use with newer operators such as `transformWithState`
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
Existing tests
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #47625: [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState
URL: https://github.com/apache/spark/pull/47625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #47625:
URL: https://github.com/apache/spark/pull/47625#discussion_r1706512846
##########
docs/structured-streaming-state-data-source.md:
##########
@@ -310,6 +311,29 @@ Dataset<Row> df = spark
</div>
+The following options must be set for the source:
+
+<table>
+<thead><tr><th>Option</th><th>Value</th><th>Meaning</th></tr></thead>
+<tr>
+ <td>path</td>
+ <td>string</td>
+ <td>Specify the root directory of the checkpoint location. You can either specify the path via option("path", `path`) or load(`path`).</td>
+</tr>
+</table>
+
+The following configurations are optional:
+
+<table>
+<thead><tr><th>Option</th><th>Value</th><th>Default</th><th>Meaning</th></tr></thead>
+<tr>
+ <td>batchId</td>
+ <td>numeric value</td>
+ <td>Last committed batch if available, else 0</td>
+ <td>Optional batchId used to retrieve operator metadata at that batch. Only applicable for operators using schema format version 2.</td>
Review Comment:
Let's just hide the fact it's effective only for format version 2. Users wouldn't need to know about it and the expected behavior would be the same.
##########
docs/structured-streaming-state-data-source.md:
##########
@@ -264,13 +264,14 @@ The output schema will also be different from the normal output.
</tr>
</table>
-## State metadata source
+## State Metadata Source
Before querying the state from existing checkpoint via state data source, users would like to understand the information for the checkpoint, especially about state operator. This includes which operators and state store instances are available in the checkpoint, available range of batch IDs, etc.
Structured Streaming provides a data source named "State metadata source" to provide the state-related metadata information from the checkpoint.
Note: The metadata is constructed when the streaming query is running with Spark 4.0+. The existing checkpoint which has been running with lower Spark version does not have the metadata and will be unable to query/use with this metadata source. It is required to run the streaming query pointing the existing checkpoint in Spark 4.0+ to construct the metadata before querying.
+For operators using the operator metadata format version 1, the metadata is written once and does not change. For operators using metadata format version 2, this metadata can change and can be queried by providing the relevant batchId.
Review Comment:
We don't expect users to know about the operator metadata format, which operator uses the version 1 and which operator doesn't.
Flipping the coin, would there be a practice which users can simply follow without knowing the details? Shall we just guide users to always specify the batch ID if they want to know the metadata in "point of time", regardless of format version? My understanding is that the outcome should be the same.
##########
docs/structured-streaming-state-data-source.md:
##########
@@ -344,6 +368,11 @@ Each row in the source has the following schema:
<td>int</td>
<td>The maximum batch ID available for querying state. The value could be invalid if the streaming query taking the checkpoint is running, as the query will commit further batches.</td>
</tr>
+<tr>
+ <td>operatorProperties</td>
+ <td>string</td>
+ <td>List of properties used by the operator encoded as JSON. Available only for operators using schema format version 2.</td>
Review Comment:
Let's not expose the format version - we can just say "operator dependent".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #47625:
URL: https://github.com/apache/spark/pull/47625#issuecomment-2274260532
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on PR #47625:
URL: https://github.com/apache/spark/pull/47625#issuecomment-2270447483
cc - @HeartSaVioR - PTAL, thx !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49115][DOCS] Add docs for state metadata source for operators using schema format v2 such as transformWithState [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #47625:
URL: https://github.com/apache/spark/pull/47625#discussion_r1707446767
##########
docs/structured-streaming-state-data-source.md:
##########
@@ -264,13 +264,14 @@ The output schema will also be different from the normal output.
</tr>
</table>
-## State metadata source
+## State Metadata Source
Before querying the state from existing checkpoint via state data source, users would like to understand the information for the checkpoint, especially about state operator. This includes which operators and state store instances are available in the checkpoint, available range of batch IDs, etc.
Structured Streaming provides a data source named "State metadata source" to provide the state-related metadata information from the checkpoint.
Note: The metadata is constructed when the streaming query is running with Spark 4.0+. The existing checkpoint which has been running with lower Spark version does not have the metadata and will be unable to query/use with this metadata source. It is required to run the streaming query pointing the existing checkpoint in Spark 4.0+ to construct the metadata before querying.
+For operators using the operator metadata format version 1, the metadata is written once and does not change. For operators using metadata format version 2, this metadata can change and can be queried by providing the relevant batchId.
Review Comment:
Done
##########
docs/structured-streaming-state-data-source.md:
##########
@@ -344,6 +368,11 @@ Each row in the source has the following schema:
<td>int</td>
<td>The maximum batch ID available for querying state. The value could be invalid if the streaming query taking the checkpoint is running, as the query will commit further batches.</td>
</tr>
+<tr>
+ <td>operatorProperties</td>
+ <td>string</td>
+ <td>List of properties used by the operator encoded as JSON. Available only for operators using schema format version 2.</td>
Review Comment:
Done
##########
docs/structured-streaming-state-data-source.md:
##########
@@ -310,6 +311,29 @@ Dataset<Row> df = spark
</div>
+The following options must be set for the source:
+
+<table>
+<thead><tr><th>Option</th><th>Value</th><th>Meaning</th></tr></thead>
+<tr>
+ <td>path</td>
+ <td>string</td>
+ <td>Specify the root directory of the checkpoint location. You can either specify the path via option("path", `path`) or load(`path`).</td>
+</tr>
+</table>
+
+The following configurations are optional:
+
+<table>
+<thead><tr><th>Option</th><th>Value</th><th>Default</th><th>Meaning</th></tr></thead>
+<tr>
+ <td>batchId</td>
+ <td>numeric value</td>
+ <td>Last committed batch if available, else 0</td>
+ <td>Optional batchId used to retrieve operator metadata at that batch. Only applicable for operators using schema format version 2.</td>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org