You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HeartSaVioR (via GitHub)" <gi...@apache.org> on 2025/05/28 05:13:53 UTC

[PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

HeartSaVioR opened a new pull request, #51036:
URL: https://github.com/apache/spark/pull/51036

   ### What changes were proposed in this pull request?
   
   This PR proposes to squeeze the protocol of retrieving timers for transformWithState in PySpark, which will help a lot on dealing with not-to-be-huge number of timers.
   
   Here are the changes:
   
   * StatefulProcessorHandleImpl.listTimers(), StatefulProcessorHandleImpl.getExpiredTimers() no longer requires additional request to notice there is no further data to read.
     * We inline the data into proto message, to ease of determine whether the iterator has fully consumed or not.
   
   This change is the same mechanism we applied for ListState & MapState. We got performance improvement in the prior case, and we also see this change to be helpful on our internal benchmark.
   
   
   
   ### Why are the changes needed?
   
   To optimize further on some timer operations.
   
   We benchmarked the change with listing 100 timers (PR for benchmarking: #50952), and we saw overall performance improvements.
   
   > Before the fix
   
   ```
   
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   78.250		141.583		184.375		635.792		962743.500
    ==================== REGISTER latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   65.375		126.125		162.792		565.833		60809.333
    ==================== DELETE latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   68.500		130.000		170.292		610.083		156733.125
    ==================== LIST latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   486.833		714.961		998.625		2695.417		167039.959
   
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   77.916		139.000		182.375		671.792		521809.958
    ==================== REGISTER latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   65.000		124.333		160.875		596.667		30860.208
    ==================== DELETE latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   67.125		127.916		170.250		740.051		64404.416
    ==================== LIST latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   482.041		710.333		1050.333		2685.500		76762.583
   
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   78.208		139.959		181.459		722.459		713788.250
    ==================== REGISTER latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   65.209		125.125		159.625		636.666		27963.167
    ==================== DELETE latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   67.417		129.000		168.875		764.602		12991.667
    ==================== LIST latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   479.000		709.584		1045.543		2776.541		92247.542
   ```
   
   > After the fix
   
   ```
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   31.250		47.250		75.875		150.000		551557.750
    ==================== REGISTER latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   26.958		39.208		65.208		122.667		78609.292
    ==================== DELETE latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   23.500		41.125		64.542		125.958		52641.042
    ==================== LIST latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   93.125		118.542		156.500		284.625		19910.000
   
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   30.875		44.083		70.417		128.875		628912.209
    ==================== REGISTER latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   26.917		36.416		61.292		109.917		164584.666
    ==================== DELETE latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   23.333		38.375		59.542		113.839		114350.250
    ==================== LIST latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   94.125		115.208		148.917		246.292		36924.292
   
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   31.375		58.375		93.041		243.750		719545.583
    ==================== REGISTER latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   26.959		50.167		81.833		194.375		67609.583
    ==================== DELETE latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   24.208		50.834		83.000		211.018		20611.959
    ==================== LIST latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   95.291		132.375		183.875		427.584		36971.792
   ```
   
   Worth noting that it is not only impacting the LIST operation - it also impacts other operations as well. It's not clear why it happens, but the direction of reducing round-trips is proven to be the right direction.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Existing UT.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "jingz-db (via GitHub)" <gi...@apache.org>.
jingz-db commented on PR #51036:
URL: https://github.com/apache/spark/pull/51036#issuecomment-2917387367

   > a high-level question: You mentioned this optimization only helps when there are "not-to-be-huge number of timers". Is that because the change reduces the number of calls from N to N - 1, so for a small N (e.g., from 2 to 1), the reduction is significant (50%)—but when N is large, the relative benefit becomes minimal?
   
   IIUC, another benefits of this PR is to get rid of the mid layer of PandasDataFrame and directly use `Row`. This also saves se/deserialization time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #51036: [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark
URL: https://github.com/apache/spark/pull/51036


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #51036:
URL: https://github.com/apache/spark/pull/51036#issuecomment-2918123063

   > IIUC, another benefits of this PR is to get rid of the mid layer of PandasDataFrame and directly use Row. 
   
   It is actually not proved - Scala side does not have the way to pass the Row which Python does not need to deserialize, and vice versa. They also need serde. Arrow RecordBatch is no doubt.
   
   So it's really a matter of which approach has more overhead on how many records and how many columns. Unfortunately I don't have a definite answer for this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "jingz-db (via GitHub)" <gi...@apache.org>.
jingz-db commented on code in PR #51036:
URL: https://github.com/apache/spark/pull/51036#discussion_r2112753141


##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -222,76 +224,96 @@ def delete_timer(self, expiry_time_stamp_ms: int) -> None:
             # TODO(SPARK-49233): Classify user facing errors.
             raise PySparkRuntimeError(f"Error deleting timer: " f"{response_message[1]}")
 
-    def get_list_timer_row(self, iterator_id: str) -> int:
+    def get_list_timer_row(self, iterator_id: str) -> Tuple[int, bool]:
         import pyspark.sql.streaming.proto.StateMessage_pb2 as stateMessage
 
         if iterator_id in self.list_timer_iterator_cursors:
             # if the iterator is already in the dictionary, return the next row
-            pandas_df, index = self.list_timer_iterator_cursors[iterator_id]
+            data_batch, index, require_next_fetch = self.list_timer_iterator_cursors[iterator_id]
         else:
             list_call = stateMessage.ListTimers(iteratorId=iterator_id)
             state_call_command = stateMessage.TimerStateCallCommand(list=list_call)
             call = stateMessage.StatefulProcessorCall(timerStateCall=state_call_command)
             message = stateMessage.StateRequest(statefulProcessorCall=call)
 
             self._send_proto_message(message.SerializeToString())
-            response_message = self._receive_proto_message()
+            response_message = self._receive_proto_message_with_timers()
             status = response_message[0]
             if status == 0:
-                iterator = self._read_arrow_state()
-                # We need to exhaust the iterator here to make sure all the arrow batches are read,
-                # even though there is only one batch in the iterator. Otherwise, the stream might
-                # block further reads since it thinks there might still be some arrow batches left.
-                # We only need to read the first batch in the iterator because it's guaranteed that
-                # there would only be one batch sent from the JVM side.
-                data_batch = None
-                for batch in iterator:
-                    if data_batch is None:
-                        data_batch = batch
-                if data_batch is None:
-                    # TODO(SPARK-49233): Classify user facing errors.
-                    raise PySparkRuntimeError("Error getting map state entry.")
-                pandas_df = data_batch.to_pandas()
+                data_batch = list(

Review Comment:
   Correct me if I am wrong: We are now not sending expired timer using arrow batch, but sending in batched list?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #51036:
URL: https://github.com/apache/spark/pull/51036#discussion_r2113095885


##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -222,76 +224,96 @@ def delete_timer(self, expiry_time_stamp_ms: int) -> None:
             # TODO(SPARK-49233): Classify user facing errors.
             raise PySparkRuntimeError(f"Error deleting timer: " f"{response_message[1]}")
 
-    def get_list_timer_row(self, iterator_id: str) -> int:
+    def get_list_timer_row(self, iterator_id: str) -> Tuple[int, bool]:
         import pyspark.sql.streaming.proto.StateMessage_pb2 as stateMessage
 
         if iterator_id in self.list_timer_iterator_cursors:
             # if the iterator is already in the dictionary, return the next row
-            pandas_df, index = self.list_timer_iterator_cursors[iterator_id]
+            data_batch, index, require_next_fetch = self.list_timer_iterator_cursors[iterator_id]
         else:
             list_call = stateMessage.ListTimers(iteratorId=iterator_id)
             state_call_command = stateMessage.TimerStateCallCommand(list=list_call)
             call = stateMessage.StatefulProcessorCall(timerStateCall=state_call_command)
             message = stateMessage.StateRequest(statefulProcessorCall=call)
 
             self._send_proto_message(message.SerializeToString())
-            response_message = self._receive_proto_message()
+            response_message = self._receive_proto_message_with_timers()
             status = response_message[0]
             if status == 0:
-                iterator = self._read_arrow_state()
-                # We need to exhaust the iterator here to make sure all the arrow batches are read,
-                # even though there is only one batch in the iterator. Otherwise, the stream might
-                # block further reads since it thinks there might still be some arrow batches left.
-                # We only need to read the first batch in the iterator because it's guaranteed that
-                # there would only be one batch sent from the JVM side.
-                data_batch = None
-                for batch in iterator:
-                    if data_batch is None:
-                        data_batch = batch
-                if data_batch is None:
-                    # TODO(SPARK-49233): Classify user facing errors.
-                    raise PySparkRuntimeError("Error getting map state entry.")
-                pandas_df = data_batch.to_pandas()
+                data_batch = list(
+                    map(
+                        lambda x: x.timestampMs,
+                        response_message[2]
+                    )
+                )
+                require_next_fetch = response_message[3]
                 index = 0
             else:
                 raise StopIteration()
+
+        is_last_row = False
         new_index = index + 1
-        if new_index < len(pandas_df):
+        if new_index < len(data_batch):
             # Update the index in the dictionary.
-            self.list_timer_iterator_cursors[iterator_id] = (pandas_df, new_index)
+            self.list_timer_iterator_cursors[iterator_id] = (data_batch, new_index, require_next_fetch)
         else:
-            # If the index is at the end of the DataFrame, remove the state from the dictionary.
+            # If the index is at the end of the data batch, remove the state from the dictionary.
             self.list_timer_iterator_cursors.pop(iterator_id, None)
-        return pandas_df.at[index, "timestamp"].item()
+            is_last_row = True
+
+        is_last_row_from_iterator = is_last_row and not require_next_fetch
+        timestamp = data_batch[index]
+        return (timestamp, is_last_row_from_iterator)
 
     def get_expiry_timers_iterator(
-        self, expiry_timestamp: int
-    ) -> Iterator[list[Tuple[Tuple, int]]]:
+        self,
+        iterator_id: str,
+        expiry_timestamp: int
+    ) -> Tuple[Tuple, int, bool]:
         import pyspark.sql.streaming.proto.StateMessage_pb2 as stateMessage
 
-        while True:
-            expiry_timer_call = stateMessage.ExpiryTimerRequest(expiryTimestampMs=expiry_timestamp)
+        if iterator_id in self.expiry_timer_iterator_cursors:
+            # If the state is already in the dictionary, return the next row.
+            data_batch, index, require_next_fetch = self.expiry_timer_iterator_cursors[iterator_id]
+        else:
+            expiry_timer_call = stateMessage.ExpiryTimerRequest(
+                expiryTimestampMs=expiry_timestamp,
+                iteratorId=iterator_id
+            )
             timer_request = stateMessage.TimerRequest(expiryTimerRequest=expiry_timer_call)
             message = stateMessage.StateRequest(timerRequest=timer_request)
 
             self._send_proto_message(message.SerializeToString())
-            response_message = self._receive_proto_message()
+            response_message = self._receive_proto_message_with_timers()
             status = response_message[0]
-            if status == 1:
-                break
-            elif status == 0:
-                result_list = []
-                iterator = self._read_arrow_state()
-                for batch in iterator:
-                    batch_df = batch.to_pandas()
-                    for i in range(batch.num_rows):
-                        deserialized_key = self.pickleSer.loads(batch_df.at[i, "key"])
-                        timestamp = batch_df.at[i, "timestamp"].item()
-                        result_list.append((tuple(deserialized_key), timestamp))
-                yield result_list
+            if status == 0:

Review Comment:
   Two things:
   1. We see the benefit of inlining the data into proto message to save one round-trip.
   2. Arrow is the columnar format, which is known to be efficient when there are multiple data. It's not a good usage (though sometimes needed) to use Arrow RecordBatch with small number of records. It "might be" a bit different when there are enough number of records, especially the fact that pickled Python Row looks to contain the "schema" as "json", which is not needed at all with Arrow RecordBatch. Haven't tested with large number.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "jingz-db (via GitHub)" <gi...@apache.org>.
jingz-db commented on code in PR #51036:
URL: https://github.com/apache/spark/pull/51036#discussion_r2112757938


##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -222,76 +224,96 @@ def delete_timer(self, expiry_time_stamp_ms: int) -> None:
             # TODO(SPARK-49233): Classify user facing errors.
             raise PySparkRuntimeError(f"Error deleting timer: " f"{response_message[1]}")
 
-    def get_list_timer_row(self, iterator_id: str) -> int:
+    def get_list_timer_row(self, iterator_id: str) -> Tuple[int, bool]:
         import pyspark.sql.streaming.proto.StateMessage_pb2 as stateMessage
 
         if iterator_id in self.list_timer_iterator_cursors:
             # if the iterator is already in the dictionary, return the next row
-            pandas_df, index = self.list_timer_iterator_cursors[iterator_id]
+            data_batch, index, require_next_fetch = self.list_timer_iterator_cursors[iterator_id]
         else:
             list_call = stateMessage.ListTimers(iteratorId=iterator_id)
             state_call_command = stateMessage.TimerStateCallCommand(list=list_call)
             call = stateMessage.StatefulProcessorCall(timerStateCall=state_call_command)
             message = stateMessage.StateRequest(statefulProcessorCall=call)
 
             self._send_proto_message(message.SerializeToString())
-            response_message = self._receive_proto_message()
+            response_message = self._receive_proto_message_with_timers()
             status = response_message[0]
             if status == 0:
-                iterator = self._read_arrow_state()
-                # We need to exhaust the iterator here to make sure all the arrow batches are read,
-                # even though there is only one batch in the iterator. Otherwise, the stream might
-                # block further reads since it thinks there might still be some arrow batches left.
-                # We only need to read the first batch in the iterator because it's guaranteed that
-                # there would only be one batch sent from the JVM side.
-                data_batch = None
-                for batch in iterator:
-                    if data_batch is None:
-                        data_batch = batch
-                if data_batch is None:
-                    # TODO(SPARK-49233): Classify user facing errors.
-                    raise PySparkRuntimeError("Error getting map state entry.")
-                pandas_df = data_batch.to_pandas()
+                data_batch = list(
+                    map(
+                        lambda x: x.timestampMs,
+                        response_message[2]
+                    )
+                )
+                require_next_fetch = response_message[3]
                 index = 0
             else:
                 raise StopIteration()
+
+        is_last_row = False
         new_index = index + 1
-        if new_index < len(pandas_df):
+        if new_index < len(data_batch):
             # Update the index in the dictionary.
-            self.list_timer_iterator_cursors[iterator_id] = (pandas_df, new_index)
+            self.list_timer_iterator_cursors[iterator_id] = (data_batch, new_index, require_next_fetch)
         else:
-            # If the index is at the end of the DataFrame, remove the state from the dictionary.
+            # If the index is at the end of the data batch, remove the state from the dictionary.
             self.list_timer_iterator_cursors.pop(iterator_id, None)
-        return pandas_df.at[index, "timestamp"].item()
+            is_last_row = True
+
+        is_last_row_from_iterator = is_last_row and not require_next_fetch
+        timestamp = data_batch[index]
+        return (timestamp, is_last_row_from_iterator)
 
     def get_expiry_timers_iterator(
-        self, expiry_timestamp: int
-    ) -> Iterator[list[Tuple[Tuple, int]]]:
+        self,
+        iterator_id: str,
+        expiry_timestamp: int
+    ) -> Tuple[Tuple, int, bool]:
         import pyspark.sql.streaming.proto.StateMessage_pb2 as stateMessage
 
-        while True:
-            expiry_timer_call = stateMessage.ExpiryTimerRequest(expiryTimestampMs=expiry_timestamp)
+        if iterator_id in self.expiry_timer_iterator_cursors:
+            # If the state is already in the dictionary, return the next row.
+            data_batch, index, require_next_fetch = self.expiry_timer_iterator_cursors[iterator_id]
+        else:
+            expiry_timer_call = stateMessage.ExpiryTimerRequest(
+                expiryTimestampMs=expiry_timestamp,
+                iteratorId=iterator_id
+            )
             timer_request = stateMessage.TimerRequest(expiryTimerRequest=expiry_timer_call)
             message = stateMessage.StateRequest(timerRequest=timer_request)
 
             self._send_proto_message(message.SerializeToString())
-            response_message = self._receive_proto_message()
+            response_message = self._receive_proto_message_with_timers()
             status = response_message[0]
-            if status == 1:
-                break
-            elif status == 0:
-                result_list = []
-                iterator = self._read_arrow_state()
-                for batch in iterator:
-                    batch_df = batch.to_pandas()
-                    for i in range(batch.num_rows):
-                        deserialized_key = self.pickleSer.loads(batch_df.at[i, "key"])
-                        timestamp = batch_df.at[i, "timestamp"].item()
-                        result_list.append((tuple(deserialized_key), timestamp))
-                yield result_list
+            if status == 0:

Review Comment:
   Correct me if I am wrong: we are now not sending expiry timer in arrow batch, but in list of Rows - Is this because it improves performance by avoiding using arrow?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #51036:
URL: https://github.com/apache/spark/pull/51036#issuecomment-2918116295

   > Why do we also see this improvements in set implicit key as well?
   
   Unfortunately I don't know clearly - one of my suspicion is that network layer might be overloaded due to lots of small messages.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "jingz-db (via GitHub)" <gi...@apache.org>.
jingz-db commented on code in PR #51036:
URL: https://github.com/apache/spark/pull/51036#discussion_r2112753141


##########
python/pyspark/sql/streaming/stateful_processor_api_client.py:
##########
@@ -222,76 +224,96 @@ def delete_timer(self, expiry_time_stamp_ms: int) -> None:
             # TODO(SPARK-49233): Classify user facing errors.
             raise PySparkRuntimeError(f"Error deleting timer: " f"{response_message[1]}")
 
-    def get_list_timer_row(self, iterator_id: str) -> int:
+    def get_list_timer_row(self, iterator_id: str) -> Tuple[int, bool]:
         import pyspark.sql.streaming.proto.StateMessage_pb2 as stateMessage
 
         if iterator_id in self.list_timer_iterator_cursors:
             # if the iterator is already in the dictionary, return the next row
-            pandas_df, index = self.list_timer_iterator_cursors[iterator_id]
+            data_batch, index, require_next_fetch = self.list_timer_iterator_cursors[iterator_id]
         else:
             list_call = stateMessage.ListTimers(iteratorId=iterator_id)
             state_call_command = stateMessage.TimerStateCallCommand(list=list_call)
             call = stateMessage.StatefulProcessorCall(timerStateCall=state_call_command)
             message = stateMessage.StateRequest(statefulProcessorCall=call)
 
             self._send_proto_message(message.SerializeToString())
-            response_message = self._receive_proto_message()
+            response_message = self._receive_proto_message_with_timers()
             status = response_message[0]
             if status == 0:
-                iterator = self._read_arrow_state()
-                # We need to exhaust the iterator here to make sure all the arrow batches are read,
-                # even though there is only one batch in the iterator. Otherwise, the stream might
-                # block further reads since it thinks there might still be some arrow batches left.
-                # We only need to read the first batch in the iterator because it's guaranteed that
-                # there would only be one batch sent from the JVM side.
-                data_batch = None
-                for batch in iterator:
-                    if data_batch is None:
-                        data_batch = batch
-                if data_batch is None:
-                    # TODO(SPARK-49233): Classify user facing errors.
-                    raise PySparkRuntimeError("Error getting map state entry.")
-                pandas_df = data_batch.to_pandas()
+                data_batch = list(

Review Comment:
   Correct me if I am wrong: We are now not sending expired timer using arrow batch, but sending in batched list?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "jingz-db (via GitHub)" <gi...@apache.org>.
jingz-db commented on PR #51036:
URL: https://github.com/apache/spark/pull/51036#issuecomment-2917641716

   I might be missing something, but I think your benchmark results listed in the description is to benchmark against timer related operations and also your code change is built around timer:
   ```
   We benchmarked the change with listing 100 timers (PR for benchmarking: https://github.com/apache/spark/pull/50952), and we saw overall performance improvements.
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   78.250		141.583		184.375		635.792		962743.500
   
    ==================== SET IMPLICIT KEY latency (micros) ======================
   perc:50		perc:95		perc:99		perc:99.9		perc:100
   31.250		47.250		75.875		150.000		551557.750
   ```
   
   Why do we also see this improvements in set implicit key as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #51036:
URL: https://github.com/apache/spark/pull/51036#issuecomment-2918539046

   CI is green. Thanks! Merging to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #51036:
URL: https://github.com/apache/spark/pull/51036#issuecomment-2918268862

   CI failed with linter - I'll merge once I fix it and CI is back to green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52333][SS][PYTHON] Squeeze the protocol of retrieving timers for transformWithState in PySpark [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #51036:
URL: https://github.com/apache/spark/pull/51036#issuecomment-2914957421

   @bogao007 @jingz-db Please take a look, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org