[GitHub] [spark] pralabhkumar commented on a change in pull request #35191: [SPARK-37491][PYTHON]Fix Series.asof for unsorted values

[GitBox <gi...@apache.org>]

pralabhkumar commented on a change in pull request #35191:
URL: https://github.com/apache/spark/pull/35191#discussion_r799222433



##########
File path: python/pyspark/pandas/series.py
##########
@@ -5228,22 +5228,128 @@ def asof(self, where: Union[Any, List]) -> Union[Scalar, "Series"]:
             where = [where]
         index_scol = self._internal.index_spark_columns[0]
         index_type = self._internal.spark_type_for(index_scol)
+
+        # e.g where = [10, 20]
+        # In the comments below , will explain how the dataframe will look after transformations.
+        # e.g pd.Series([2, 1, np.nan, 4], index=[10, 20, 30, 40], name="Koalas")
+
+        column_prefix_constant = "col_"
         cond = [
-            F.max(F.when(index_scol <= SF.lit(index).cast(index_type), self.spark.column))
-            for index in where
+            F.when(
+                index_scol <= SF.lit(index).cast(index_type),
+                F.struct(
+                    F.lit(column_prefix_constant + str(index) + "_" + str(idx)).alias("identifier"),
+                    self.spark.column.alias("col_value"),
+                ),
+            ).alias(column_prefix_constant + str(index) + "_" + str(idx))

Review comment:
       Yes @itholic , this is working (since __index_level_0__) is sorted.  However , test case with psser.asof([25, 25]) , ambiguous  of duplicate cols in psdf = ps.DataFrame(sdf) . Therefore , in order to  pass above test case , 
   below is the change. 
   ```python
   
   cond = [
               F.last(
                   F.when(index_scol <= SF.lit(index).cast(index_type), self.spark.column),
                   ignorenulls=True,
               ).alias(column_prefix_constant + str(index) + "_" + str(idx))
               for idx, index in enumerate(where)
           ]
   
   ```
   Then 
   ```python
   with ps.option_context("compute.default_index_type", "distributed", "compute.max_rows", 1):
               psdf = ps.DataFrame(sdf)  # type: DataFrame
               df = pd.DataFrame(psdf.transpose().values, columns=[self.name], index=where)
               return df[df.columns[0]]
   
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org