[GitHub] [spark] zhengruifeng commented on a diff in pull request #36594: [SPARK-39223][PS] Implement skew and kurt in Rolling/RollingGroupby/Expanding/ExpandingGroupby

[GitBox <gi...@apache.org>]

zhengruifeng commented on code in PR #36594:
URL: https://github.com/apache/spark/pull/36594#discussion_r875712772


##########
python/pyspark/pandas/tests/test_expanding.py:
##########
@@ -27,11 +27,15 @@
 
 class ExpandingTest(PandasOnSparkTestCase, TestUtils):
     def _test_expanding_func(self, f):
-        pser = pd.Series([1, 2, 3], index=np.random.rand(3))
+        pser = pd.Series([1, 2, 3, 7, 9, 8], index=np.random.rand(6), name="a")
         psser = ps.from_pandas(pser)
-        self.assert_eq(getattr(psser.expanding(2), f)(), getattr(pser.expanding(2), f)())
         self.assert_eq(
-            getattr(psser.expanding(2), f)().sum(), getattr(pser.expanding(2), f)().sum()
+            getattr(psser.expanding(2), f)(), getattr(pser.expanding(2), f)(), almost=True
+        )

Review Comment:
   diff
   
   ```
   Series values are different (50.0 %)
   [index]: [0.7805062947269015, 0.9127802801942253, 0.21918713620794383, 0.8552689956236661, 0.35405097378660766, 0.13137715720543242]
   [left]:  [nan, nan, 0.0, 1.4430588355316425, 0.6068929550675305, 1.0790820550024759e-16]
   [right]: [nan, nan, -2.398081733190341e-14, 1.4430588355316425, 0.6068929550675303, 0.0]
   ```
   
   the difference here is quite tiny, it should come from the way to compute the central moment (`m2`/`m3`/`m4`), pandas compute them by first subtracting the mean, while spark sql compute them in an online manner



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org