You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/03/24 18:32:38 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #14327: [BEAM-9546] DataFrame.corr cleanup
TheNeuralBit opened a new pull request #14327:
URL: https://github.com/apache/beam/pull/14327
Minor code cleanups for DataFrame.corr
- Defers to Series.corr, insead of calling _corr_aligned
- Uses itertools.combinations
- Remove round() calls in tests, we no longer need this since we use pd.testing.assert_frames_equal which handles floating point values.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>Lang</th>
<th>SDK</th>
<th>ULR</th>
<th>Dataflow</th>
<th>Flink</th>
<th>Samza</th>
<th>Spark</th>
<th>Twister2</th>
</tr>
</thead>
<tbody>
<tr>
<td>Go</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/">
<img src="https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>Java</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_ULR/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon">
</a>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Python</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
<tr>
<td>XLang</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
</tr>
</tbody>
</table>
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
<table>
<thead>
<tr>
<th>---</th>
<th>Java</th>
<th>Python</th>
<th>Go</th>
<th>Website</th>
<th>Whitespace</th>
<th>Typescript</th>
</tr>
</thead>
<tbody>
<tr>
<td>Non-portable</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon">
</a><br>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/">
<img src="https://camo.githubusercontent.com/4565d7b7e907114e6c1d12323408bd903aa252fefed5eeab93701b05c9628a84/68747470733a2f2f63692d6265616d2e6170616368652e6f72672f6a6f622f6265616d5f507265436f6d6d69745f507974686f6e446f636b65725f43726f6e2f62616467652f69636f6e" alt="Build Status" data-canonical-src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon">
</a><br>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/">
<img src="https://camo.githubusercontent.com/21afb097a5745440598bee5c59a027b140585eec871c3f3b883200610fabf722/68747470733a2f2f63692d6265616d2e6170616368652e6f72672f6a6f622f6265616d5f507265436f6d6d69745f507974686f6e446f63735f43726f6e2f62616467652f69636f6e" alt="Build Status" data-canonical-src="https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
</tr>
<tr>
<td>Portable</td>
<td>---</td>
<td>
<a href="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/">
<img alt="Build Status" src="https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon">
</a>
</td>
<td>---</td>
<td>---</td>
<td>---</td>
<td>---</td>
</tr>
</tbody>
</table>
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #14327:
URL: https://github.com/apache/beam/pull/14327#discussion_r609073943
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -526,13 +526,12 @@ def test_dataframe_cov_corr(self):
df = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
df.loc[df.index[:5], 'a'] = np.nan
df.loc[df.index[5:10], 'b'] = np.nan
- self._run_test(lambda df: df.corr().round(8), df)
- self._run_test(lambda df: df.cov().round(8), df)
- self._run_test(lambda df: df.corr(min_periods=12).round(8), df)
- self._run_test(lambda df: df.cov(min_periods=12).round(8), df)
- self._run_test(lambda df: df.corrwith(df.a).round(8), df)
- self._run_test(
- lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]).round(8), df)
+ self._run_test(lambda df: df.corr(), df)
+ self._run_test(lambda df: df.cov(), df)
+ self._run_test(lambda df: df.corr(min_periods=12), df)
+ self._run_test(lambda df: df.cov(min_periods=12), df)
+ self._run_test(lambda df: df.corrwith(df.a), df)
+ self._run_test(lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]), df)
Review comment:
I don't see the value in that, isn't that just verifying pandas, and not our implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #14327:
URL: https://github.com/apache/beam/pull/14327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #14327:
URL: https://github.com/apache/beam/pull/14327#discussion_r608994721
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -526,13 +526,12 @@ def test_dataframe_cov_corr(self):
df = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
df.loc[df.index[:5], 'a'] = np.nan
df.loc[df.index[5:10], 'b'] = np.nan
- self._run_test(lambda df: df.corr().round(8), df)
- self._run_test(lambda df: df.cov().round(8), df)
- self._run_test(lambda df: df.corr(min_periods=12).round(8), df)
- self._run_test(lambda df: df.cov(min_periods=12).round(8), df)
- self._run_test(lambda df: df.corrwith(df.a).round(8), df)
- self._run_test(
- lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]).round(8), df)
+ self._run_test(lambda df: df.corr(), df)
+ self._run_test(lambda df: df.cov(), df)
+ self._run_test(lambda df: df.corr(min_periods=12), df)
+ self._run_test(lambda df: df.cov(min_periods=12), df)
+ self._run_test(lambda df: df.corrwith(df.a), df)
+ self._run_test(lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]), df)
Review comment:
Because there is logic being modified for the Pearson correlation, can you please add some tests ensuring commutativity between the order of the columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #14327:
URL: https://github.com/apache/beam/pull/14327#issuecomment-814287779
R: @rohdesamuel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #14327:
URL: https://github.com/apache/beam/pull/14327#issuecomment-806081298
# [Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=h1) Report
> Merging [#14327](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=desc) (1625d7a) into [master](https://codecov.io/gh/apache/beam/commit/83bd5485047373ae0e380c54063e3769874a8b09?el=desc) (83bd548) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14327 +/- ##
==========================================
- Coverage 83.41% 83.38% -0.04%
==========================================
Files 469 469
Lines 58727 58726 -1
==========================================
- Hits 48990 48968 -22
- Misses 9737 9758 +21
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ython/apache\_beam/examples/kafkataxi/kafka\_taxi.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMva2Fma2F0YXhpL2thZmthX3RheGkucHk=) | | |
| [.../build/srcs/sdks/python/apache\_beam/io/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vX19pbml0X18ucHk=) | | |
| [.../srcs/sdks/python/apache\_beam/metrics/execution.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9leGVjdXRpb24ucHk=) | | |
| [...python/apache\_beam/typehints/typehints\_test\_py3.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3R5cGVoaW50c190ZXN0X3B5My5weQ==) | | |
| [...mples/snippets/transforms/elementwise/partition.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9lbGVtZW50d2lzZS9wYXJ0aXRpb24ucHk=) | | |
| [...ers/interactive/display/pipeline\_graph\_renderer.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9kaXNwbGF5L3BpcGVsaW5lX2dyYXBoX3JlbmRlcmVyLnB5) | | |
| [...les/complete/juliaset/juliaset/juliaset\_test\_it.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXRfdGVzdF9pdC5weQ==) | | |
| [...s/python/apache\_beam/runners/worker/log\_handler.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvbG9nX2hhbmRsZXIucHk=) | | |
| [...hon/apache\_beam/runners/dataflow/internal/names.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9uYW1lcy5weQ==) | | |
| [...hon/apache\_beam/examples/complete/game/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvZ2FtZS9fX2luaXRfXy5weQ==) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=footer). Last update [83bd548...1625d7a](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14327:
URL: https://github.com/apache/beam/pull/14327#issuecomment-806081298
# [Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=h1) Report
> Merging [#14327](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=desc) (1625d7a) into [master](https://codecov.io/gh/apache/beam/commit/83bd5485047373ae0e380c54063e3769874a8b09?el=desc) (83bd548) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14327 +/- ##
==========================================
- Coverage 83.41% 83.38% -0.04%
==========================================
Files 469 469
Lines 58727 58726 -1
==========================================
- Hits 48990 48968 -22
- Misses 9737 9758 +21
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...hon/apache\_beam/portability/api/schema\_pb2\_grpc.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL3NjaGVtYV9wYjJfZ3JwYy5weQ==) | | |
| [.../apache\_beam/examples/cookbook/group\_with\_coder.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29va2Jvb2svZ3JvdXBfd2l0aF9jb2Rlci5weQ==) | | |
| [.../py38/build/srcs/sdks/python/apache\_beam/pvalue.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcHZhbHVlLnB5) | | |
| [.../build/srcs/sdks/python/apache\_beam/tools/utils.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdG9vbHMvdXRpbHMucHk=) | | |
| [...uild/srcs/sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | | |
| [...38/build/srcs/sdks/python/apache\_beam/io/textio.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vdGV4dGlvLnB5) | | |
| [...am/examples/complete/juliaset/juliaset/juliaset.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvY29tcGxldGUvanVsaWFzZXQvanVsaWFzZXQvanVsaWFzZXQucHk=) | | |
| [...s/sdks/python/apache\_beam/dataframe/expressions.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2V4cHJlc3Npb25zLnB5) | | |
| [...rcs/sdks/python/apache\_beam/typehints/typecheck.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3R5cGVjaGVjay5weQ==) | | |
| [...ython/apache\_beam/examples/kafkataxi/kafka\_taxi.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMva2Fma2F0YXhpL2thZmthX3RheGkucHk=) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=footer). Last update [83bd548...1625d7a](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #14327:
URL: https://github.com/apache/beam/pull/14327#discussion_r609067099
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -526,13 +526,12 @@ def test_dataframe_cov_corr(self):
df = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
df.loc[df.index[:5], 'a'] = np.nan
df.loc[df.index[5:10], 'b'] = np.nan
- self._run_test(lambda df: df.corr().round(8), df)
- self._run_test(lambda df: df.cov().round(8), df)
- self._run_test(lambda df: df.corr(min_periods=12).round(8), df)
- self._run_test(lambda df: df.cov(min_periods=12).round(8), df)
- self._run_test(lambda df: df.corrwith(df.a).round(8), df)
- self._run_test(
- lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]).round(8), df)
+ self._run_test(lambda df: df.corr(), df)
+ self._run_test(lambda df: df.cov(), df)
+ self._run_test(lambda df: df.corr(min_periods=12), df)
+ self._run_test(lambda df: df.cov(min_periods=12), df)
+ self._run_test(lambda df: df.corrwith(df.a), df)
+ self._run_test(lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]), df)
Review comment:
Right so something like:
pd.testing.assert_frame_equal(pd.DataFrame({'a': [1, 2, 3], 'b': [1, 2, 4]}).corr(), pd.DataFrame({'a': [1, 2, 4], 'b': [1, 2, 3]}).corr()).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #14327:
URL: https://github.com/apache/beam/pull/14327#issuecomment-806081298
# [Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=h1) Report
> Merging [#14327](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=desc) (1625d7a) into [master](https://codecov.io/gh/apache/beam/commit/83bd5485047373ae0e380c54063e3769874a8b09?el=desc) (83bd548) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #14327 +/- ##
==========================================
- Coverage 83.41% 83.38% -0.04%
==========================================
Files 469 469
Lines 58727 58726 -1
==========================================
- Hits 48990 48968 -22
- Misses 9737 9758 +21
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...s/dataflow/internal/clients/cloudbuild/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9pbnRlcm5hbC9jbGllbnRzL2Nsb3VkYnVpbGQvX19pbml0X18ucHk=) | | |
| [.../apache\_beam/ml/gcp/videointelligenceml\_test\_it.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvZ2NwL3ZpZGVvaW50ZWxsaWdlbmNlbWxfdGVzdF9pdC5weQ==) | | |
| [.../srcs/sdks/python/apache\_beam/io/flink/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmxpbmsvX19pbml0X18ucHk=) | | |
| [...sdks/python/apache\_beam/portability/common\_urns.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvY29tbW9uX3VybnMucHk=) | | |
| [.../python/apache\_beam/io/gcp/datastore/v1new/util.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2RhdGFzdG9yZS92MW5ldy91dGlsLnB5) | | |
| [...rcs/sdks/python/apache\_beam/io/aws/s3filesystem.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vYXdzL3MzZmlsZXN5c3RlbS5weQ==) | | |
| [...n/apache\_beam/runners/direct/test\_direct\_runner.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9kaXJlY3RfcnVubmVyLnB5) | | |
| [...ache\_beam/examples/snippets/transforms/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvc25pcHBldHMvdHJhbnNmb3Jtcy9fX2luaXRfXy5weQ==) | | |
| [...s/python/apache\_beam/testing/synthetic\_pipeline.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9zeW50aGV0aWNfcGlwZWxpbmUucHk=) | | |
| [.../srcs/sdks/python/apache\_beam/portability/utils.py](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree#diff-YmVhbV9QcmVDb21taXRfUHl0aG9uX0Nyb24vc3JjL3Nka3MvcHl0aG9uL3Rlc3Qtc3VpdGVzL3RveC9weTM4L2J1aWxkL3NyY3Mvc2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvdXRpbHMucHk=) | | |
| ... and [928 more](https://codecov.io/gh/apache/beam/pull/14327/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=footer). Last update [83bd548...1625d7a](https://codecov.io/gh/apache/beam/pull/14327?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #14327:
URL: https://github.com/apache/beam/pull/14327#discussion_r609017021
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -526,13 +526,12 @@ def test_dataframe_cov_corr(self):
df = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
df.loc[df.index[:5], 'a'] = np.nan
df.loc[df.index[5:10], 'b'] = np.nan
- self._run_test(lambda df: df.corr().round(8), df)
- self._run_test(lambda df: df.cov().round(8), df)
- self._run_test(lambda df: df.corr(min_periods=12).round(8), df)
- self._run_test(lambda df: df.cov(min_periods=12).round(8), df)
- self._run_test(lambda df: df.corrwith(df.a).round(8), df)
- self._run_test(
- lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]).round(8), df)
+ self._run_test(lambda df: df.corr(), df)
+ self._run_test(lambda df: df.cov(), df)
+ self._run_test(lambda df: df.corr(min_periods=12), df)
+ self._run_test(lambda df: df.cov(min_periods=12), df)
+ self._run_test(lambda df: df.corrwith(df.a), df)
+ self._run_test(lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]), df)
Review comment:
I'm not sure what you mean by that - do you mean I should add some more tests to verify this works correctly when columns are reordered? Something like:
```
df[['c', 'a', 'b']].corr()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rohdesamuel commented on a change in pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
rohdesamuel commented on a change in pull request #14327:
URL: https://github.com/apache/beam/pull/14327#discussion_r610024537
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -526,13 +526,12 @@ def test_dataframe_cov_corr(self):
df = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
df.loc[df.index[:5], 'a'] = np.nan
df.loc[df.index[5:10], 'b'] = np.nan
- self._run_test(lambda df: df.corr().round(8), df)
- self._run_test(lambda df: df.cov().round(8), df)
- self._run_test(lambda df: df.corr(min_periods=12).round(8), df)
- self._run_test(lambda df: df.cov(min_periods=12).round(8), df)
- self._run_test(lambda df: df.corrwith(df.a).round(8), df)
- self._run_test(
- lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]).round(8), df)
+ self._run_test(lambda df: df.corr(), df)
+ self._run_test(lambda df: df.cov(), df)
+ self._run_test(lambda df: df.corr(min_periods=12), df)
+ self._run_test(lambda df: df.cov(min_periods=12), df)
+ self._run_test(lambda df: df.corrwith(df.a), df)
+ self._run_test(lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]), df)
Review comment:
That would be true if there was only logic that wrapped the DataFrame.corr method, but there is manual logic here that unwraps the columns into every 2-combination and correlates those. It's a non-blocking issue so up to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #14327: [BEAM-9547] DataFrame.corr cleanup
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #14327:
URL: https://github.com/apache/beam/pull/14327#discussion_r609017021
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -526,13 +526,12 @@ def test_dataframe_cov_corr(self):
df = pd.DataFrame(np.random.randn(20, 3), columns=['a', 'b', 'c'])
df.loc[df.index[:5], 'a'] = np.nan
df.loc[df.index[5:10], 'b'] = np.nan
- self._run_test(lambda df: df.corr().round(8), df)
- self._run_test(lambda df: df.cov().round(8), df)
- self._run_test(lambda df: df.corr(min_periods=12).round(8), df)
- self._run_test(lambda df: df.cov(min_periods=12).round(8), df)
- self._run_test(lambda df: df.corrwith(df.a).round(8), df)
- self._run_test(
- lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]).round(8), df)
+ self._run_test(lambda df: df.corr(), df)
+ self._run_test(lambda df: df.cov(), df)
+ self._run_test(lambda df: df.corr(min_periods=12), df)
+ self._run_test(lambda df: df.cov(min_periods=12), df)
+ self._run_test(lambda df: df.corrwith(df.a), df)
+ self._run_test(lambda df: df[['a', 'b']].corrwith(df[['b', 'c']]), df)
Review comment:
I'm not sure what you mean by that - do you mean I should add some more tests to verify this works correctly when columns are reordered? Something like:
```py
df[['c', 'a', 'b']].corr()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org