You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/01 00:02:36 UTC
[GitHub] [beam] robertwb opened a new pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
robertwb opened a new pull request #12982:
URL: https://github.com/apache/beam/pull/12982
This implements correlation, covariance, and standard deviation for series and dataframes.
It is split up into three commits. The first is the math-heavy one that implements the distributed algorithm, the second uses it everywhere, and the third adds an apply method to scalars to simplify things.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12982:
URL: https://github.com/apache/beam/pull/12982#discussion_r501310799
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ def compute_moments(x):
+ n = len(x)
+ m = x.std(ddof=0)**2 * n
+ s = x.sum()
+ return pd.DataFrame(dict(m=[m], s=[s], n=[n]))
+
+ def combine_moments(data):
+ m = s = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ m, s, n = datum.m, datum.s, datum.n
+ else:
+ m += datum.m + (s / n - datum.s / datum.n)**2 * n * datum.n / (
Review comment:
nit: consider defining a variable for `delta` to make this easier to relate to the formula in https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -36,7 +36,7 @@ def _run_test(self, func, *args):
expressions.ConstantExpression(arg, arg[0:0])) for arg in args
]
expected = func(*args)
- actual = expressions.PartitioningSession({}).evaluate(
+ actual = expressions.Session({}).evaluate(
Review comment:
Why revert this?
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
Review comment:
Can this link directly to https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm and/or https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford's_online_algorithm?
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ def compute_moments(x):
+ n = len(x)
+ m = x.std(ddof=0)**2 * n
+ s = x.sum()
+ return pd.DataFrame(dict(m=[m], s=[s], n=[n]))
+
+ def combine_moments(data):
+ m = s = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ m, s, n = datum.m, datum.s, datum.n
+ else:
+ m += datum.m + (s / n - datum.s / datum.n)**2 * n * datum.n / (
+ n + datum.n)
+ s += datum.s
+ n += datum.n
+ if n <= ddof:
+ return float('nan')
+ else:
+ return math.sqrt(m / (n - ddof))
+
+ moments = expressions.ComputedExpression(
+ 'compute_moments',
+ compute_moments, [self._expr],
+ requires_partition_by=partitionings.Nothing())
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'combine_moments',
+ combine_moments, [moments],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def corr(self, other, method, min_periods):
+ if method == 'pearson': # Note that this is the default.
+ x = self.dropna()
+ y = other.dropna()
+
+ # Do this first to filter to the entries that are present on both sides.
+ def join(x, y):
+ return pd.concat([x, y], axis=1, join='inner').rename(
+ lambda c: 'xy'[c], axis=1)
+
+ # Use the formulae from
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance
+ def compute_co_moments(x, y):
+ n = len(x)
+ if n <= 1:
+ c = 0
+ else:
+ c = x.corr(y) * x.std() * y.std() * (n - 1)
+ sx = x.sum()
+ sy = y.sum()
+ return pd.DataFrame(dict(c=[c], sx=[sx], sy=[sy], n=[n]))
+
+ def combine_co_moments(data, std_x, std_y):
+ c = sx = sy = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ c, sx, sy, n = datum.c, datum.sx, datum.sy, datum.n
+ else:
+ c += (
+ datum.c + (sx / n - datum.sx / datum.n) *
+ (sy / n - datum.sy / datum.n) * n * datum.n / (n + datum.n))
+ sx += datum.sx
+ sy += datum.sy
+ n += datum.n
+ if n < max(2, min_periods or 0):
+ return float('nan')
+ else:
+ return c / (n - 1) / std_x / std_y
+
+ joined = frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'join',
+ join, [x._expr, y._expr],
+ requires_partition_by=partitionings.Index()))
+ std_x = joined.x.std()
+ std_y = joined.y.std()
+
+ moments = expressions.ComputedExpression(
+ 'compute_co_moments',
+ compute_co_moments, [joined.x._expr, joined.y._expr])
+
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'comnine_co_moments',
+ combine_co_moments, [moments, std_x._expr, std_y._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ else:
+ # The rank-based correlations are not obviously parallelizable, though
+ # perhaps an approximation could be done with a knowledge of quantiles
+ # and custom partitioning.
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'corr',
+ lambda df,
+ other: df.corr(other, method=method, DataFrame=min_periods)[
Review comment:
```suggestion
other: df.corr(other, method=method, min_periods=min_periods)[
```
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -150,8 +115,72 @@ def combine_co_moments(data, std_x, std_y):
expressions.ComputedExpression(
'corr',
lambda df,
- other: df.corr(other, method=method, DataFrame=min_periods)[
- self._expr, other._expr],
+ other: df.corr(other, method=method, min_periods=min_periods),
+ [self._expr, other._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ def _corr_aligned(self, other, method, min_periods):
+ std_x = self.std()
+ std_y = other.std()
+ cov = self._cov_aligned(other, min_periods)
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'normalize',
+ lambda cov,
+ std_x,
+ std_y: cov / (std_x * std_y),
+ [cov._expr, std_x._expr, std_y._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def cov(self, other, min_periods, ddof):
+ x, y = self.dropna().align(other.dropna(), 'inner')
+ return x._cov_aligned(y, min_periods, ddof)
+
+ def _cov_aligned(self, other, min_periods, ddof=1):
+ # Use the formulae from
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance
+ def compute_co_moments(x, y):
+ n = len(x)
+ if n <= 1:
+ c = 0
+ else:
+ c = x.corr(y) * x.std() * y.std() * (n - 1)
Review comment:
nit: this could become `x.cov(y) * (n-1)` which makes this more easily relatable to the wiki link
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ def compute_moments(x):
+ n = len(x)
+ m = x.std(ddof=0)**2 * n
+ s = x.sum()
+ return pd.DataFrame(dict(m=[m], s=[s], n=[n]))
+
+ def combine_moments(data):
+ m = s = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ m, s, n = datum.m, datum.s, datum.n
+ else:
+ m += datum.m + (s / n - datum.s / datum.n)**2 * n * datum.n / (
+ n + datum.n)
+ s += datum.s
+ n += datum.n
+ if n <= ddof:
+ return float('nan')
+ else:
+ return math.sqrt(m / (n - ddof))
+
+ moments = expressions.ComputedExpression(
+ 'compute_moments',
+ compute_moments, [self._expr],
+ requires_partition_by=partitionings.Nothing())
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'combine_moments',
+ combine_moments, [moments],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def corr(self, other, method, min_periods):
+ if method == 'pearson': # Note that this is the default.
+ x = self.dropna()
+ y = other.dropna()
+
+ # Do this first to filter to the entries that are present on both sides.
+ def join(x, y):
+ return pd.concat([x, y], axis=1, join='inner').rename(
+ lambda c: 'xy'[c], axis=1)
+
+ # Use the formulae from
Review comment:
It would help to be more specific here, I spent a while trying to find references. It looks like `combine_co_moments` is the formula for combining covariance from two sets hidden at the bottom of the [Online Covariance section](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online).
The other critical piece is the translation between co-moment and pearson correlation coefficient. Is there something we can reference for that? It seems to follow from the last definition [here](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient#For_a_sample). IIUC the co-moment is the numerator in that definition, and that fits with your code.
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ def compute_moments(x):
+ n = len(x)
+ m = x.std(ddof=0)**2 * n
+ s = x.sum()
+ return pd.DataFrame(dict(m=[m], s=[s], n=[n]))
+
+ def combine_moments(data):
+ m = s = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ m, s, n = datum.m, datum.s, datum.n
+ else:
+ m += datum.m + (s / n - datum.s / datum.n)**2 * n * datum.n / (
+ n + datum.n)
+ s += datum.s
+ n += datum.n
+ if n <= ddof:
+ return float('nan')
+ else:
+ return math.sqrt(m / (n - ddof))
+
+ moments = expressions.ComputedExpression(
+ 'compute_moments',
+ compute_moments, [self._expr],
+ requires_partition_by=partitionings.Nothing())
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'combine_moments',
+ combine_moments, [moments],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def corr(self, other, method, min_periods):
+ if method == 'pearson': # Note that this is the default.
+ x = self.dropna()
+ y = other.dropna()
+
+ # Do this first to filter to the entries that are present on both sides.
+ def join(x, y):
+ return pd.concat([x, y], axis=1, join='inner').rename(
+ lambda c: 'xy'[c], axis=1)
+
+ # Use the formulae from
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance
+ def compute_co_moments(x, y):
+ n = len(x)
+ if n <= 1:
+ c = 0
+ else:
+ c = x.corr(y) * x.std() * y.std() * (n - 1)
+ sx = x.sum()
+ sy = y.sum()
+ return pd.DataFrame(dict(c=[c], sx=[sx], sy=[sy], n=[n]))
+
+ def combine_co_moments(data, std_x, std_y):
+ c = sx = sy = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ c, sx, sy, n = datum.c, datum.sx, datum.sy, datum.n
+ else:
+ c += (
+ datum.c + (sx / n - datum.sx / datum.n) *
+ (sy / n - datum.sy / datum.n) * n * datum.n / (n + datum.n))
+ sx += datum.sx
+ sy += datum.sy
+ n += datum.n
+ if n < max(2, min_periods or 0):
+ return float('nan')
+ else:
+ return c / (n - 1) / std_x / std_y
+
+ joined = frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'join',
+ join, [x._expr, y._expr],
+ requires_partition_by=partitionings.Index()))
+ std_x = joined.x.std()
+ std_y = joined.y.std()
+
+ moments = expressions.ComputedExpression(
+ 'compute_co_moments',
+ compute_co_moments, [joined.x._expr, joined.y._expr])
+
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'comnine_co_moments',
+ combine_co_moments, [moments, std_x._expr, std_y._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ else:
+ # The rank-based correlations are not obviously parallelizable, though
+ # perhaps an approximation could be done with a knowledge of quantiles
+ # and custom partitioning.
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'corr',
+ lambda df,
+ other: df.corr(other, method=method, DataFrame=min_periods)[
Review comment:
Nevermind looks like you corrected this later
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -150,8 +115,72 @@ def combine_co_moments(data, std_x, std_y):
expressions.ComputedExpression(
'corr',
lambda df,
- other: df.corr(other, method=method, DataFrame=min_periods)[
- self._expr, other._expr],
+ other: df.corr(other, method=method, min_periods=min_periods),
+ [self._expr, other._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ def _corr_aligned(self, other, method, min_periods):
+ std_x = self.std()
+ std_y = other.std()
+ cov = self._cov_aligned(other, min_periods)
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'normalize',
+ lambda cov,
+ std_x,
+ std_y: cov / (std_x * std_y),
+ [cov._expr, std_x._expr, std_y._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def cov(self, other, min_periods, ddof):
+ x, y = self.dropna().align(other.dropna(), 'inner')
+ return x._cov_aligned(y, min_periods, ddof)
+
+ def _cov_aligned(self, other, min_periods, ddof=1):
+ # Use the formulae from
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance
+ def compute_co_moments(x, y):
+ n = len(x)
+ if n <= 1:
+ c = 0
+ else:
+ c = x.corr(y) * x.std() * y.std() * (n - 1)
+ sx = x.sum()
+ sy = y.sum()
+ return pd.DataFrame(dict(c=[c], sx=[sx], sy=[sy], n=[n]))
+
+ def combine_co_moments(data):
+ c = sx = sy = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ c, sx, sy, n = datum.c, datum.sx, datum.sy, datum.n
+ else:
+ c += (
+ datum.c + (sx / n - datum.sx / datum.n) *
+ (sy / n - datum.sy / datum.n) * n * datum.n / (n + datum.n))
+ sx += datum.sx
+ sy += datum.sy
+ n += datum.n
+ if n < max(2, min_periods or 0):
+ return float('nan')
Review comment:
Should this also check against ddof like std?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/1958eae6218c7ff4829f9caf82ce2708dfb9af31?el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.49% 82.54% +0.04%
==========================================
Files 455 453 -2
Lines 54871 54753 -118
==========================================
- Hits 45265 45194 -71
+ Misses 9606 9559 -47
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+0.77%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
| [...ython/apache\_beam/io/gcp/tests/bigquery\_matcher.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL2JpZ3F1ZXJ5X21hdGNoZXIucHk=) | `79.83% <0.00%> (-2.87%)` | :arrow_down: |
| [conftest.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/retry.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcmV0cnkucHk=) | `86.27% <0.00%> (-0.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/metrics/metricbase.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9tZXRyaWNiYXNlLnB5) | `87.87% <0.00%> (-0.36%)` | :arrow_down: |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | `77.77% <0.00%> (-0.23%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/urns.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvdXJucy5weQ==) | `88.88% <0.00%> (-0.18%)` | :arrow_down: |
| ... and [24 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [a66454b...83c9058](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.34% 82.54% +0.19%
==========================================
Files 455 453 -2
Lines 54650 54753 +103
==========================================
+ Hits 45002 45194 +192
+ Misses 9648 9559 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+1.29%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.49% <0.00%> (-0.66%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `80.08% <0.00%> (-0.63%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.66% <0.00%> (ø)` | |
| [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [cee7388...6c557a0](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.34% 82.54% +0.19%
==========================================
Files 455 453 -2
Lines 54650 54753 +103
==========================================
+ Hits 45002 45194 +192
+ Misses 9648 9559 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+1.29%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.49% <0.00%> (-0.66%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `80.08% <0.00%> (-0.63%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.66% <0.00%> (ø)` | |
| [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [cee7388...6c557a0](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-701708339
R: @TheNeuralBit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.34% 82.54% +0.19%
==========================================
Files 455 453 -2
Lines 54650 54753 +103
==========================================
+ Hits 45002 45194 +192
+ Misses 9648 9559 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+1.29%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.49% <0.00%> (-0.66%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `80.08% <0.00%> (-0.63%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.66% <0.00%> (ø)` | |
| [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [cee7388...8c0ac9b](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.34% 82.54% +0.19%
==========================================
Files 455 453 -2
Lines 54650 54753 +103
==========================================
+ Hits 45002 45194 +192
+ Misses 9648 9559 -89
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+1.29%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.49% <0.00%> (-0.66%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `80.08% <0.00%> (-0.63%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `94.18% <0.00%> (-0.27%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.66% <0.00%> (ø)` | |
| [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [cee7388...6c557a0](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/6bf56f92b34f7c15b752c46eca19489a604c4775?el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.51% 82.54% +0.02%
==========================================
Files 455 453 -2
Lines 54867 54753 -114
==========================================
- Hits 45272 45194 -78
+ Misses 9595 9559 -36
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+0.77%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `83.33% <0.00%> (-9.53%)` | :arrow_down: |
| [...ython/apache\_beam/io/gcp/tests/bigquery\_matcher.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3Rlc3RzL2JpZ3F1ZXJ5X21hdGNoZXIucHk=) | `79.83% <0.00%> (-2.87%)` | :arrow_down: |
| [conftest.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-Y29uZnRlc3QucHk=) | `83.33% <0.00%> (-2.39%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/source\_test\_utils.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vc291cmNlX3Rlc3RfdXRpbHMucHk=) | `88.28% <0.00%> (-1.36%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/retry.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvcmV0cnkucHk=) | `86.27% <0.00%> (-0.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.75% <0.00%> (-0.45%)` | :arrow_down: |
| [sdks/python/apache\_beam/metrics/metricbase.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9tZXRyaWNiYXNlLnB5) | `87.87% <0.00%> (-0.36%)` | :arrow_down: |
| ... and [25 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [6bf56f9...e5b9a39](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc) will **increase** coverage by `0.20%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.34% 82.54% +0.20%
==========================================
Files 455 453 -2
Lines 54650 54753 +103
==========================================
+ Hits 45002 45197 +195
+ Misses 9648 9556 -92
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+1.29%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `80.23% <0.00%> (-0.48%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `83.75% <0.00%> (-0.29%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.98% <0.00%> (-0.17%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.66% <0.00%> (ø)` | |
| [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [8 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [cee7388...0387ef2](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12982:
URL: https://github.com/apache/beam/pull/12982#discussion_r502197895
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ def compute_moments(x):
+ n = len(x)
+ m = x.std(ddof=0)**2 * n
+ s = x.sum()
+ return pd.DataFrame(dict(m=[m], s=[s], n=[n]))
+
+ def combine_moments(data):
+ m = s = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ m, s, n = datum.m, datum.s, datum.n
+ else:
+ m += datum.m + (s / n - datum.s / datum.n)**2 * n * datum.n / (
Review comment:
Done.
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -150,8 +115,72 @@ def combine_co_moments(data, std_x, std_y):
expressions.ComputedExpression(
'corr',
lambda df,
- other: df.corr(other, method=method, DataFrame=min_periods)[
- self._expr, other._expr],
+ other: df.corr(other, method=method, min_periods=min_periods),
+ [self._expr, other._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ def _corr_aligned(self, other, method, min_periods):
+ std_x = self.std()
+ std_y = other.std()
+ cov = self._cov_aligned(other, min_periods)
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'normalize',
+ lambda cov,
+ std_x,
+ std_y: cov / (std_x * std_y),
+ [cov._expr, std_x._expr, std_y._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def cov(self, other, min_periods, ddof):
+ x, y = self.dropna().align(other.dropna(), 'inner')
+ return x._cov_aligned(y, min_periods, ddof)
+
+ def _cov_aligned(self, other, min_periods, ddof=1):
+ # Use the formulae from
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance
+ def compute_co_moments(x, y):
+ n = len(x)
+ if n <= 1:
+ c = 0
+ else:
+ c = x.corr(y) * x.std() * y.std() * (n - 1)
+ sx = x.sum()
+ sy = y.sum()
+ return pd.DataFrame(dict(c=[c], sx=[sx], sy=[sy], n=[n]))
+
+ def combine_co_moments(data):
+ c = sx = sy = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ c, sx, sy, n = datum.c, datum.sx, datum.sy, datum.n
+ else:
+ c += (
+ datum.c + (sx / n - datum.sx / datum.n) *
+ (sy / n - datum.sy / datum.n) * n * datum.n / (n + datum.n))
+ sx += datum.sx
+ sy += datum.sy
+ n += datum.n
+ if n < max(2, min_periods or 0):
+ return float('nan')
Review comment:
Yes. Done.
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
Review comment:
Done. Even if the subheading change, they're descriptive enough to identify what is meant.
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -36,7 +36,7 @@ def _run_test(self, func, *args):
expressions.ConstantExpression(arg, arg[0:0])) for arg in args
]
expected = func(*args)
- actual = expressions.PartitioningSession({}).evaluate(
+ actual = expressions.Session({}).evaluate(
Review comment:
Some of the other tests in this file were relying on the ordering.
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -150,8 +115,72 @@ def combine_co_moments(data, std_x, std_y):
expressions.ComputedExpression(
'corr',
lambda df,
- other: df.corr(other, method=method, DataFrame=min_periods)[
- self._expr, other._expr],
+ other: df.corr(other, method=method, min_periods=min_periods),
+ [self._expr, other._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ def _corr_aligned(self, other, method, min_periods):
+ std_x = self.std()
+ std_y = other.std()
+ cov = self._cov_aligned(other, min_periods)
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'normalize',
+ lambda cov,
+ std_x,
+ std_y: cov / (std_x * std_y),
+ [cov._expr, std_x._expr, std_y._expr],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def cov(self, other, min_periods, ddof):
+ x, y = self.dropna().align(other.dropna(), 'inner')
+ return x._cov_aligned(y, min_periods, ddof)
+
+ def _cov_aligned(self, other, min_periods, ddof=1):
+ # Use the formulae from
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Covariance
+ def compute_co_moments(x, y):
+ n = len(x)
+ if n <= 1:
+ c = 0
+ else:
+ c = x.corr(y) * x.std() * y.std() * (n - 1)
Review comment:
Good point. Done.
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -36,7 +36,7 @@ def _run_test(self, func, *args):
expressions.ConstantExpression(arg, arg[0:0])) for arg in args
]
expected = func(*args)
- actual = expressions.PartitioningSession({}).evaluate(
+ actual = expressions.Session({}).evaluate(
Review comment:
I've made this an option, we can try to switch it in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
robertwb merged pull request #12982:
URL: https://github.com/apache/beam/pull/12982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12982:
URL: https://github.com/apache/beam/pull/12982#issuecomment-702284790
# [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=h1) Report
> Merging [#12982](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/cee7388365970774969e63fe90fdf3cdd804e79f?el=desc) will **increase** coverage by `0.20%`.
> The diff coverage is `96.57%`.
[](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12982 +/- ##
==========================================
+ Coverage 82.34% 82.54% +0.20%
==========================================
Files 455 453 -2
Lines 54650 54753 +103
==========================================
+ Hits 45002 45197 +195
+ Misses 9648 9556 -92
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `92.06% <96.45%> (+1.29%)` | :arrow_up: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `85.63% <100.00%> (+0.32%)` | :arrow_up: |
| [...dks/python/apache\_beam/options/pipeline\_options.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vb3B0aW9ucy9waXBlbGluZV9vcHRpb25zLnB5) | `93.76% <0.00%> (-1.03%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `80.23% <0.00%> (-0.48%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/iobase.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vaW9iYXNlLnB5) | `83.75% <0.00%> (-0.29%)` | :arrow_down: |
| [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `89.98% <0.00%> (-0.17%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery\_tools.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3Rvb2xzLnB5) | `88.35% <0.00%> (-0.15%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/fileio.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZmlsZWlvLnB5) | `95.80% <0.00%> (-0.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `91.66% <0.00%> (ø)` | |
| [...e\_beam/portability/api/beam\_runner\_api\_pb2\_urns.py](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcG9ydGFiaWxpdHkvYXBpL2JlYW1fcnVubmVyX2FwaV9wYjJfdXJucy5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [8 more](https://codecov.io/gh/apache/beam/pull/12982/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=footer). Last update [cee7388...6c557a0](https://codecov.io/gh/apache/beam/pull/12982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12982: [BEAM-9547] Dataframe covariance and correlation.
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12982:
URL: https://github.com/apache/beam/pull/12982#discussion_r501362686
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -34,6 +36,124 @@ def __array__(self, dtype=None):
between = frame_base._elementwise_method('between')
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def std(self, axis, skipna, level, ddof, **kwargs):
+ if level is not None:
+ raise NotImplementedError("per-level aggregation")
+ if skipna:
+ self = self.dropna()
+
+ # See the online, numerically stable formulae at
+ # https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
+ def compute_moments(x):
+ n = len(x)
+ m = x.std(ddof=0)**2 * n
+ s = x.sum()
+ return pd.DataFrame(dict(m=[m], s=[s], n=[n]))
+
+ def combine_moments(data):
+ m = s = n = 0.0
+ for datum in data.itertuples():
+ if datum.n == 0:
+ continue
+ elif n == 0:
+ m, s, n = datum.m, datum.s, datum.n
+ else:
+ m += datum.m + (s / n - datum.s / datum.n)**2 * n * datum.n / (
+ n + datum.n)
+ s += datum.s
+ n += datum.n
+ if n <= ddof:
+ return float('nan')
+ else:
+ return math.sqrt(m / (n - ddof))
+
+ moments = expressions.ComputedExpression(
+ 'compute_moments',
+ compute_moments, [self._expr],
+ requires_partition_by=partitionings.Nothing())
+ with expressions.allow_non_parallel_operations(True):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'combine_moments',
+ combine_moments, [moments],
+ requires_partition_by=partitionings.Singleton()))
+
+ @frame_base.args_to_kwargs(pd.Series)
+ @frame_base.populate_defaults(pd.Series)
+ def corr(self, other, method, min_periods):
+ if method == 'pearson': # Note that this is the default.
+ x = self.dropna()
+ y = other.dropna()
+
+ # Do this first to filter to the entries that are present on both sides.
+ def join(x, y):
+ return pd.concat([x, y], axis=1, join='inner').rename(
+ lambda c: 'xy'[c], axis=1)
+
+ # Use the formulae from
Review comment:
I think the reference to pearson correlation coefficient is no longer necessary after this was moved to `_cov_aligned`, which can be understood just from https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online. It would still be helpful to be more specific here though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org