You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/31 22:21:17 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
TheNeuralBit opened a new pull request #12440:
URL: https://github.com/apache/beam/pull/12440
The report looks like this when run with `pytest -s`:
```
apache_beam/dataframe/pandas_doctests_test.py 829 total test cases.
722 will implement, 107 won't implement.
257 skipped.
69.0% (572/829) of all test cases pass or raise WontImplementError
.676 total test cases.
631 will implement, 45 won't implement.
265 skipped.
60.8% (411/676) of all test cases pass or raise WontImplementError
```
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12440:
URL: https://github.com/apache/beam/pull/12440#issuecomment-667419671
Run PythonDocker PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r466688418
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
with self._test_env.context():
- return super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ result = super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ return result
def fake_pandas_module(self):
return self._test_env.fake_pandas_module()
+ def summarize(self):
+ super(BeamDataframeDoctestRunner, self).summarize()
+ if self.failures:
+ return
Review comment:
I added this because I wasn't sure how to accurately report the stats in the case of failures. In theory one of the wont implement tests could fail, then skipped/wont implement/passed/failed wouldn't be a partitioning of tries.
If there's a way to get specific information about the wont implement test cases we could de-dup it though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r466686042
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
Review comment:
We could record it here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/dataframe/doctests.py#L242
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r467238866
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -90,6 +113,16 @@ def test_series_tests(self):
result = doctests.testmod(
pd.core.series,
use_beam=False,
+ report=True,
+ wont_implement={
Review comment:
Done!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12440:
URL: https://github.com/apache/beam/pull/12440#issuecomment-670226776
Updated the report to collect information about wont implement, now it looks like this:
DataFrame:
```
572 total test cases:
257 skipped (44.9%)
107 won't implement (18.7%)
32 (29.9%)
20 WontImplementError: order-sensitive (18.7%)
18 DeferredBase[*] (16.8%)
11 WontImplementError: non-deferred value (10.3%)
7 WontImplementError: non-deferred column values (6.5%)
7 NameError (6.5%)
6 WontImplementError: non-lazy (5.6%)
2 col1 int64\ncol2 int64\ndtype: object\n (1.9%)
2 name object\nscore float64\nemployed bool\nkids int64\ndtype: object\n (1.9%)
2 WontImplementError: scalar value (1.9%)
208 passed (36.4%)
```
Series:
```
411 total test cases:
265 skipped (64.5%)
45 won't implement (10.9%)
19 WontImplementError: order-sensitive (42.2%)
12 (26.7%)
9 DeferredBase[*] (20.0%)
3 WontImplementError: Conversion to a non-deferred a numpy array. (6.7%)
2 WontImplementError: non-deferred column values (4.4%)
101 passed (24.6%)
```
I didn't add any logic to de-dup failures vs wont implement. It still just doesn't print the report if there are failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #12440:
URL: https://github.com/apache/beam/pull/12440
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r466760298
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -90,6 +113,16 @@ def test_series_tests(self):
result = doctests.testmod(
pd.core.series,
use_beam=False,
+ report=True,
+ wont_implement={
Review comment:
Maybe rename this to "wont_implement_ok"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r466721877
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
Review comment:
I found that I could override report_success to collect this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r464566521
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
with self._test_env.context():
- return super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ result = super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ return result
def fake_pandas_module(self):
return self._test_env.fake_pandas_module()
+ def summarize(self):
+ super(BeamDataframeDoctestRunner, self).summarize()
+ if self.failures:
+ return
Review comment:
Worth printing the stats regardless of failure?
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
Review comment:
It'd be nice to track how many actually triggered this error (and perhaps track the reasons why).
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
with self._test_env.context():
- return super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ result = super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ return result
def fake_pandas_module(self):
return self._test_env.fake_pandas_module()
+ def summarize(self):
+ super(BeamDataframeDoctestRunner, self).summarize()
+ if self.failures:
+ return
+ total_test_cases = self.skipped + self.tries
+ will_implement = total_test_cases - self.wont_implement
+ print("%d total test cases." % total_test_cases)
+ print(
+ "%d will implement, %d won't implement." %
Review comment:
Just because they're not won't implement, doesn't mean we will (e.g .they could be skipped, or they're already implemented in which case the future tense is odd). Maybe break this down as
```
Xxx total test cases.
Xxx skipped
Xxx won't implement
Yyy reason A
Yyy reason B
...
Xxx passed
```
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
with self._test_env.context():
- return super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ result = super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ return result
def fake_pandas_module(self):
return self._test_env.fake_pandas_module()
+ def summarize(self):
+ super(BeamDataframeDoctestRunner, self).summarize()
+ if self.failures:
+ return
+ total_test_cases = self.skipped + self.tries
Review comment:
The way this is implemented, self.tries includes the skipped ones (which get mutated to pass). I suppose we could instead remove them from the examples list altogether.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12440:
URL: https://github.com/apache/beam/pull/12440#issuecomment-670726192
Run PythonDocker PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r466723059
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
with self._test_env.context():
- return super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ result = super(BeamDataframeDoctestRunner, self).run(test, **kwargs)
+ return result
def fake_pandas_module(self):
return self._test_env.fake_pandas_module()
+ def summarize(self):
+ super(BeamDataframeDoctestRunner, self).summarize()
+ if self.failures:
+ return
+ total_test_cases = self.skipped + self.tries
Review comment:
Good catch, I'm not sure what I was thinking here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12440:
URL: https://github.com/apache/beam/pull/12440#discussion_r466684332
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -290,34 +298,60 @@ def to_callable(cond):
super(BeamDataframeDoctestRunner, self).__init__(
checker=_DeferrredDataframeOutputChecker(self._test_env, use_beam),
**kwargs)
+ self.skipped = 0
+ self.wont_implement = 0
def run(self, test, **kwargs):
self._checker.reset()
- if test.name in self._skip:
- for example in test.examples:
- if any(should_skip(example) for should_skip in self._skip[test.name]):
- example.source = 'pass'
- example.want = ''
for example in test.examples:
- if example.exc_msg is None:
+ if any(should_skip(example)
+ for should_skip in self._skip.get(test.name, [])):
+ example.source = 'pass'
+ example.want = ''
+ self.skipped += 1
+ elif example.exc_msg is None and any(
+ wont_implement(example)
+ for wont_implement in self._wont_implement.get(test.name, [])):
# Don't fail doctests that raise this error.
example.exc_msg = (
'apache_beam.dataframe.frame_base.WontImplementError: ...')
+ self.wont_implement += 1
Review comment:
Yeah that would be really good. Is it possible to get at that information though?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12440:
URL: https://github.com/apache/beam/pull/12440#issuecomment-670571085
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12440:
URL: https://github.com/apache/beam/pull/12440#issuecomment-667427305
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12440: [BEAM-10619] Report ratio of implemented pandas tests
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12440:
URL: https://github.com/apache/beam/pull/12440#issuecomment-670726149
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org