You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/15 00:35:33 UTC
[GitHub] [beam] robertwb opened a new pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
robertwb opened a new pull request #12259:
URL: https://github.com/apache/beam/pull/12259
Few tests are actually run at this point, but we can enable them as implementations are added and/or validated.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12259:
URL: https://github.com/apache/beam/pull/12259#discussion_r456143877
##########
File path: sdks/python/apache_beam/dataframe/doctests_test.py
##########
@@ -63,6 +63,20 @@
ignored result
'''
+ERROR_RAISING_NAME_ERROR_TESTS = '''
+>>> import apache_beam
+>>> raise apache_beam.dataframe.frame_base.WontImplementError('anything')
+ignored exception
+>>> raise NameError
+ignored exception
+>>> undefined_name
+ignored exception
+>>> 2 + 2
Review comment:
I don't want to ignore the whole rest of the docstring, which often has other examples.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a change in pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #12259:
URL: https://github.com/apache/beam/pull/12259#discussion_r456143684
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -240,13 +256,31 @@ class BeamDataframeDoctestRunner(doctest.DocTestRunner):
"""A Doctest runner suitable for replacing the `pd` module with one backed
by beam.
"""
- def __init__(self, env, use_beam=True, **kwargs):
+ def __init__(self, env, use_beam=True, skip=None, **kwargs):
self._test_env = env
+
+ def to_callable(cond):
+ if cond == '*':
Review comment:
I started out supporting regex, but realized it'd be painful to have to escape things like parentheses, brackets, dots, etc. in code snippets. (This is internal, so we could always add this later. Some ideas I've had would be to special case `"/regex-here/"` or pass through callables to allow `re.compile("regex").match`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
robertwb merged pull request #12259:
URL: https://github.com/apache/beam/pull/12259
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb closed pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
robertwb closed pull request #12259:
URL: https://github.com/apache/beam/pull/12259
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12259:
URL: https://github.com/apache/beam/pull/12259#issuecomment-658956391
R: @ibzib Could you take a look at this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12259:
URL: https://github.com/apache/beam/pull/12259#issuecomment-660238202
Thanks for the review! The failing DisplayDataTest tests are completely unrelated, merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #12259:
URL: https://github.com/apache/beam/pull/12259#discussion_r456160558
##########
File path: sdks/python/apache_beam/dataframe/doctests_test.py
##########
@@ -63,6 +63,20 @@
ignored result
'''
+ERROR_RAISING_NAME_ERROR_TESTS = '''
+>>> import apache_beam
+>>> raise apache_beam.dataframe.frame_base.WontImplementError('anything')
+ignored exception
+>>> raise NameError
+ignored exception
+>>> undefined_name
+ignored exception
+>>> 2 + 2
Review comment:
That sounds fine. If we encounter an issue like this example simulates, I assume we can always manually add the test to the skip map.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #12259:
URL: https://github.com/apache/beam/pull/12259#discussion_r456073212
##########
File path: sdks/python/apache_beam/dataframe/doctests.py
##########
@@ -240,13 +256,31 @@ class BeamDataframeDoctestRunner(doctest.DocTestRunner):
"""A Doctest runner suitable for replacing the `pd` module with one backed
by beam.
"""
- def __init__(self, env, use_beam=True, **kwargs):
+ def __init__(self, env, use_beam=True, skip=None, **kwargs):
self._test_env = env
+
+ def to_callable(cond):
+ if cond == '*':
Review comment:
Should we support regex matching here, or is that overkill?
##########
File path: sdks/python/apache_beam/dataframe/doctests_test.py
##########
@@ -63,6 +63,20 @@
ignored result
'''
+ERROR_RAISING_NAME_ERROR_TESTS = '''
+>>> import apache_beam
+>>> raise apache_beam.dataframe.frame_base.WontImplementError('anything')
+ignored exception
+>>> raise NameError
+ignored exception
+>>> undefined_name
+ignored exception
+>>> 2 + 2
Review comment:
Why should running one line successfully reset `_seen_wont_implement`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12259: [BEAM-9561] Run Pandas tests on Beam.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12259:
URL: https://github.com/apache/beam/pull/12259#issuecomment-659743917
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org