You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/15 17:18:47 UTC
[GitHub] [beam] monicadsong opened a new pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
monicadsong opened a new pull request #12851:
URL: https://github.com/apache/beam/pull/12851
If element for ApproximateUniqueCombineFn.add_input() is numpy type, convert to python type.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] monicadsong commented on pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
monicadsong commented on pull request #12851:
URL: https://github.com/apache/beam/pull/12851#issuecomment-692856430
R: @tvalentyn
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] monicadsong commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
monicadsong commented on a change in pull request #12851:
URL: https://github.com/apache/beam/pull/12851#discussion_r488865232
##########
File path: sdks/python/apache_beam/transforms/stats_test.py
##########
@@ -89,6 +90,12 @@ def setUp(self):
None,
0.1,
'assert:global_by_error_with_large_population'),
+ (
+ 'numpy_input_data',
+ np.array(range(10)),
Review comment:
I can allow multidimensional numpy arrays as input if you want.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] monicadsong closed pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
monicadsong closed pull request #12851:
URL: https://github.com/apache/beam/pull/12851
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12851:
URL: https://github.com/apache/beam/pull/12851#issuecomment-692881256
# [Codecov](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=h1) Report
> Merging [#12851](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.10%`.
> The diff coverage is `87.12%`.
[](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12851 +/- ##
==========================================
- Coverage 82.38% 82.27% -0.11%
==========================================
Files 451 451
Lines 53775 53730 -45
==========================================
- Hits 44303 44207 -96
- Misses 9472 9523 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.14% <50.00%> (-0.02%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | |
| [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.47% <88.00%> (+3.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+1.27%)` | :arrow_up: |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=footer). Last update [63f54fd...4f71b9b](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12851:
URL: https://github.com/apache/beam/pull/12851#issuecomment-692881256
# [Codecov](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=h1) Report
> Merging [#12851](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/803efdd931699c24e60350e5ccf6e54482f5916f?el=desc) will **decrease** coverage by `0.10%`.
> The diff coverage is `87.12%`.
[](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12851 +/- ##
==========================================
- Coverage 82.38% 82.27% -0.11%
==========================================
Files 451 451
Lines 53775 53730 -45
==========================================
- Hits 44303 44207 -96
- Misses 9472 9523 +51
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...n/apache\_beam/runners/dataflow/dataflow\_metrics.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19tZXRyaWNzLnB5) | `74.32% <33.33%> (-0.85%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `92.14% <50.00%> (-0.02%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frame\_base.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lX2Jhc2UucHk=) | `84.70% <57.14%> (ø)` | |
| [sdks/python/apache\_beam/dataframe/doctests.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2RvY3Rlc3RzLnB5) | `96.83% <83.33%> (-0.93%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/opcodes.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL29wY29kZXMucHk=) | `87.65% <83.33%> (-0.35%)` | :arrow_down: |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.20% <86.66%> (-3.41%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/stats.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9zdGF0cy5weQ==) | `90.47% <88.00%> (+3.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/coders/row\_coder.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL3Jvd19jb2Rlci5weQ==) | `94.36% <90.90%> (-0.64%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/frames.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2ZyYW1lcy5weQ==) | `90.48% <100.00%> (+0.07%)` | :arrow_up: |
| [sdks/python/apache\_beam/io/gcp/pubsub.py](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL3B1YnN1Yi5weQ==) | `93.58% <100.00%> (+1.27%)` | :arrow_up: |
| ... and [31 more](https://codecov.io/gh/apache/beam/pull/12851/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=footer). Last update [63f54fd...4f71b9b](https://codecov.io/gh/apache/beam/pull/12851?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] monicadsong commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
monicadsong commented on a change in pull request #12851:
URL: https://github.com/apache/beam/pull/12851#discussion_r488863704
##########
File path: sdks/python/apache_beam/transforms/stats_test.py
##########
@@ -89,6 +90,12 @@ def setUp(self):
None,
0.1,
'assert:global_by_error_with_large_population'),
+ (
+ 'numpy_input_data',
+ np.array(range(10)),
Review comment:
No. For numpy input I am assuming that for ApproxUniqueCombineFn.add_input(self, accumulator, element), element is a numpy scalar (not a numpy array).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12851:
URL: https://github.com/apache/beam/pull/12851#discussion_r488853550
##########
File path: sdks/python/apache_beam/transforms/stats_test.py
##########
@@ -89,6 +90,12 @@ def setUp(self):
None,
0.1,
'assert:global_by_error_with_large_population'),
+ (
+ 'numpy_input_data',
+ np.array(range(10)),
Review comment:
Do multidimensional arrays also work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] monicadsong commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
monicadsong commented on a change in pull request #12851:
URL: https://github.com/apache/beam/pull/12851#discussion_r489074298
##########
File path: sdks/python/apache_beam/transforms/stats_test.py
##########
@@ -89,6 +90,12 @@ def setUp(self):
None,
0.1,
'assert:global_by_error_with_large_population'),
+ (
+ 'numpy_input_data',
+ np.array(range(10)),
Review comment:
Ah sorry, just saw this. I didn't like the assumption I made about the type of elements in the PCollection either, and the use case you mention is definitely unsupported with this draft edit but would not alert the user if they made that mistake.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] monicadsong commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
monicadsong commented on a change in pull request #12851:
URL: https://github.com/apache/beam/pull/12851#discussion_r488863704
##########
File path: sdks/python/apache_beam/transforms/stats_test.py
##########
@@ -89,6 +90,12 @@ def setUp(self):
None,
0.1,
'assert:global_by_error_with_large_population'),
+ (
+ 'numpy_input_data',
+ np.array(range(10)),
Review comment:
No. For numpy input I am assuming that for ApproxUniqueCombineFn.add_input(self, accumulator, element), element is a numpy scalar (not a numpy array).
However, if the element is a multidimensional numpy array, then calling element.item(0) returns the value at the first index of the array as a python type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12851:
URL: https://github.com/apache/beam/pull/12851#discussion_r488878863
##########
File path: sdks/python/apache_beam/transforms/stats_test.py
##########
@@ -89,6 +90,12 @@ def setUp(self):
None,
0.1,
'assert:global_by_error_with_large_population'),
+ (
+ 'numpy_input_data',
+ np.array(range(10)),
Review comment:
I see. At this point I am curious about how users use ApproximateUnique.
The use-case you are addressing is when users pass a PCollection of elements, where each element is a single value stored in a numpy datatype. Since it's a single value, we convert it to a scalar. Is that right?
I am wondering if there is also a use-case when users pass a PCollection of numpy arrays (perhaps erroneously). In which case the current combiner will pick the first element of the array, so approximation may not be very precise. I wonder if a more precise implementation makes sense or this use-case is not common.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] monicadsong commented on a change in pull request #12851: [BEAM-10900] add ability for ApproximateUniqueCombineFn to handle numpy input data
Posted by GitBox <gi...@apache.org>.
monicadsong commented on a change in pull request #12851:
URL: https://github.com/apache/beam/pull/12851#discussion_r488863704
##########
File path: sdks/python/apache_beam/transforms/stats_test.py
##########
@@ -89,6 +90,12 @@ def setUp(self):
None,
0.1,
'assert:global_by_error_with_large_population'),
+ (
+ 'numpy_input_data',
+ np.array(range(10)),
Review comment:
No. For numpy input I am assuming that for ApproxUniqueCombineFn.add_input(self, accumulator, element), element is a numpy scalar or numpy array with 1 element.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org