You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "shunping (via GitHub)" <gi...@apache.org> on 2025/05/09 01:25:44 UTC
[PR] Support keyed and unkeyed input for AnomalyDetection. [beam]
shunping opened a new pull request, #34897:
URL: https://github.com/apache/beam/pull/34897
- Add tests for unkeyed input.
- Optimize the case when running one non-ensemble detector (using empty string instead of computing uuid).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2864960600
Run Python_Dataframes PreCommit 3.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2866296151
Run Python_Coverage PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2866414381
Assigning reviewers:
R: @claudevdm for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #34897:
URL: https://github.com/apache/beam/pull/34897#discussion_r2082249617
##########
sdks/python/apache_beam/ml/anomaly/transforms.py:
##########
@@ -37,13 +38,16 @@
from apache_beam.ml.anomaly.specifiable import Specifiable
from apache_beam.ml.inference.base import RunInference
from apache_beam.transforms.userstate import ReadModifyWriteStateSpec
+from apache_beam.typehints.typehints import TupleConstraint
KeyT = TypeVar('KeyT')
-TempKeyT = TypeVar('TempKeyT', bound=int)
-InputT = tuple[KeyT, beam.Row]
-KeyedInputT = tuple[KeyT, tuple[TempKeyT, beam.Row]]
-KeyedOutputT = tuple[KeyT, tuple[TempKeyT, AnomalyResult]]
-OutputT = tuple[KeyT, AnomalyResult]
+TempKeyT = TypeVar('TempKeyT', bound=str)
+InputT = beam.Row
+OutputT = AnomalyResult
Review Comment:
Ack. I will keep those for now in case we want to add alternative input or output types later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2867770270
Run Python_Dataframes PreCommit 3.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2866321279
ML tests (3.9-3.11) are flaky and not related to this changes. (https://github.com/apache/beam/issues/31285)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2865004032
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #34897:
URL: https://github.com/apache/beam/pull/34897#discussion_r2082251630
##########
sdks/python/apache_beam/ml/anomaly/transforms.py:
##########
@@ -600,20 +610,43 @@ def expand(
#
# We select uuid.uuid1() for its inclusion of node information, making it
# more suitable for parallel execution environments.
- add_temp_key_fn: Callable[[InputT], KeyedInputT] \
- = lambda e: (e[0], (str(uuid.uuid1()), e[1]))
- keyed_input = (input | "Add temp key" >> beam.Map(add_temp_key_fn))
+ if isinstance(input.element_type, TupleConstraint):
+ keyed_input = input
+ else:
+ # Add a None key if the input is unkeyed.
+ keyed_input = input | beam.WithKeys(None)
Review Comment:
Sure, I agree with you on this. Let me change that to a solid value 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2866316371
Run Python_Dataframes PreCommit 3.12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #34897:
URL: https://github.com/apache/beam/pull/34897#discussion_r2082135603
##########
sdks/python/apache_beam/ml/anomaly/transforms.py:
##########
@@ -37,13 +38,16 @@
from apache_beam.ml.anomaly.specifiable import Specifiable
from apache_beam.ml.inference.base import RunInference
from apache_beam.transforms.userstate import ReadModifyWriteStateSpec
+from apache_beam.typehints.typehints import TupleConstraint
KeyT = TypeVar('KeyT')
-TempKeyT = TypeVar('TempKeyT', bound=int)
-InputT = tuple[KeyT, beam.Row]
-KeyedInputT = tuple[KeyT, tuple[TempKeyT, beam.Row]]
-KeyedOutputT = tuple[KeyT, tuple[TempKeyT, AnomalyResult]]
-OutputT = tuple[KeyT, AnomalyResult]
+TempKeyT = TypeVar('TempKeyT', bound=str)
+InputT = beam.Row
+OutputT = AnomalyResult
Review Comment:
Nit: these type aliases aren't strictly necessary given that they're specific single types, but since you use them to compose the tuple types they're fine to keep
##########
sdks/python/apache_beam/ml/anomaly/transforms.py:
##########
@@ -600,20 +610,43 @@ def expand(
#
# We select uuid.uuid1() for its inclusion of node information, making it
# more suitable for parallel execution environments.
- add_temp_key_fn: Callable[[InputT], KeyedInputT] \
- = lambda e: (e[0], (str(uuid.uuid1()), e[1]))
- keyed_input = (input | "Add temp key" >> beam.Map(add_temp_key_fn))
+ if isinstance(input.element_type, TupleConstraint):
+ keyed_input = input
+ else:
+ # Add a None key if the input is unkeyed.
+ keyed_input = input | beam.WithKeys(None)
Review Comment:
Nit: I'd shy away from a `None` key and use a real-valued constant. Once again not wholly necessary, just a thought
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2864960859
Run Python_Dataframes PreCommit 3.12
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey merged PR #34897:
URL: https://github.com/apache/beam/pull/34897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2864933917
cc @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AnomalyDetection] Support unkeyed input for the transform. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #34897:
URL: https://github.com/apache/beam/pull/34897#issuecomment-2866305213
Run Python_PVR_Flink PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org