You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "hjtran (via GitHub)" <gi...@apache.org> on 2025/07/15 23:28:20 UTC

[PR] Combine sideinputs [beam]

hjtran opened a new pull request, #35601:
URL: https://github.com/apache/beam/pull/35601

   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35601:
URL: https://github.com/apache/beam/pull/35601#discussion_r2210788701


##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2917,6 +2917,22 @@ class CombinePerKey(PTransformWithSideInputs):
   Returns:
     A PObject holding the result of the combine operation.
   """
+  def __new__(cls, *args, **kwargs):
+    def has_side_inputs():
+      return (
+          any(isinstance(arg, pvalue.AsSideInput) for arg in args) or
+          any(isinstance(arg, pvalue.AsSideInput) for arg in kwargs.values()))
+
+    if has_side_inputs():
+      # If the CombineFn has deferred side inputs, the python SDK
+      # doesn't implement it.
+      # Use a ParDo-based CombinePerKey instead.
+      from apache_beam.runners.direct.helper_transforms import \

Review Comment:
   Is this meant to only apply to pipelines running on direct runner?
   
   It seems strange to import transforms from direct runner into transforms.core.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35601:
URL: https://github.com/apache/beam/pull/35601#discussion_r2222828849


##########
sdks/python/apache_beam/transforms/combiners.py:
##########
@@ -985,3 +986,126 @@ def merge_accumulators(self, accumulators):
 
   def extract_output(self, accumulator):
     return accumulator[0]
+
+
+class LiftedCombinePerKey(core.PTransform):

Review Comment:
   That makes sense, thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #35601:
URL: https://github.com/apache/beam/pull/35601


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on code in PR #35601:
URL: https://github.com/apache/beam/pull/35601#discussion_r2222824780


##########
sdks/python/apache_beam/transforms/combiners.py:
##########
@@ -985,3 +986,126 @@ def merge_accumulators(self, accumulators):
 
   def extract_output(self, accumulator):
     return accumulator[0]
+
+
+class LiftedCombinePerKey(core.PTransform):

Review Comment:
   I don't think so. I don't think we want to advertise this since I dont think there are cases where a user should use this CPK specifically over the general CPK



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #35601:
URL: https://github.com/apache/beam/pull/35601#issuecomment-3082191354

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/35601?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `95.89041%` with `3 lines` in your changes missing coverage. Please review.
   > Project coverage is 56.58%. Comparing base [(`84d423f`)](https://app.codecov.io/gh/apache/beam/commit/84d423fef34725e5801ff575a6836bb827d6e763?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`42d5929`)](https://app.codecov.io/gh/apache/beam/commit/42d5929a35c361cb90bd183271d3c1ec64847af1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 2 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/35601?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [sdks/python/apache\_beam/transforms/combiners.py](https://app.codecov.io/gh/apache/beam/pull/35601?src=pr&el=tree&filepath=sdks%2Fpython%2Fapache_beam%2Ftransforms%2Fcombiners.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | 95.31% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/35601?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #35601      +/-   ##
   ============================================
   + Coverage     56.56%   56.58%   +0.02%     
     Complexity     3376     3376              
   ============================================
     Files          1205     1204       -1     
     Lines        183574   183701     +127     
     Branches       3458     3458              
   ============================================
   + Hits         103845   103955     +110     
   - Misses        76410    76427      +17     
     Partials       3319     3319              
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/beam/pull/35601/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [python](https://app.codecov.io/gh/apache/beam/pull/35601/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.71% <95.89%> (+<0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/35601?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on PR #35601:
URL: https://github.com/apache/beam/pull/35601#issuecomment-3085877038

   R: @damccorm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35601:
URL: https://github.com/apache/beam/pull/35601#discussion_r2222622732


##########
sdks/python/apache_beam/transforms/combiners.py:
##########
@@ -985,3 +986,126 @@ def merge_accumulators(self, accumulators):
 
   def extract_output(self, accumulator):
     return accumulator[0]
+
+
+class LiftedCombinePerKey(core.PTransform):

Review Comment:
   Should this be part of `__all__` for imports?



##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2927,6 +2927,22 @@ class CombinePerKey(PTransformWithSideInputs):
   Returns:
     A PObject holding the result of the combine operation.
   """
+  def __new__(cls, *args, **kwargs):
+    def has_side_inputs():
+      return (
+          any(isinstance(arg, pvalue.AsSideInput) for arg in args) or
+          any(isinstance(arg, pvalue.AsSideInput) for arg in kwargs.values()))
+
+    if has_side_inputs():
+      # If the CombineFn has deferred side inputs, the python SDK
+      # doesn't implement it.
+      # Use a ParDo-based CombinePerKey instead.
+      from apache_beam.transforms.combiners import \
+        LiftedCombinePerKey
+      combine_fn, *args = args
+      return LiftedCombinePerKey(combine_fn, args, kwargs)

Review Comment:
   I'd probably vote to leave it as is. We could warn and have a kwarg to silence the warning (or something similar), but I tend to agree that this is a rare case; while the perf implication is non-obvious, adding side inputs basically always comes with some perf cost



##########
sdks/python/apache_beam/transforms/combiners.py:
##########
@@ -985,3 +986,126 @@ def merge_accumulators(self, accumulators):
 
   def extract_output(self, accumulator):
     return accumulator[0]
+
+
+class LiftedCombinePerKey(core.PTransform):
+  """An implementation of CombinePerKey that does mapper-side pre-combining.
+
+  This shouldn't generally be used directly except for use-cases where a
+  runner doesn't support CombinePerKey. This implementation manually implements
+  a CombinePerKey using ParDos, as opposed to runner implementations which may
+  use a more efficient implementation.
+  """
+  def __init__(self, combine_fn, args, kwargs):
+    side_inputs = _pack_side_inputs(args, kwargs)
+    self._side_inputs: dict = side_inputs
+    if not isinstance(combine_fn, core.CombineFn):
+      combine_fn = core.CombineFn.from_callable(combine_fn)
+    self._combine_fn = combine_fn
+
+  def expand(self, pcoll):
+    return (
+        pcoll
+        | core.ParDo(
+            PartialGroupByKeyCombiningValues(self._combine_fn),
+            **self._side_inputs)
+        | core.GroupByKey()
+        | core.ParDo(FinishCombine(self._combine_fn), **self._side_inputs))
+
+
+def _pack_side_inputs(side_input_args, side_input_kwargs):
+  if len(side_input_args) >= 10:
+    # If we have more than 10 side inputs, we can't use the
+    # _side_input_arg_{i} as our keys since they won't sort
+    # correctly. Just punt for now, more than 10 args probably
+    # doesn't happen often.
+    raise NotImplementedError
+  side_inputs = {}
+  for i, si in enumerate(side_input_args):
+    side_inputs[f'_side_input_arg_{i}'] = si
+  for k, v in side_input_kwargs.items():
+    side_inputs[k] = v
+  return side_inputs
+
+
+def _unpack_side_inputs(side_inputs):
+  side_input_args = []
+  side_input_kwargs = {}
+  for k, v in sorted(side_inputs.items(), key=lambda x: x[0]):
+    if k.startswith('_side_input_arg_'):
+      side_input_args.append(v)
+    else:
+      side_input_kwargs[k] = v
+  return side_input_args, side_input_kwargs
+
+
+class PartialGroupByKeyCombiningValues(core.DoFn):

Review Comment:
   ```suggestion
   class _PartialGroupByKeyCombiningValues(core.DoFn):
   ```
   
   We never expect people to use this (or `FinishCombine`) directly, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on code in PR #35601:
URL: https://github.com/apache/beam/pull/35601#discussion_r2214530015


##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2927,6 +2927,22 @@ class CombinePerKey(PTransformWithSideInputs):
   Returns:
     A PObject holding the result of the combine operation.
   """
+  def __new__(cls, *args, **kwargs):
+    def has_side_inputs():
+      return (
+          any(isinstance(arg, pvalue.AsSideInput) for arg in args) or
+          any(isinstance(arg, pvalue.AsSideInput) for arg in kwargs.values()))
+
+    if has_side_inputs():
+      # If the CombineFn has deferred side inputs, the python SDK
+      # doesn't implement it.
+      # Use a ParDo-based CombinePerKey instead.
+      from apache_beam.transforms.combiners import \
+        LiftedCombinePerKey
+      combine_fn, *args = args
+      return LiftedCombinePerKey(combine_fn, args, kwargs)

Review Comment:
   I'm a bit split. I agree that users will probably not realize this subtle impact from using deferred side inputs with their combiner, but I think ideally there'd also be a way for a user to understand the risk and dismiss the warning. 
   
   I'm okay with whichever though, let me know if you still think we should have the warning. I suspect that in most cases even knowing that there may be a reduction in performance that it'd be difficult to actually observe (but maybe I'm underestimating the `operations.py` combine optimizations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on PR #35601:
URL: https://github.com/apache/beam/pull/35601#issuecomment-3078577011

   I could use some help debugging the failing unit test. It passes locally for me but seems to fail on the GH Runner for some reason due to a grpc timeout:
   ```
   grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
   	status = StatusCode.DEADLINE_EXCEEDED
   	details = "Deadline Exceeded"
   	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2025-07-16T11:40:51.901440152+00:00"}"
   >
   self = <apache_beam.transforms.combiners_test.CombinerWithSideInputs testMethod=test_cpk_with_windows>
   
       def test_cpk_with_windows(self):
         # With global window side input
   >     with TestPipeline() as p:
   
   apache_beam/transforms/combiners_test.py:1173: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   target/.tox-py311-cloud/py311-cloud/lib/python3.11/site-packages/apache_beam/pipeline.py:661: in __exit__
       self.result = self.run()
   target/.tox-py311-cloud/py311-cloud/lib/python3.11/site-packages/apache_beam/testing/test_pipeline.py:119: in run
       state = result.wait_until_finish()
   target/.tox-py311-cloud/py311-cloud/lib/python3.11/site-packages/apache_beam/runners/portability/portable_runner.py:568: in wait_until_finish
       raise self._runtime_exception
   target/.tox-py311-cloud/py311-cloud/lib/python3.11/site-packages/apache_beam/runners/portability/portable_runner.py:574: in _observe_state
       for state_response in self._state_stream:
   target/.tox-py311-cloud/py311-cloud/lib/python3.11/site-packages/grpc/_channel.py:543: in __next__
       return self._next()
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   self = <_MultiThreadedRendezvous of RPC that terminated with:
   	status = StatusCode.DEADLINE_EXCEEDED
   	details = "Deadline Exc...ved from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2025-07-16T11:40:51.901440152+00:00"}"
   >
   
       def _next(self) -> Any:
           with self._state.condition:
               if self._state.code is None:
                   event_handler = _event_handler(
                       self._state, self._response_deserializer
                   )
                   self._state.due.add(cygrpc.OperationType.receive_message)
                   operating = self._call.operate(
                       (cygrpc.ReceiveMessageOperation(_EMPTY_FLAGS),),
                       event_handler,
                   )
                   if not operating:
                       self._state.due.remove(cygrpc.OperationType.receive_message)
               elif self._state.code is grpc.StatusCode.OK:
                   raise StopIteration()
               else:
                   raise self
       
               def _response_ready():
                   return self._state.response is not None or (
                       cygrpc.OperationType.receive_message not in self._state.due
                       and self._state.code is not None
                   )
       
               _common.wait(self._state.condition.wait, _response_ready)
               if self._state.response is not None:
                   response = self._state.response
                   self._state.response = None
                   return response
               elif cygrpc.OperationType.receive_message not in self._state.due:
                   if self._state.code is grpc.StatusCode.OK:
                       raise StopIteration()
                   elif self._state.code is not None:
   >                   raise self
   E                   grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
   E                   	status = StatusCode.DEADLINE_EXCEEDED
   E                   	details = "Deadline Exceeded"
   E                   	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2025-07-16T11:40:51.901440152+00:00"}"
   E                   >
   
   target/.tox-py311-cloud/py311-cloud/lib/python3.11/site-packages/grpc/_channel.py:969: _MultiThreadedRendezvous
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #35601:
URL: https://github.com/apache/beam/pull/35601#discussion_r2214501933


##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2927,6 +2927,22 @@ class CombinePerKey(PTransformWithSideInputs):
   Returns:
     A PObject holding the result of the combine operation.
   """
+  def __new__(cls, *args, **kwargs):
+    def has_side_inputs():
+      return (
+          any(isinstance(arg, pvalue.AsSideInput) for arg in args) or
+          any(isinstance(arg, pvalue.AsSideInput) for arg in kwargs.values()))
+
+    if has_side_inputs():
+      # If the CombineFn has deferred side inputs, the python SDK
+      # doesn't implement it.
+      # Use a ParDo-based CombinePerKey instead.
+      from apache_beam.transforms.combiners import \
+        LiftedCombinePerKey
+      combine_fn, *args = args
+      return LiftedCombinePerKey(combine_fn, args, kwargs)

Review Comment:
   Maybe we should log a warning here that runner implemented CBK has been replaced due to side inputs and may affect performance? Users might not be aware this is happening.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on code in PR #35601:
URL: https://github.com/apache/beam/pull/35601#discussion_r2211736009


##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2917,6 +2917,22 @@ class CombinePerKey(PTransformWithSideInputs):
   Returns:
     A PObject holding the result of the combine operation.
   """
+  def __new__(cls, *args, **kwargs):
+    def has_side_inputs():
+      return (
+          any(isinstance(arg, pvalue.AsSideInput) for arg in args) or
+          any(isinstance(arg, pvalue.AsSideInput) for arg in kwargs.values()))
+
+    if has_side_inputs():
+      # If the CombineFn has deferred side inputs, the python SDK
+      # doesn't implement it.
+      # Use a ParDo-based CombinePerKey instead.
+      from apache_beam.runners.direct.helper_transforms import \

Review Comment:
   No this should apply to all runners.
   
   I've moved it into `apache_beam.transforms.combiners`. Let me know if you think it should go somewhere else



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35601:
URL: https://github.com/apache/beam/pull/35601#issuecomment-3085879275

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Python] Implement combiner deferred side inputs [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35601:
URL: https://github.com/apache/beam/pull/35601#issuecomment-3078434166

   Assigning reviewers:
   
   R: @claudevdm for label python.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org