You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/02/09 00:35:42 UTC

[GitHub] [beam] robertwb commented on pull request #13877: [BEAM-11715] Recreate subtransforms for packed combiners

robertwb commented on pull request #13877:
URL: https://github.com/apache/beam/pull/13877#issuecomment-775564361


   > OK, still running into a bunch of issues:
   > 
   > * FnApiRunner doesn't actually use the sub-transforms in CombinePerKey. So I can't use it to test that the subtransforms in the packed CombinePerKey are correct. Is there any runner that actually uses the subtransforms that I can use for testing?
   
   You could disable combiner lifting in th FnApiRunner to test things out. 
   
   > * DataflowRunner has a [CombineValuesPTransformOverride](https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/python/apache_beam/runners/dataflow/ptransform_overrides.py#L155) that replaces the composite `CombineValues` with a primitive `CombineValuesReplacement`. We need a _another_ code path to mimic this replacement.
   
   Ugh. 
   
   Maybe we could simplify things by only supporting portable job submission? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org