[GitHub] [beam] tvalentyn commented on pull request #16920: [BEAM-13993] Import beam plugins before starting SdkHarness

[GitBox <gi...@apache.org>]

tvalentyn commented on pull request #16920:
URL: https://github.com/apache/beam/pull/16920#issuecomment-1049436020


   This looks to be one of the lesser known features of Dataflow runner V1, and is marked experimental. I wasn't aware of it. @rahuliyer95 would you be interested in making it a non-experimental first-class citizen in Beam that would work for all runners? I would recommend an advertisement on dev describing your use-cases for this, settle on the expected plugin format (class vs module to remove the ambiguity or allow both if that makes sense sometimes - if so you could add rationale), adding unit tests and proper documentation, removing the 'experimental' status from the doc. You can start the conversation on dev@ with a proposal to remove the 'experimental' bit and reference your PR.
   
   This is what the import code looks like in the Dataflow Python worker:
   
   ```
     for beam_plugin in sdk_pipeline_options.get('beam_plugins', []):
       try:
         module_name, class_name = beam_plugin.rsplit('.', 1)
         _ = getattr(importlib.import_module(module_name), class_name)
         logging.info('Successfully imported beam plugin %s', beam_plugin)
       except ImportError:
         logging.warning('Failed to import beam plugin %s', beam_plugin,
                         exc_info=True)
   ```
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org