[GitHub] [beam] kennknowles opened a new issue, #18464: Allow using side inputs as ValueProviders

[GitBox <gi...@apache.org>]

kennknowles opened a new issue, #18464:
URL: https://github.com/apache/beam/issues/18464

   Allow dynamically computed information in a pipeline which is materialized as a side input to be used in the same places as ValueProviders.
   
   This could solve the problem mentioned in [https://stackoverflow.com/questions/45649244/google-cloud-dataflow-apache-beam-can-i-use-sideinputs-with-textio-write](https://stackoverflow.com/questions/45649244/google-cloud-dataflow-apache-beam-can-i-use-sideinputs-with-textio-write) if TextSink supported ValueProviders for headers/footers.
   
   
   
   
   Imported from Jira [BEAM-2771](https://issues.apache.org/jira/browse/BEAM-2771). Original Jira may contain additional context.
   Reported by: lcwik.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] sharvi1 commented on issue #18464: Allow using side inputs as ValueProviders

[GitBox <gi...@apache.org>]

sharvi1 commented on issue #18464:
URL: https://github.com/apache/beam/issues/18464#issuecomment-1153176323

   Hi, I'm trying to pass as Sideinput a value from a step in the pipeline to another step but not able to see the results when I try to print it, If I pass a hardcoded value then it works but I'd like to pass some variable from previous step, How can I achieve this in Google Dataflow?
   
   This is the first step with outputs 
   `| 'Fetch File Name' >> beam.ParDo(FetchFileName('start')).with_outputs('sdata',main='gdata')`
   
   Then I try to pass a sideinput in another pipeline step like so
   `| 'Parallel Transform of Data' >> beam.ParDo(AdjustObject(), beam.pvalue.AsSingleton(lines['sdata']), 'test')`
   This doesn't seem to work in Google Dataflow but works in an environment like Google Colab when I test it
   
   Is this a beam issue or does Google Dataflow not support Side Inputs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org