You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by OrielResearch Eila Arich-Landkof <ei...@orielresearch.org> on 2018/03/12 13:26:41 UTC
Dynamic output - Python
Hello all,
I would like to print each PCollection element to a different folder on
gs://.../
the folder name is not available in advance, but it is available on the
pCollection data (or can be calculated prior to printing).
My question are:
- Is it possible to use the folder path as a parameter to the
beam.io.WriteToText?
- Can you refer me to a code example where dynamic output (or similar) is
being used in Python?
Thanks,
Eila
--
Eila
www.orielresearch.org
https://www.meetup.com/Deep-Learning-In-Production/
Re: Dynamic output - Python
Posted by Chamikara Jayalath <ch...@google.com>.
Hi Eila,
Python textio currently does not support dynamic writes. So, to achieve
this, you'll have to implement a custom DoFn that writes data to dynamic
destinations.
Thanks,
Cham
On Mon, Mar 12, 2018 at 6:26 AM OrielResearch Eila Arich-Landkof <
eila@orielresearch.org> wrote:
> Hello all,
>
> I would like to print each PCollection element to a different folder on
> gs://.../
> the folder name is not available in advance, but it is available on the
> pCollection data (or can be calculated prior to printing).
> My question are:
> - Is it possible to use the folder path as a parameter to the
> beam.io.WriteToText?
> - Can you refer me to a code example where dynamic output (or similar) is
> being used in Python?
> Thanks,
> Eila
>
> --
> Eila
> www.orielresearch.org
> https://www.meetup.com/Deep-Learning-In-Production/
>