You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by "Morand, Sebastien" <se...@veolia.com> on 2017/06/01 13:05:09 UTC
Python SDK 2.0.0
Hi,
The Python SDK has been released on may, 16th but with very little
information about new or breaking changes.
Some important questions:
1. Is Streamline mode supported now?
2. Is PubSub source fully functional and stable?
Thanks by advance,
Regards,
*Sébastien MORAND*
Team Lead Solution Architect
Technology & Operations / Digital Factory
Veolia - Group Information Systems & Technology (IS&T)
Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
Bureau 0144C (Ouest)
30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
*www.veolia.com <http://www.veolia.com>*
<http://www.veolia.com>
<https://www.facebook.com/veoliaenvironment/>
<https://www.youtube.com/user/veoliaenvironnement>
<https://www.linkedin.com/company/veolia-environnement>
<https://twitter.com/veolia>
--
--------------------------------------------------------------------------------------------
This e-mail transmission (message and any attached files) may contain
information that is proprietary, privileged and/or confidential to Veolia
Environnement and/or its affiliates and is intended exclusively for the
person(s) to whom it is addressed. If you are not the intended recipient,
please notify the sender by return e-mail and delete all copies of this
e-mail, including all attachments. Unless expressly authorized, any use,
disclosure, publication, retransmission or dissemination of this e-mail
and/or of its attachments is strictly prohibited.
Ce message electronique et ses fichiers attaches sont strictement
confidentiels et peuvent contenir des elements dont Veolia Environnement
et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
message par erreur, merci de le retourner a son emetteur et de le detruire
ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
publication, la distribution, ou la reproduction non expressement
autorisees de ce message et de ses pieces attachees sont interdites.
--------------------------------------------------------------------------------------------
Re: Python SDK 2.0.0
Posted by Dmitry Demeshchuk <dm...@postmates.com>.
Thanks for the clarification, Robert! I actually thought that Direct Runner
was already supporting streaming (since the streaming_wordcount.py example
was already in 2.0.0 release), but guess I was at most partially right.
How can I help the development or/and testing process of streaming?
Also, is there any known timeline on getting Python SDK 2.0.0 supported on
Dataflow?
On Thu, Jun 1, 2017 at 9:47 AM, Robert Bradshaw <ro...@google.com> wrote:
> Currently Beam Python is batch only, with support for the Dataflow and
> Direct (single machine) runners only. However, there is active work
> making Python run on other runners (the key term to follow is "Fn
> API") and support streaming which I expect are not too far away.
>
> On Thu, Jun 1, 2017 at 6:11 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > Hi Sébastien,
> >
> > First, the Python SDK currently uses its own runner (kind of python
> direct
> > runner). It doesn't yet leverage the existing runner as the Java SDK
> does.
> > It's something we plan.
> >
> > 1. So, assuming you use the Python Direct runner, some preparation work
> has
> > been done, like support of bundles. However, the streaming support is not
> > yet fully complete. Some pull request are in preparation around that:
> > https://issues.apache.org/jira/browse/BEAM-1265
> >
> > 2. Yes, PubSub support has been added:
> >
> > https://github.com/apache/beam/blob/master/sdks/python/
> apache_beam/io/gcp/pubsub.py
> >
> > as part of the GCP "IOs" (bigquery, bigtable, ...)
> >
> > However, I don't say how stable it is ;)
> >
> > I let the "Python guys" provide more details.
> >
> > Regards
> > JB
> >
> > On 06/01/2017 03:05 PM, Morand, Sebastien wrote:
> >>
> >> Hi,
> >>
> >> The Python SDK has been released on may, 16th but with very little
> >> information about new or breaking changes.
> >>
> >> Some important questions:
> >>
> >> 1. Is Streamline mode supported now?
> >>
> >> 2. Is PubSub source fully functional and stable?
> >>
> >> Thanks by advance,
> >> Regards,
> >>
> >> *Sébastien MORAND*
> >> Team Lead Solution Architect
> >> Technology & Operations / Digital Factory
> >> Veolia - Group Information Systems & Technology (IS&T)
> >> Cell.:+33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
> >> Bureau 0144C (Ouest)
> >> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
> >> _www.veolia.com <http://www.veolia.com>_
> >> <http://www.veolia.com>
> >> <https://www.facebook.com/veoliaenvironment/>
> >> <https://www.youtube.com/user/veoliaenvironnement>
> >> <https://www.linkedin.com/company/veolia-environnement>
> >> <https://twitter.com/veolia>
> >>
> >>
> >>
> >> ------------------------------------------------------------
> --------------------------------
> >> This e-mail transmission (message and any attached files) may contain
> >> information that is proprietary, privileged and/or confidential to
> Veolia
> >> Environnement and/or its affiliates and is intended exclusively for the
> >> person(s) to whom it is addressed. If you are not the intended
> recipient,
> >> please notify the sender by return e-mail and delete all copies of this
> >> e-mail, including all attachments. Unless expressly authorized, any use,
> >> disclosure, publication, retransmission or dissemination of this e-mail
> >> and/or of its attachments is strictly prohibited.
> >>
> >> Ce message electronique et ses fichiers attaches sont strictement
> >> confidentiels et peuvent contenir des elements dont Veolia Environnement
> >> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
> >> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
> >> message par erreur, merci de le retourner a son emetteur et de le
> detruire
> >> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
> >> publication, la distribution, ou la reproduction non expressement
> autorisees
> >> de ce message et de ses pieces attachees sont interdites.
> >>
> >> ------------------------------------------------------------
> --------------------------------
> >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
--
Best regards,
Dmitry Demeshchuk.
Re: Python SDK 2.0.0
Posted by Robert Bradshaw <ro...@google.com>.
Currently Beam Python is batch only, with support for the Dataflow and
Direct (single machine) runners only. However, there is active work
making Python run on other runners (the key term to follow is "Fn
API") and support streaming which I expect are not too far away.
On Thu, Jun 1, 2017 at 6:11 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Sébastien,
>
> First, the Python SDK currently uses its own runner (kind of python direct
> runner). It doesn't yet leverage the existing runner as the Java SDK does.
> It's something we plan.
>
> 1. So, assuming you use the Python Direct runner, some preparation work has
> been done, like support of bundles. However, the streaming support is not
> yet fully complete. Some pull request are in preparation around that:
> https://issues.apache.org/jira/browse/BEAM-1265
>
> 2. Yes, PubSub support has been added:
>
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/pubsub.py
>
> as part of the GCP "IOs" (bigquery, bigtable, ...)
>
> However, I don't say how stable it is ;)
>
> I let the "Python guys" provide more details.
>
> Regards
> JB
>
> On 06/01/2017 03:05 PM, Morand, Sebastien wrote:
>>
>> Hi,
>>
>> The Python SDK has been released on may, 16th but with very little
>> information about new or breaking changes.
>>
>> Some important questions:
>>
>> 1. Is Streamline mode supported now?
>>
>> 2. Is PubSub source fully functional and stable?
>>
>> Thanks by advance,
>> Regards,
>>
>> *Sébastien MORAND*
>> Team Lead Solution Architect
>> Technology & Operations / Digital Factory
>> Veolia - Group Information Systems & Technology (IS&T)
>> Cell.:+33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
>> Bureau 0144C (Ouest)
>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
>> _www.veolia.com <http://www.veolia.com>_
>> <http://www.veolia.com>
>> <https://www.facebook.com/veoliaenvironment/>
>> <https://www.youtube.com/user/veoliaenvironnement>
>> <https://www.linkedin.com/company/veolia-environnement>
>> <https://twitter.com/veolia>
>>
>>
>>
>> --------------------------------------------------------------------------------------------
>> This e-mail transmission (message and any attached files) may contain
>> information that is proprietary, privileged and/or confidential to Veolia
>> Environnement and/or its affiliates and is intended exclusively for the
>> person(s) to whom it is addressed. If you are not the intended recipient,
>> please notify the sender by return e-mail and delete all copies of this
>> e-mail, including all attachments. Unless expressly authorized, any use,
>> disclosure, publication, retransmission or dissemination of this e-mail
>> and/or of its attachments is strictly prohibited.
>>
>> Ce message electronique et ses fichiers attaches sont strictement
>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>> message par erreur, merci de le retourner a son emetteur et de le detruire
>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>> publication, la distribution, ou la reproduction non expressement autorisees
>> de ce message et de ses pieces attachees sont interdites.
>>
>> --------------------------------------------------------------------------------------------
>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: Python SDK 2.0.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Sébastien,
First, the Python SDK currently uses its own runner (kind of python direct
runner). It doesn't yet leverage the existing runner as the Java SDK does.
It's something we plan.
1. So, assuming you use the Python Direct runner, some preparation work has been
done, like support of bundles. However, the streaming support is not yet fully
complete. Some pull request are in preparation around that:
https://issues.apache.org/jira/browse/BEAM-1265
2. Yes, PubSub support has been added:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/pubsub.py
as part of the GCP "IOs" (bigquery, bigtable, ...)
However, I don't say how stable it is ;)
I let the "Python guys" provide more details.
Regards
JB
On 06/01/2017 03:05 PM, Morand, Sebastien wrote:
> Hi,
>
> The Python SDK has been released on may, 16th but with very little information
> about new or breaking changes.
>
> Some important questions:
>
> 1. Is Streamline mode supported now?
>
> 2. Is PubSub source fully functional and stable?
>
> Thanks by advance,
> Regards,
>
> *Sébastien MORAND*
> Team Lead Solution Architect
> Technology & Operations / Digital Factory
> Veolia - Group Information Systems & Technology (IS&T)
> Cell.:+33 7 52 66 20 81 / Direct: +33 1 85 57 71 08
> Bureau 0144C (Ouest)
> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
> _www.veolia.com <http://www.veolia.com>_
> <http://www.veolia.com>
> <https://www.facebook.com/veoliaenvironment/>
> <https://www.youtube.com/user/veoliaenvironnement>
> <https://www.linkedin.com/company/veolia-environnement> <https://twitter.com/veolia>
>
>
> --------------------------------------------------------------------------------------------
> This e-mail transmission (message and any attached files) may contain
> information that is proprietary, privileged and/or confidential to Veolia
> Environnement and/or its affiliates and is intended exclusively for the
> person(s) to whom it is addressed. If you are not the intended recipient, please
> notify the sender by return e-mail and delete all copies of this e-mail,
> including all attachments. Unless expressly authorized, any use, disclosure,
> publication, retransmission or dissemination of this e-mail and/or of its
> attachments is strictly prohibited.
>
> Ce message electronique et ses fichiers attaches sont strictement confidentiels
> et peuvent contenir des elements dont Veolia Environnement et/ou l'une de ses
> entites affiliees sont proprietaires. Ils sont donc destines a l'usage de leurs
> seuls destinataires. Si vous avez recu ce message par erreur, merci de le
> retourner a son emetteur et de le detruire ainsi que toutes les pieces
> attachees. L'utilisation, la divulgation, la publication, la distribution, ou la
> reproduction non expressement autorisees de ce message et de ses pieces
> attachees sont interdites.
> --------------------------------------------------------------------------------------------
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com