You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Lukasz Cwik <lc...@google.com> on 2018/06/21 14:53:34 UTC
Re: Using user developped source in streamline python
+dev@beam.apache.org
Python streaming custom source support will be available via
SplittableDoFn. It is actively being worked on by a few contributors but to
my knowledge there is no roadmap yet for having support for this for
Dataflow.
On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand <
sebastien.morand@veolia.com> wrote:
> Hi,
>
> We need to setup streaming dataflow in python using developed source (in
> the opposite of native source) for Cloud SQL integration and Firestore
> integration.
>
> This is currently not supported as far as I understood the documentation,
> can you confirm? Any roadmap on the topic?
>
> Thanks by advance,
> Regards,
>
> *Sébastien MORAND*
> Team Lead Solution Architect
> Technology & Operations / Digital Factory
> Veolia - Group Information Systems & Technology (IS&T)
> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
> Bureau 0144C (Ouest)
> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
> *www.veolia.com <http://www.veolia.com>*
> <http://www.veolia.com>
> <https://www.facebook.com/veoliaenvironment/>
> <https://www.youtube.com/user/veoliaenvironnement>
> <https://www.linkedin.com/company/veolia-environnement>
> <https://twitter.com/veolia>
>
>
>
> --------------------------------------------------------------------------------------------
> This e-mail transmission (message and any attached files) may contain
> information that is proprietary, privileged and/or confidential to Veolia
> Environnement and/or its affiliates and is intended exclusively for the
> person(s) to whom it is addressed. If you are not the intended recipient,
> please notify the sender by return e-mail and delete all copies of this
> e-mail, including all attachments. Unless expressly authorized, any use,
> disclosure, publication, retransmission or dissemination of this e-mail
> and/or of its attachments is strictly prohibited.
>
> Ce message electronique et ses fichiers attaches sont strictement
> confidentiels et peuvent contenir des elements dont Veolia Environnement
> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
> message par erreur, merci de le retourner a son emetteur et de le detruire
> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
> publication, la distribution, ou la reproduction non expressement
> autorisees de ce message et de ses pieces attachees sont interdites.
>
> --------------------------------------------------------------------------------------------
>
Re: Using user developped source in streamline python
Posted by Sebastien Morand <se...@veolia.com>.
Hi
I'm completly fine to contribute, let's start with Cloud SQL and Firestore.
Get me right: I've already written a working writer and sink. So how to
make them native so they will be able to be supported in the streamline
pipeline? Just inheritance from the native_io?
My sinks inherit from iobase.Sink and my Writers from iobase.Writer.
Regards,
*Sébastien MORAND*
Team Lead Solution Architect
Technology & Operations / Digital Factory
Veolia - Group Information Systems & Technology (IS&T)
Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
Bureau 0144C (Ouest)
30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
*www.veolia.com <http://www.veolia.com>*
<http://www.veolia.com>
<https://www.facebook.com/veoliaenvironment/>
<https://www.youtube.com/user/veoliaenvironnement>
<https://www.linkedin.com/company/veolia-environnement>
<https://twitter.com/veolia>
On Wed, 27 Jun 2018 at 22:30, Ismaël Mejía <ie...@gmail.com> wrote:
> It seems like a nice opportunity also to contribute them to the project in
> case you are able to work on them.
> Don't hesitate to contact us or ask for help if needed.
>
>
> On Wed, Jun 27, 2018 at 4:45 PM Ahmet Altay <al...@google.com> wrote:
>
>> Hi Sébastien,
>>
>> Currently there is no work in progress for including the write transforms
>> for the locations you listed. You could develop your own version if
>> interested. Please see WriteToBigquery transform [1] for reference.
>>
>> Ahmet
>>
>> [1]
>> https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dcb322875e1b4dca/sdks/python/apache_beam/io/gcp/bigquery.py#L1287
>>
>> On Wed, Jun 27, 2018 at 2:48 AM, Sebastien Morand <
>> sebastien.morand@veolia.com> wrote:
>>
>>> Hi,
>>>
>>> Thanks for your answer.
>>>
>>> Ok looking forward and ready to test alpha on this. Because we have
>>> actually some use cases to send data to CloudSQL or Spanner or BigTable or
>>> Firestore. As far as I read the documentation, there is no native support
>>> for them and so we have already implemented a custom source support.
>>>
>>> So is there any work in progress for any of the 4 sinks above for native
>>> support in the python SDK?
>>>
>>> Regards,
>>>
>>> *Sébastien MORAND*
>>> Team Lead Solution Architect
>>> Technology & Operations / Digital Factory
>>> Veolia - Group Information Systems & Technology (IS&T)
>>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>>> Bureau 0144C (Ouest)
>>> 30, rue Madeleine-Vionnet
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>> -
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
>>> Aubervilliers, France
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>> *www.veolia.com <http://www.veolia.com>*
>>> <http://www.veolia.com>
>>> <https://www.facebook.com/veoliaenvironment/>
>>> <https://www.youtube.com/user/veoliaenvironnement>
>>> <https://www.linkedin.com/company/veolia-environnement>
>>> <https://twitter.com/veolia>
>>>
>>>
>>> On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> +dev@beam.apache.org
>>>>
>>>> Python streaming custom source support will be available via
>>>> SplittableDoFn. It is actively being worked on by a few contributors but to
>>>> my knowledge there is no roadmap yet for having support for this for
>>>> Dataflow.
>>>>
>>>>
>>>> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand <
>>>> sebastien.morand@veolia.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We need to setup streaming dataflow in python using developed source
>>>>> (in the opposite of native source) for Cloud SQL integration and Firestore
>>>>> integration.
>>>>>
>>>>> This is currently not supported as far as I understood the
>>>>> documentation, can you confirm? Any roadmap on the topic?
>>>>>
>>>>> Thanks by advance,
>>>>> Regards,
>>>>>
>>>>> *Sébastien MORAND*
>>>>> Team Lead Solution Architect
>>>>> Technology & Operations / Digital Factory
>>>>> Veolia - Group Information Systems & Technology (IS&T)
>>>>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>>>>> Bureau 0144C (Ouest)
>>>>> 30, rue Madeleine-Vionnet
>>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>>>> -
>>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
>>>>> Aubervilliers, France
>>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>>>> *www.veolia.com <http://www.veolia.com>*
>>>>> <http://www.veolia.com>
>>>>> <https://www.facebook.com/veoliaenvironment/>
>>>>> <https://www.youtube.com/user/veoliaenvironnement>
>>>>> <https://www.linkedin.com/company/veolia-environnement>
>>>>> <https://twitter.com/veolia>
>>>>>
>>>>>
>>>>>
>>>>> --------------------------------------------------------------------------------------------
>>>>> This e-mail transmission (message and any attached files) may contain
>>>>> information that is proprietary, privileged and/or confidential to Veolia
>>>>> Environnement and/or its affiliates and is intended exclusively for the
>>>>> person(s) to whom it is addressed. If you are not the intended recipient,
>>>>> please notify the sender by return e-mail and delete all copies of this
>>>>> e-mail, including all attachments. Unless expressly authorized, any use,
>>>>> disclosure, publication, retransmission or dissemination of this e-mail
>>>>> and/or of its attachments is strictly prohibited.
>>>>>
>>>>> Ce message electronique et ses fichiers attaches sont strictement
>>>>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>>>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>>>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>>>>> message par erreur, merci de le retourner a son emetteur et de le detruire
>>>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>>>>> publication, la distribution, ou la reproduction non expressement
>>>>> autorisees de ce message et de ses pieces attachees sont interdites.
>>>>>
>>>>> --------------------------------------------------------------------------------------------
>>>>>
>>>>
>>>
>>>
>>> --------------------------------------------------------------------------------------------
>>> This e-mail transmission (message and any attached files) may contain
>>> information that is proprietary, privileged and/or confidential to Veolia
>>> Environnement and/or its affiliates and is intended exclusively for the
>>> person(s) to whom it is addressed. If you are not the intended recipient,
>>> please notify the sender by return e-mail and delete all copies of this
>>> e-mail, including all attachments. Unless expressly authorized, any use,
>>> disclosure, publication, retransmission or dissemination of this e-mail
>>> and/or of its attachments is strictly prohibited.
>>>
>>> Ce message electronique et ses fichiers attaches sont strictement
>>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>>> message par erreur, merci de le retourner a son emetteur et de le detruire
>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>>> publication, la distribution, ou la reproduction non expressement
>>> autorisees de ce message et de ses pieces attachees sont interdites.
>>>
>>> --------------------------------------------------------------------------------------------
>>>
>>
>>
--
--------------------------------------------------------------------------------------------This
e-mail transmission (message and any attached files) may contain
information that is proprietary, privileged and/or confidential to Veolia
Environnement and/or its affiliates and is intended exclusively for the
person(s) to whom it is addressed. If you are not the intended recipient,
please notify the sender by return e-mail and delete all copies of this
e-mail, including all attachments. Unless expressly authorized, any use,
disclosure, publication, retransmission or dissemination of this e-mail
and/or of its attachments is strictly prohibited.
Ce message
electronique et ses fichiers attaches sont strictement confidentiels et
peuvent contenir des elements dont Veolia Environnement et/ou l'une de ses
entites affiliees sont proprietaires. Ils sont donc destines a l'usage de
leurs seuls destinataires. Si vous avez recu ce message par erreur, merci
de le retourner a son emetteur et de le detruire ainsi que toutes les
pieces attachees. L'utilisation, la divulgation, la publication, la
distribution, ou la reproduction non expressement autorisees de ce message
et de ses pieces attachees sont interdites.
--------------------------------------------------------------------------------------------
Re: Using user developped source in streamline python
Posted by Ismaël Mejía <ie...@gmail.com>.
It seems like a nice opportunity also to contribute them to the project in
case you are able to work on them.
Don't hesitate to contact us or ask for help if needed.
On Wed, Jun 27, 2018 at 4:45 PM Ahmet Altay <al...@google.com> wrote:
> Hi Sébastien,
>
> Currently there is no work in progress for including the write transforms
> for the locations you listed. You could develop your own version if
> interested. Please see WriteToBigquery transform [1] for reference.
>
> Ahmet
>
> [1]
> https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dcb322875e1b4dca/sdks/python/apache_beam/io/gcp/bigquery.py#L1287
>
> On Wed, Jun 27, 2018 at 2:48 AM, Sebastien Morand <
> sebastien.morand@veolia.com> wrote:
>
>> Hi,
>>
>> Thanks for your answer.
>>
>> Ok looking forward and ready to test alpha on this. Because we have
>> actually some use cases to send data to CloudSQL or Spanner or BigTable or
>> Firestore. As far as I read the documentation, there is no native support
>> for them and so we have already implemented a custom source support.
>>
>> So is there any work in progress for any of the 4 sinks above for native
>> support in the python SDK?
>>
>> Regards,
>>
>> *Sébastien MORAND*
>> Team Lead Solution Architect
>> Technology & Operations / Digital Factory
>> Veolia - Group Information Systems & Technology (IS&T)
>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>> Bureau 0144C (Ouest)
>> 30, rue Madeleine-Vionnet
>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>> -
>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
>> Aubervilliers, France
>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>> *www.veolia.com <http://www.veolia.com>*
>> <http://www.veolia.com>
>> <https://www.facebook.com/veoliaenvironment/>
>> <https://www.youtube.com/user/veoliaenvironnement>
>> <https://www.linkedin.com/company/veolia-environnement>
>> <https://twitter.com/veolia>
>>
>>
>> On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <lc...@google.com> wrote:
>>
>>> +dev@beam.apache.org
>>>
>>> Python streaming custom source support will be available via
>>> SplittableDoFn. It is actively being worked on by a few contributors but to
>>> my knowledge there is no roadmap yet for having support for this for
>>> Dataflow.
>>>
>>>
>>> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand <
>>> sebastien.morand@veolia.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We need to setup streaming dataflow in python using developed source
>>>> (in the opposite of native source) for Cloud SQL integration and Firestore
>>>> integration.
>>>>
>>>> This is currently not supported as far as I understood the
>>>> documentation, can you confirm? Any roadmap on the topic?
>>>>
>>>> Thanks by advance,
>>>> Regards,
>>>>
>>>> *Sébastien MORAND*
>>>> Team Lead Solution Architect
>>>> Technology & Operations / Digital Factory
>>>> Veolia - Group Information Systems & Technology (IS&T)
>>>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>>>> Bureau 0144C (Ouest)
>>>> 30, rue Madeleine-Vionnet
>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>>> -
>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
>>>> Aubervilliers, France
>>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>>> *www.veolia.com <http://www.veolia.com>*
>>>> <http://www.veolia.com>
>>>> <https://www.facebook.com/veoliaenvironment/>
>>>> <https://www.youtube.com/user/veoliaenvironnement>
>>>> <https://www.linkedin.com/company/veolia-environnement>
>>>> <https://twitter.com/veolia>
>>>>
>>>>
>>>>
>>>> --------------------------------------------------------------------------------------------
>>>> This e-mail transmission (message and any attached files) may contain
>>>> information that is proprietary, privileged and/or confidential to Veolia
>>>> Environnement and/or its affiliates and is intended exclusively for the
>>>> person(s) to whom it is addressed. If you are not the intended recipient,
>>>> please notify the sender by return e-mail and delete all copies of this
>>>> e-mail, including all attachments. Unless expressly authorized, any use,
>>>> disclosure, publication, retransmission or dissemination of this e-mail
>>>> and/or of its attachments is strictly prohibited.
>>>>
>>>> Ce message electronique et ses fichiers attaches sont strictement
>>>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>>>> message par erreur, merci de le retourner a son emetteur et de le detruire
>>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>>>> publication, la distribution, ou la reproduction non expressement
>>>> autorisees de ce message et de ses pieces attachees sont interdites.
>>>>
>>>> --------------------------------------------------------------------------------------------
>>>>
>>>
>>
>>
>> --------------------------------------------------------------------------------------------
>> This e-mail transmission (message and any attached files) may contain
>> information that is proprietary, privileged and/or confidential to Veolia
>> Environnement and/or its affiliates and is intended exclusively for the
>> person(s) to whom it is addressed. If you are not the intended recipient,
>> please notify the sender by return e-mail and delete all copies of this
>> e-mail, including all attachments. Unless expressly authorized, any use,
>> disclosure, publication, retransmission or dissemination of this e-mail
>> and/or of its attachments is strictly prohibited.
>>
>> Ce message electronique et ses fichiers attaches sont strictement
>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>> message par erreur, merci de le retourner a son emetteur et de le detruire
>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>> publication, la distribution, ou la reproduction non expressement
>> autorisees de ce message et de ses pieces attachees sont interdites.
>>
>> --------------------------------------------------------------------------------------------
>>
>
>
Re: Using user developped source in streamline python
Posted by Ahmet Altay <al...@google.com>.
Hi Sébastien,
Currently there is no work in progress for including the write transforms
for the locations you listed. You could develop your own version if
interested. Please see WriteToBigquery transform [1] for reference.
Ahmet
[1]
https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dcb322875e1b4dca/sdks/python/apache_beam/io/gcp/bigquery.py#L1287
On Wed, Jun 27, 2018 at 2:48 AM, Sebastien Morand <
sebastien.morand@veolia.com> wrote:
> Hi,
>
> Thanks for your answer.
>
> Ok looking forward and ready to test alpha on this. Because we have
> actually some use cases to send data to CloudSQL or Spanner or BigTable or
> Firestore. As far as I read the documentation, there is no native support
> for them and so we have already implemented a custom source support.
>
> So is there any work in progress for any of the 4 sinks above for native
> support in the python SDK?
>
> Regards,
>
> *Sébastien MORAND*
> Team Lead Solution Architect
> Technology & Operations / Digital Factory
> Veolia - Group Information Systems & Technology (IS&T)
> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
> Bureau 0144C (Ouest)
> 30, rue Madeleine-Vionnet
> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
> -
> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
> Aubervilliers, France
> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
> *www.veolia.com <http://www.veolia.com>*
> <http://www.veolia.com>
> <https://www.facebook.com/veoliaenvironment/>
> <https://www.youtube.com/user/veoliaenvironnement>
> <https://www.linkedin.com/company/veolia-environnement>
> <https://twitter.com/veolia>
>
>
> On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <lc...@google.com> wrote:
>
>> +dev@beam.apache.org
>>
>> Python streaming custom source support will be available via
>> SplittableDoFn. It is actively being worked on by a few contributors but to
>> my knowledge there is no roadmap yet for having support for this for
>> Dataflow.
>>
>>
>> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand <
>> sebastien.morand@veolia.com> wrote:
>>
>>> Hi,
>>>
>>> We need to setup streaming dataflow in python using developed source (in
>>> the opposite of native source) for Cloud SQL integration and Firestore
>>> integration.
>>>
>>> This is currently not supported as far as I understood the
>>> documentation, can you confirm? Any roadmap on the topic?
>>>
>>> Thanks by advance,
>>> Regards,
>>>
>>> *Sébastien MORAND*
>>> Team Lead Solution Architect
>>> Technology & Operations / Digital Factory
>>> Veolia - Group Information Systems & Technology (IS&T)
>>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>>> Bureau 0144C (Ouest)
>>> 30, rue Madeleine-Vionnet
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>> -
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
>>> Aubervilliers, France
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>> *www.veolia.com <http://www.veolia.com>*
>>> <http://www.veolia.com>
>>> <https://www.facebook.com/veoliaenvironment/>
>>> <https://www.youtube.com/user/veoliaenvironnement>
>>> <https://www.linkedin.com/company/veolia-environnement>
>>> <https://twitter.com/veolia>
>>>
>>>
>>> ------------------------------------------------------------
>>> --------------------------------
>>> This e-mail transmission (message and any attached files) may contain
>>> information that is proprietary, privileged and/or confidential to Veolia
>>> Environnement and/or its affiliates and is intended exclusively for the
>>> person(s) to whom it is addressed. If you are not the intended recipient,
>>> please notify the sender by return e-mail and delete all copies of this
>>> e-mail, including all attachments. Unless expressly authorized, any use,
>>> disclosure, publication, retransmission or dissemination of this e-mail
>>> and/or of its attachments is strictly prohibited.
>>>
>>> Ce message electronique et ses fichiers attaches sont strictement
>>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>>> message par erreur, merci de le retourner a son emetteur et de le detruire
>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>>> publication, la distribution, ou la reproduction non expressement
>>> autorisees de ce message et de ses pieces attachees sont interdites.
>>> ------------------------------------------------------------
>>> --------------------------------
>>>
>>
>
> ------------------------------------------------------------
> --------------------------------
> This e-mail transmission (message and any attached files) may contain
> information that is proprietary, privileged and/or confidential to Veolia
> Environnement and/or its affiliates and is intended exclusively for the
> person(s) to whom it is addressed. If you are not the intended recipient,
> please notify the sender by return e-mail and delete all copies of this
> e-mail, including all attachments. Unless expressly authorized, any use,
> disclosure, publication, retransmission or dissemination of this e-mail
> and/or of its attachments is strictly prohibited.
>
> Ce message electronique et ses fichiers attaches sont strictement
> confidentiels et peuvent contenir des elements dont Veolia Environnement
> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
> message par erreur, merci de le retourner a son emetteur et de le detruire
> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
> publication, la distribution, ou la reproduction non expressement
> autorisees de ce message et de ses pieces attachees sont interdites.
> ------------------------------------------------------------
> --------------------------------
>
Re: Using user developped source in streamline python
Posted by Ahmet Altay <al...@google.com>.
Hi Sébastien,
Currently there is no work in progress for including the write transforms
for the locations you listed. You could develop your own version if
interested. Please see WriteToBigquery transform [1] for reference.
Ahmet
[1]
https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dcb322875e1b4dca/sdks/python/apache_beam/io/gcp/bigquery.py#L1287
On Wed, Jun 27, 2018 at 2:48 AM, Sebastien Morand <
sebastien.morand@veolia.com> wrote:
> Hi,
>
> Thanks for your answer.
>
> Ok looking forward and ready to test alpha on this. Because we have
> actually some use cases to send data to CloudSQL or Spanner or BigTable or
> Firestore. As far as I read the documentation, there is no native support
> for them and so we have already implemented a custom source support.
>
> So is there any work in progress for any of the 4 sinks above for native
> support in the python SDK?
>
> Regards,
>
> *Sébastien MORAND*
> Team Lead Solution Architect
> Technology & Operations / Digital Factory
> Veolia - Group Information Systems & Technology (IS&T)
> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
> Bureau 0144C (Ouest)
> 30, rue Madeleine-Vionnet
> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
> -
> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
> Aubervilliers, France
> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
> *www.veolia.com <http://www.veolia.com>*
> <http://www.veolia.com>
> <https://www.facebook.com/veoliaenvironment/>
> <https://www.youtube.com/user/veoliaenvironnement>
> <https://www.linkedin.com/company/veolia-environnement>
> <https://twitter.com/veolia>
>
>
> On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <lc...@google.com> wrote:
>
>> +dev@beam.apache.org
>>
>> Python streaming custom source support will be available via
>> SplittableDoFn. It is actively being worked on by a few contributors but to
>> my knowledge there is no roadmap yet for having support for this for
>> Dataflow.
>>
>>
>> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand <
>> sebastien.morand@veolia.com> wrote:
>>
>>> Hi,
>>>
>>> We need to setup streaming dataflow in python using developed source (in
>>> the opposite of native source) for Cloud SQL integration and Firestore
>>> integration.
>>>
>>> This is currently not supported as far as I understood the
>>> documentation, can you confirm? Any roadmap on the topic?
>>>
>>> Thanks by advance,
>>> Regards,
>>>
>>> *Sébastien MORAND*
>>> Team Lead Solution Architect
>>> Technology & Operations / Digital Factory
>>> Veolia - Group Information Systems & Technology (IS&T)
>>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>>> Bureau 0144C (Ouest)
>>> 30, rue Madeleine-Vionnet
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>> -
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300
>>> Aubervilliers, France
>>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>
>>> *www.veolia.com <http://www.veolia.com>*
>>> <http://www.veolia.com>
>>> <https://www.facebook.com/veoliaenvironment/>
>>> <https://www.youtube.com/user/veoliaenvironnement>
>>> <https://www.linkedin.com/company/veolia-environnement>
>>> <https://twitter.com/veolia>
>>>
>>>
>>> ------------------------------------------------------------
>>> --------------------------------
>>> This e-mail transmission (message and any attached files) may contain
>>> information that is proprietary, privileged and/or confidential to Veolia
>>> Environnement and/or its affiliates and is intended exclusively for the
>>> person(s) to whom it is addressed. If you are not the intended recipient,
>>> please notify the sender by return e-mail and delete all copies of this
>>> e-mail, including all attachments. Unless expressly authorized, any use,
>>> disclosure, publication, retransmission or dissemination of this e-mail
>>> and/or of its attachments is strictly prohibited.
>>>
>>> Ce message electronique et ses fichiers attaches sont strictement
>>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>>> message par erreur, merci de le retourner a son emetteur et de le detruire
>>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>>> publication, la distribution, ou la reproduction non expressement
>>> autorisees de ce message et de ses pieces attachees sont interdites.
>>> ------------------------------------------------------------
>>> --------------------------------
>>>
>>
>
> ------------------------------------------------------------
> --------------------------------
> This e-mail transmission (message and any attached files) may contain
> information that is proprietary, privileged and/or confidential to Veolia
> Environnement and/or its affiliates and is intended exclusively for the
> person(s) to whom it is addressed. If you are not the intended recipient,
> please notify the sender by return e-mail and delete all copies of this
> e-mail, including all attachments. Unless expressly authorized, any use,
> disclosure, publication, retransmission or dissemination of this e-mail
> and/or of its attachments is strictly prohibited.
>
> Ce message electronique et ses fichiers attaches sont strictement
> confidentiels et peuvent contenir des elements dont Veolia Environnement
> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
> message par erreur, merci de le retourner a son emetteur et de le detruire
> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
> publication, la distribution, ou la reproduction non expressement
> autorisees de ce message et de ses pieces attachees sont interdites.
> ------------------------------------------------------------
> --------------------------------
>
Re: Using user developped source in streamline python
Posted by Sebastien Morand <se...@veolia.com>.
Hi,
Thanks for your answer.
Ok looking forward and ready to test alpha on this. Because we have
actually some use cases to send data to CloudSQL or Spanner or BigTable or
Firestore. As far as I read the documentation, there is no native support
for them and so we have already implemented a custom source support.
So is there any work in progress for any of the 4 sinks above for native
support in the python SDK?
Regards,
*Sébastien MORAND*
Team Lead Solution Architect
Technology & Operations / Digital Factory
Veolia - Group Information Systems & Technology (IS&T)
Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
Bureau 0144C (Ouest)
30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
*www.veolia.com <http://www.veolia.com>*
<http://www.veolia.com>
<https://www.facebook.com/veoliaenvironment/>
<https://www.youtube.com/user/veoliaenvironnement>
<https://www.linkedin.com/company/veolia-environnement>
<https://twitter.com/veolia>
On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <lc...@google.com> wrote:
> +dev@beam.apache.org
>
> Python streaming custom source support will be available via
> SplittableDoFn. It is actively being worked on by a few contributors but to
> my knowledge there is no roadmap yet for having support for this for
> Dataflow.
>
>
> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand <
> sebastien.morand@veolia.com> wrote:
>
>> Hi,
>>
>> We need to setup streaming dataflow in python using developed source (in
>> the opposite of native source) for Cloud SQL integration and Firestore
>> integration.
>>
>> This is currently not supported as far as I understood the documentation,
>> can you confirm? Any roadmap on the topic?
>>
>> Thanks by advance,
>> Regards,
>>
>> *Sébastien MORAND*
>> Team Lead Solution Architect
>> Technology & Operations / Digital Factory
>> Veolia - Group Information Systems & Technology (IS&T)
>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>> Bureau 0144C (Ouest)
>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
>> *www.veolia.com <http://www.veolia.com>*
>> <http://www.veolia.com>
>> <https://www.facebook.com/veoliaenvironment/>
>> <https://www.youtube.com/user/veoliaenvironnement>
>> <https://www.linkedin.com/company/veolia-environnement>
>> <https://twitter.com/veolia>
>>
>>
>>
>> --------------------------------------------------------------------------------------------
>> This e-mail transmission (message and any attached files) may contain
>> information that is proprietary, privileged and/or confidential to Veolia
>> Environnement and/or its affiliates and is intended exclusively for the
>> person(s) to whom it is addressed. If you are not the intended recipient,
>> please notify the sender by return e-mail and delete all copies of this
>> e-mail, including all attachments. Unless expressly authorized, any use,
>> disclosure, publication, retransmission or dissemination of this e-mail
>> and/or of its attachments is strictly prohibited.
>>
>> Ce message electronique et ses fichiers attaches sont strictement
>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>> message par erreur, merci de le retourner a son emetteur et de le detruire
>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>> publication, la distribution, ou la reproduction non expressement
>> autorisees de ce message et de ses pieces attachees sont interdites.
>>
>> --------------------------------------------------------------------------------------------
>>
>
--
--------------------------------------------------------------------------------------------This
e-mail transmission (message and any attached files) may contain
information that is proprietary, privileged and/or confidential to Veolia
Environnement and/or its affiliates and is intended exclusively for the
person(s) to whom it is addressed. If you are not the intended recipient,
please notify the sender by return e-mail and delete all copies of this
e-mail, including all attachments. Unless expressly authorized, any use,
disclosure, publication, retransmission or dissemination of this e-mail
and/or of its attachments is strictly prohibited.
Ce message
electronique et ses fichiers attaches sont strictement confidentiels et
peuvent contenir des elements dont Veolia Environnement et/ou l'une de ses
entites affiliees sont proprietaires. Ils sont donc destines a l'usage de
leurs seuls destinataires. Si vous avez recu ce message par erreur, merci
de le retourner a son emetteur et de le detruire ainsi que toutes les
pieces attachees. L'utilisation, la divulgation, la publication, la
distribution, ou la reproduction non expressement autorisees de ce message
et de ses pieces attachees sont interdites.
--------------------------------------------------------------------------------------------
Re: Using user developped source in streamline python
Posted by Sebastien Morand <se...@veolia.com>.
Hi,
Thanks for your answer.
Ok looking forward and ready to test alpha on this. Because we have
actually some use cases to send data to CloudSQL or Spanner or BigTable or
Firestore. As far as I read the documentation, there is no native support
for them and so we have already implemented a custom source support.
So is there any work in progress for any of the 4 sinks above for native
support in the python SDK?
Regards,
*Sébastien MORAND*
Team Lead Solution Architect
Technology & Operations / Digital Factory
Veolia - Group Information Systems & Technology (IS&T)
Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
Bureau 0144C (Ouest)
30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
*www.veolia.com <http://www.veolia.com>*
<http://www.veolia.com>
<https://www.facebook.com/veoliaenvironment/>
<https://www.youtube.com/user/veoliaenvironnement>
<https://www.linkedin.com/company/veolia-environnement>
<https://twitter.com/veolia>
On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <lc...@google.com> wrote:
> +dev@beam.apache.org
>
> Python streaming custom source support will be available via
> SplittableDoFn. It is actively being worked on by a few contributors but to
> my knowledge there is no roadmap yet for having support for this for
> Dataflow.
>
>
> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand <
> sebastien.morand@veolia.com> wrote:
>
>> Hi,
>>
>> We need to setup streaming dataflow in python using developed source (in
>> the opposite of native source) for Cloud SQL integration and Firestore
>> integration.
>>
>> This is currently not supported as far as I understood the documentation,
>> can you confirm? Any roadmap on the topic?
>>
>> Thanks by advance,
>> Regards,
>>
>> *Sébastien MORAND*
>> Team Lead Solution Architect
>> Technology & Operations / Digital Factory
>> Veolia - Group Information Systems & Technology (IS&T)
>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08
>> Bureau 0144C (Ouest)
>> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France
>> *www.veolia.com <http://www.veolia.com>*
>> <http://www.veolia.com>
>> <https://www.facebook.com/veoliaenvironment/>
>> <https://www.youtube.com/user/veoliaenvironnement>
>> <https://www.linkedin.com/company/veolia-environnement>
>> <https://twitter.com/veolia>
>>
>>
>>
>> --------------------------------------------------------------------------------------------
>> This e-mail transmission (message and any attached files) may contain
>> information that is proprietary, privileged and/or confidential to Veolia
>> Environnement and/or its affiliates and is intended exclusively for the
>> person(s) to whom it is addressed. If you are not the intended recipient,
>> please notify the sender by return e-mail and delete all copies of this
>> e-mail, including all attachments. Unless expressly authorized, any use,
>> disclosure, publication, retransmission or dissemination of this e-mail
>> and/or of its attachments is strictly prohibited.
>>
>> Ce message electronique et ses fichiers attaches sont strictement
>> confidentiels et peuvent contenir des elements dont Veolia Environnement
>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>> message par erreur, merci de le retourner a son emetteur et de le detruire
>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
>> publication, la distribution, ou la reproduction non expressement
>> autorisees de ce message et de ses pieces attachees sont interdites.
>>
>> --------------------------------------------------------------------------------------------
>>
>
--
--------------------------------------------------------------------------------------------This
e-mail transmission (message and any attached files) may contain
information that is proprietary, privileged and/or confidential to Veolia
Environnement and/or its affiliates and is intended exclusively for the
person(s) to whom it is addressed. If you are not the intended recipient,
please notify the sender by return e-mail and delete all copies of this
e-mail, including all attachments. Unless expressly authorized, any use,
disclosure, publication, retransmission or dissemination of this e-mail
and/or of its attachments is strictly prohibited.
Ce message
electronique et ses fichiers attaches sont strictement confidentiels et
peuvent contenir des elements dont Veolia Environnement et/ou l'une de ses
entites affiliees sont proprietaires. Ils sont donc destines a l'usage de
leurs seuls destinataires. Si vous avez recu ce message par erreur, merci
de le retourner a son emetteur et de le detruire ainsi que toutes les
pieces attachees. L'utilisation, la divulgation, la publication, la
distribution, ou la reproduction non expressement autorisees de ce message
et de ses pieces attachees sont interdites.
--------------------------------------------------------------------------------------------