You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Stefan Djelekar <st...@redbox.com> on 2019/11/06 09:56:04 UTC
RE: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
Thanks for the valuable information.
We’ve been exactly looking up to that Datastore connector, but there are some differences of course.
I’ll make the PR in the next few days and let’s pick it up from there.
King regards,
Stefan
From: Chamikara Jayalath <ch...@google.com>
Sent: Tuesday, November 5, 2019 2:24 AM
To: dev <de...@beam.apache.org>
Subject: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
Thanks for the contribution. Happy to help with the review.
Also, probably it'll be good to follow patterns employed by the existing Datastore connector when applicable: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java<https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java>
Thanks,
Cham
On Mon, Nov 4, 2019 at 2:37 AM Ismaël Mejía <ie...@gmail.com>> wrote:
Hello,
Please open a PR with the code of the new connector rebased against
the latest master.
It is worth to take a look at Beam's contribution guide [1] and
PTransform style guide [2] in advance.
In any case we can guide you on any issue during the PR review so
please go ahead.
Regards,
Ismaël
[1] https://beam.apache.org/contribute/<https://beam.apache.org/contribute/>
[2] https://beam.apache.org/contribute/ptransform-style-guide/<https://beam.apache.org/contribute/ptransform-style-guide/>
On Mon, Nov 4, 2019 at 10:48 AM Stefan Djelekar
<st...@redbox.com>> wrote:
>
> Hi beam devs,
>
>
>
> I'm Stefan from Redbox. We are a customer of GCP and we are in need of beam Java connector for Firestore.
>
>
>
> There is a pending JIRA item for this. https://issues.apache.org/jira/browse/BEAM-8376<https://issues.apache.org/jira/browse/BEAM-8376>
>
>
>
> The team inside of the company has been working on this for a while and we would like to contribute!
>
> What is the best way to do this? Can we perhaps get a review from an experienced beam member?
>
> Let me know what do you think.
>
>
>
> All the best,
>
>
>
> Stefan Đelekar
>
> Sofware Engineer
>
> stefan.djelekar@redbox.com<ma...@redbox.com>
>
> djelekar.com<http://djelekar.com>
>
>
Re: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
Posted by Chamikara Jayalath <ch...@google.com>.
Thanks. Taking a look. It'll probably be helpful if you can write a short
doc on design decisions you took including API and client library choice.
Also, as for any other sink, we should carefully design this so that
duplicate data is not written to the data store when bundles are retried by
runners due to failures.
Thanks,
Cham
On Mon, Dec 2, 2019 at 1:26 AM Stefan Djelekar <st...@redbox.com>
wrote:
> Hi dev team,
>
>
>
> I’ve submitted the pull request
> <https://github.com/apache/beam/pull/10187> for BEAM-8376
> <https://issues.apache.org/jira/browse/BEAM-8376>.
>
>
>
> Getting a review would be nice.
>
>
>
> All the best,
>
> Stefan
>
>
>
> *From:* Chamikara Jayalath <ch...@google.com>
> *Sent:* Wednesday, November 6, 2019 8:05 PM
> *To:* dev <de...@beam.apache.org>
> *Subject:* Re: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
>
>
>
> BTW, FYI, I'm also talking with folks from Google Firestore team regarding
> this. I think they had shown some interest in taking this up but I'm not
> sure.
>
> If they are able to contribute here, and if we can coordinate with some of
> those folks on this effort, it will be great for the long term health and
> maintenance of the connector.
>
>
>
> Thanks,
>
> Cham
>
>
>
> On Wed, Nov 6, 2019 at 1:56 AM Stefan Djelekar <st...@redbox.com>
> wrote:
>
> Thanks for the valuable information.
>
>
>
> We’ve been exactly looking up to that Datastore connector, but there are
> some differences of course.
>
>
>
> I’ll make the PR in the next few days and let’s pick it up from there.
>
>
>
> King regards,
>
> Stefan
>
>
>
>
>
> *From:* Chamikara Jayalath <ch...@google.com>
> *Sent:* Tuesday, November 5, 2019 2:24 AM
> *To:* dev <de...@beam.apache.org>
> *Subject:* [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
>
>
>
> Thanks for the contribution. Happy to help with the review.
>
> Also, probably it'll be good to follow patterns employed by the existing
> Datastore connector when applicable:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
>
>
>
> Thanks,
>
> Cham
>
>
>
> On Mon, Nov 4, 2019 at 2:37 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
> Hello,
>
> Please open a PR with the code of the new connector rebased against
> the latest master.
>
> It is worth to take a look at Beam's contribution guide [1] and
> PTransform style guide [2] in advance.
> In any case we can guide you on any issue during the PR review so
> please go ahead.
>
> Regards,
> Ismaël
>
> [1] https://beam.apache.org/contribute/
> [2] https://beam.apache.org/contribute/ptransform-style-guide/
>
>
> On Mon, Nov 4, 2019 at 10:48 AM Stefan Djelekar
> <st...@redbox.com> wrote:
> >
> > Hi beam devs,
> >
> >
> >
> > I'm Stefan from Redbox. We are a customer of GCP and we are in need of
> beam Java connector for Firestore.
> >
> >
> >
> > There is a pending JIRA item for this.
> https://issues.apache.org/jira/browse/BEAM-8376
> >
> >
> >
> > The team inside of the company has been working on this for a while and
> we would like to contribute!
> >
> > What is the best way to do this? Can we perhaps get a review from an
> experienced beam member?
> >
> > Let me know what do you think.
> >
> >
> >
> > All the best,
> >
> >
> >
> > Stefan Đelekar
> >
> > Sofware Engineer
> >
> > stefan.djelekar@redbox.com
> >
> > djelekar.com
> >
> >
>
>
RE: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
Posted by Stefan Djelekar <st...@redbox.com>.
Hi dev team,
I’ve submitted the pull request<https://github.com/apache/beam/pull/10187> for BEAM-8376<https://issues.apache.org/jira/browse/BEAM-8376>.
Getting a review would be nice.
All the best,
Stefan
From: Chamikara Jayalath <ch...@google.com>
Sent: Wednesday, November 6, 2019 8:05 PM
To: dev <de...@beam.apache.org>
Subject: Re: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
BTW, FYI, I'm also talking with folks from Google Firestore team regarding this. I think they had shown some interest in taking this up but I'm not sure.
If they are able to contribute here, and if we can coordinate with some of those folks on this effort, it will be great for the long term health and maintenance of the connector.
Thanks,
Cham
On Wed, Nov 6, 2019 at 1:56 AM Stefan Djelekar <st...@redbox.com>> wrote:
Thanks for the valuable information.
We’ve been exactly looking up to that Datastore connector, but there are some differences of course.
I’ll make the PR in the next few days and let’s pick it up from there.
King regards,
Stefan
From: Chamikara Jayalath <ch...@google.com>>
Sent: Tuesday, November 5, 2019 2:24 AM
To: dev <de...@beam.apache.org>>
Subject: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
Thanks for the contribution. Happy to help with the review.
Also, probably it'll be good to follow patterns employed by the existing Datastore connector when applicable: https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java<https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java>
Thanks,
Cham
On Mon, Nov 4, 2019 at 2:37 AM Ismaël Mejía <ie...@gmail.com>> wrote:
Hello,
Please open a PR with the code of the new connector rebased against
the latest master.
It is worth to take a look at Beam's contribution guide [1] and
PTransform style guide [2] in advance.
In any case we can guide you on any issue during the PR review so
please go ahead.
Regards,
Ismaël
[1] https://beam.apache.org/contribute/<https://beam.apache.org/contribute/>
[2] https://beam.apache.org/contribute/ptransform-style-guide/<https://beam.apache.org/contribute/ptransform-style-guide/>
On Mon, Nov 4, 2019 at 10:48 AM Stefan Djelekar
<st...@redbox.com>> wrote:
>
> Hi beam devs,
>
>
>
> I'm Stefan from Redbox. We are a customer of GCP and we are in need of beam Java connector for Firestore.
>
>
>
> There is a pending JIRA item for this. https://issues.apache.org/jira/browse/BEAM-8376<https://issues.apache.org/jira/browse/BEAM-8376>
>
>
>
> The team inside of the company has been working on this for a while and we would like to contribute!
>
> What is the best way to do this? Can we perhaps get a review from an experienced beam member?
>
> Let me know what do you think.
>
>
>
> All the best,
>
>
>
> Stefan Đelekar
>
> Sofware Engineer
>
> stefan.djelekar@redbox.com<ma...@redbox.com>
>
> djelekar.com<http://djelekar.com>
>
>
Re: [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
Posted by Chamikara Jayalath <ch...@google.com>.
BTW, FYI, I'm also talking with folks from Google Firestore team regarding
this. I think they had shown some interest in taking this up but I'm not
sure.
If they are able to contribute here, and if we can coordinate with some of
those folks on this effort, it will be great for the long term health and
maintenance of the connector.
Thanks,
Cham
On Wed, Nov 6, 2019 at 1:56 AM Stefan Djelekar <st...@redbox.com>
wrote:
> Thanks for the valuable information.
>
>
>
> We’ve been exactly looking up to that Datastore connector, but there are
> some differences of course.
>
>
>
> I’ll make the PR in the next few days and let’s pick it up from there.
>
>
>
> King regards,
>
> Stefan
>
>
>
>
>
> *From:* Chamikara Jayalath <ch...@google.com>
> *Sent:* Tuesday, November 5, 2019 2:24 AM
> *To:* dev <de...@beam.apache.org>
> *Subject:* [EXTERNAL] Re: FirestoreIO connector [JavaSDK]
>
>
>
> Thanks for the contribution. Happy to help with the review.
>
> Also, probably it'll be good to follow patterns employed by the existing
> Datastore connector when applicable:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
>
>
>
> Thanks,
>
> Cham
>
>
>
> On Mon, Nov 4, 2019 at 2:37 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
> Hello,
>
> Please open a PR with the code of the new connector rebased against
> the latest master.
>
> It is worth to take a look at Beam's contribution guide [1] and
> PTransform style guide [2] in advance.
> In any case we can guide you on any issue during the PR review so
> please go ahead.
>
> Regards,
> Ismaël
>
> [1] https://beam.apache.org/contribute/
> [2] https://beam.apache.org/contribute/ptransform-style-guide/
>
>
> On Mon, Nov 4, 2019 at 10:48 AM Stefan Djelekar
> <st...@redbox.com> wrote:
> >
> > Hi beam devs,
> >
> >
> >
> > I'm Stefan from Redbox. We are a customer of GCP and we are in need of
> beam Java connector for Firestore.
> >
> >
> >
> > There is a pending JIRA item for this.
> https://issues.apache.org/jira/browse/BEAM-8376
> >
> >
> >
> > The team inside of the company has been working on this for a while and
> we would like to contribute!
> >
> > What is the best way to do this? Can we perhaps get a review from an
> experienced beam member?
> >
> > Let me know what do you think.
> >
> >
> >
> > All the best,
> >
> >
> >
> > Stefan Đelekar
> >
> > Sofware Engineer
> >
> > stefan.djelekar@redbox.com
> >
> > djelekar.com
> >
> >
>
>