You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Peter Dolan <pe...@google.com.INVALID> on 2017/05/31 19:10:17 UTC
Cloud ML Operators
Hello developers,
I work with Google Cloud ML, and my team and I are interested in
contributing a set of Operators to support working with the Cloud ML
platform. The platform supports using the TensorFlow deep neural network
framework as a managed system.
In particular, we would like to contribute
* CloudMLTrainingOperator, which would launch and monitor a Cloud ML
Training Job (https://cloud.google.com/ml-engine/docs/how-tos/training-jobs
),
* CloudMLBatchPredictionOperator, which would launch and monitor a Cloud
ML Batch Prediction Job (
https://cloud.google.com/ml-engine/docs/how-tos/batch-predict), and
* CloudMLVersionOperator, which can create, update, and delete TensorFlow
model versions (
https://cloud.google.com/ml-engine/docs/how-tos/managing-models-jobs)
I'm eager to hear if the Airflow project is open to these contributions,
and if any changes are suggested. We have working prototype versions of all
of them.
Thanks in advance,
Peter
Re: Cloud ML Operators
Posted by Peter Dolan <pe...@google.com.INVALID>.
Yes, thanks Chris! I really appreciate your responsiveness especially
during a release. We have one more PR incoming, which we held off because
there was so much code in common. It should be smaller than the previous
ones.
On Thu, Jun 29, 2017 at 8:58 AM Chris Riccomini <cr...@apache.org>
wrote:
> I think we're all good here. :)
>
> On Mon, Jun 26, 2017 at 2:48 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Hey Peter,
> >
> > Thanks, will have a look. I'm a bit tied up with the 1.8.2 release at the
> > moment, so it might be a little longer than usual, but I'll get there. I
> > did do a bunch of Google PRs on Friday, but I must have missed this one.
> > Stay tuned. If you don't get enough response by Wednesday, please
> re-ping.
> >
> > Cheers,
> > Chris
> >
> > On Mon, Jun 26, 2017 at 1:46 PM, Peter Dolan <
> > peterdolan@google.com.invalid> wrote:
> >
> >> Hey Chris,
> >>
> >> We sent our first pull request out for review last week, let me know if
> >> you or someone else has time to take a look at it -
> >> https://github.com/apache/incubator-airflow/pull/2390. Definitely
> >> appreciate quick feedback as this is our first PR and I'm sure we're
> making
> >> some novice errors.
> >>
> >> Thanks,
> >> Peter
> >>
> >> On Mon, Jun 5, 2017 at 1:32 PM Chris Riccomini <cr...@apache.org>
> >> wrote:
> >>
> >>> Awesome, feel free to ping for code reviews if you don't get them in a
> >>> timely manner.
> >>>
> >>> On Fri, Jun 2, 2017 at 2:46 PM, Peter Dolan
> >>> <pe...@google.com.invalid>
> >>> wrote:
> >>>
> >>> > Just created https://issues.apache.org/jira/browse/AIRFLOW-1273,
> >>> > https://issues.apache.org/jira/browse/AIRFLOW-1272, and
> >>> > https://issues.apache.org/jira/browse/AIRFLOW-1271. We'll follow up
> >>> with
> >>> > pull requests in the coming weeks.
> >>> >
> >>> > On Thu, Jun 1, 2017 at 10:44 AM Peter Dolan <pe...@google.com>
> >>> wrote:
> >>> >
> >>> >> Hi Bolke,
> >>> >>
> >>> >> Great! And yes, we will be able to maintain these operators after
> the
> >>> >> contributions. We further would always insist on strong test
> coverage
> >>> as
> >>> >> well.
> >>> >>
> >>> >> I'll open some JIRA tickets to track their implementation.
> >>> >>
> >>> >> Thanks,
> >>> >> Peter
> >>> >>
> >>> >> On Thu, Jun 1, 2017 at 6:05 AM Bolke de Bruin <bd...@gmail.com>
> >>> wrote:
> >>> >>
> >>> >>> Hi Peter,
> >>> >>>
> >>> >>> That sounds great! I think the main criteria for this is will you
> >>> >>> maintain the code afterwards? The contrib section is slowly but
> >>> steadily
> >>> >>> growing and with operators/hooks we are particularly dependent on
> the
> >>> >>> community as not all (or even none in some case) of the committers
> >>> use
> >>> >>> these themselves.
> >>> >>>
> >>> >>> In any case test coverage is required, but that is a given I think.
> >>> >>>
> >>> >>> Kind regards,
> >>> >>> Bolke
> >>> >>>
> >>> >>> > On 31 May 2017, at 21:10, Peter Dolan
> >>> <pe...@google.com.INVALID>
> >>> >>> wrote:
> >>> >>> >
> >>> >>> > Hello developers,
> >>> >>> >
> >>> >>> > I work with Google Cloud ML, and my team and I are interested in
> >>> >>> contributing a set of Operators to support working with the Cloud
> ML
> >>> >>> platform. The platform supports using the TensorFlow deep neural
> >>> network
> >>> >>> framework as a managed system.
> >>> >>> >
> >>> >>> > In particular, we would like to contribute
> >>> >>> > * CloudMLTrainingOperator, which would launch and monitor a
> Cloud
> >>> ML
> >>> >>> Training Job (https://cloud.google.com/ml-e
> >>> ngine/docs/how-tos/training-
> >>> >>> jobs <
> https://cloud.google.com/ml-engine/docs/how-tos/training-jobs
> >>> >),
> >>> >>> > * CloudMLBatchPredictionOperator, which would launch and monitor
> >>> a
> >>> >>> Cloud ML Batch Prediction Job (https://cloud.google.com/ml-
> >>> >>> engine/docs/how-tos/batch-predict <https://cloud.google.com/ml-
> >>> >>> engine/docs/how-tos/batch-predict>), and
> >>> >>> > * CloudMLVersionOperator, which can create, update, and delete
> >>> >>> TensorFlow model versions (https://cloud.google.com/ml-
> >>> >>> engine/docs/how-tos/managing-models-jobs <
> >>> https://cloud.google.com/ml-
> >>> >>> engine/docs/how-tos/managing-models-jobs>)
> >>> >>> >
> >>> >>> > I'm eager to hear if the Airflow project is open to these
> >>> >>> contributions, and if any changes are suggested. We have working
> >>> prototype
> >>> >>> versions of all of them.
> >>> >>> >
> >>> >>> > Thanks in advance,
> >>> >>> > Peter
> >>> >>>
> >>> >>>
> >>>
> >>
> >
>
Re: Cloud ML Operators
Posted by Chris Riccomini <cr...@apache.org>.
I think we're all good here. :)
On Mon, Jun 26, 2017 at 2:48 PM, Chris Riccomini <cr...@apache.org>
wrote:
> Hey Peter,
>
> Thanks, will have a look. I'm a bit tied up with the 1.8.2 release at the
> moment, so it might be a little longer than usual, but I'll get there. I
> did do a bunch of Google PRs on Friday, but I must have missed this one.
> Stay tuned. If you don't get enough response by Wednesday, please re-ping.
>
> Cheers,
> Chris
>
> On Mon, Jun 26, 2017 at 1:46 PM, Peter Dolan <
> peterdolan@google.com.invalid> wrote:
>
>> Hey Chris,
>>
>> We sent our first pull request out for review last week, let me know if
>> you or someone else has time to take a look at it -
>> https://github.com/apache/incubator-airflow/pull/2390. Definitely
>> appreciate quick feedback as this is our first PR and I'm sure we're making
>> some novice errors.
>>
>> Thanks,
>> Peter
>>
>> On Mon, Jun 5, 2017 at 1:32 PM Chris Riccomini <cr...@apache.org>
>> wrote:
>>
>>> Awesome, feel free to ping for code reviews if you don't get them in a
>>> timely manner.
>>>
>>> On Fri, Jun 2, 2017 at 2:46 PM, Peter Dolan
>>> <pe...@google.com.invalid>
>>> wrote:
>>>
>>> > Just created https://issues.apache.org/jira/browse/AIRFLOW-1273,
>>> > https://issues.apache.org/jira/browse/AIRFLOW-1272, and
>>> > https://issues.apache.org/jira/browse/AIRFLOW-1271. We'll follow up
>>> with
>>> > pull requests in the coming weeks.
>>> >
>>> > On Thu, Jun 1, 2017 at 10:44 AM Peter Dolan <pe...@google.com>
>>> wrote:
>>> >
>>> >> Hi Bolke,
>>> >>
>>> >> Great! And yes, we will be able to maintain these operators after the
>>> >> contributions. We further would always insist on strong test coverage
>>> as
>>> >> well.
>>> >>
>>> >> I'll open some JIRA tickets to track their implementation.
>>> >>
>>> >> Thanks,
>>> >> Peter
>>> >>
>>> >> On Thu, Jun 1, 2017 at 6:05 AM Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>> >>
>>> >>> Hi Peter,
>>> >>>
>>> >>> That sounds great! I think the main criteria for this is will you
>>> >>> maintain the code afterwards? The contrib section is slowly but
>>> steadily
>>> >>> growing and with operators/hooks we are particularly dependent on the
>>> >>> community as not all (or even none in some case) of the committers
>>> use
>>> >>> these themselves.
>>> >>>
>>> >>> In any case test coverage is required, but that is a given I think.
>>> >>>
>>> >>> Kind regards,
>>> >>> Bolke
>>> >>>
>>> >>> > On 31 May 2017, at 21:10, Peter Dolan
>>> <pe...@google.com.INVALID>
>>> >>> wrote:
>>> >>> >
>>> >>> > Hello developers,
>>> >>> >
>>> >>> > I work with Google Cloud ML, and my team and I are interested in
>>> >>> contributing a set of Operators to support working with the Cloud ML
>>> >>> platform. The platform supports using the TensorFlow deep neural
>>> network
>>> >>> framework as a managed system.
>>> >>> >
>>> >>> > In particular, we would like to contribute
>>> >>> > * CloudMLTrainingOperator, which would launch and monitor a Cloud
>>> ML
>>> >>> Training Job (https://cloud.google.com/ml-e
>>> ngine/docs/how-tos/training-
>>> >>> jobs <https://cloud.google.com/ml-engine/docs/how-tos/training-jobs
>>> >),
>>> >>> > * CloudMLBatchPredictionOperator, which would launch and monitor
>>> a
>>> >>> Cloud ML Batch Prediction Job (https://cloud.google.com/ml-
>>> >>> engine/docs/how-tos/batch-predict <https://cloud.google.com/ml-
>>> >>> engine/docs/how-tos/batch-predict>), and
>>> >>> > * CloudMLVersionOperator, which can create, update, and delete
>>> >>> TensorFlow model versions (https://cloud.google.com/ml-
>>> >>> engine/docs/how-tos/managing-models-jobs <
>>> https://cloud.google.com/ml-
>>> >>> engine/docs/how-tos/managing-models-jobs>)
>>> >>> >
>>> >>> > I'm eager to hear if the Airflow project is open to these
>>> >>> contributions, and if any changes are suggested. We have working
>>> prototype
>>> >>> versions of all of them.
>>> >>> >
>>> >>> > Thanks in advance,
>>> >>> > Peter
>>> >>>
>>> >>>
>>>
>>
>
Re: Cloud ML Operators
Posted by Chris Riccomini <cr...@apache.org>.
Hey Peter,
Thanks, will have a look. I'm a bit tied up with the 1.8.2 release at the
moment, so it might be a little longer than usual, but I'll get there. I
did do a bunch of Google PRs on Friday, but I must have missed this one.
Stay tuned. If you don't get enough response by Wednesday, please re-ping.
Cheers,
Chris
On Mon, Jun 26, 2017 at 1:46 PM, Peter Dolan <pe...@google.com.invalid>
wrote:
> Hey Chris,
>
> We sent our first pull request out for review last week, let me know if
> you or someone else has time to take a look at it -
> https://github.com/apache/incubator-airflow/pull/2390. Definitely
> appreciate quick feedback as this is our first PR and I'm sure we're making
> some novice errors.
>
> Thanks,
> Peter
>
> On Mon, Jun 5, 2017 at 1:32 PM Chris Riccomini <cr...@apache.org>
> wrote:
>
>> Awesome, feel free to ping for code reviews if you don't get them in a
>> timely manner.
>>
>> On Fri, Jun 2, 2017 at 2:46 PM, Peter Dolan <peterdolan@google.com.invalid
>> >
>> wrote:
>>
>> > Just created https://issues.apache.org/jira/browse/AIRFLOW-1273,
>> > https://issues.apache.org/jira/browse/AIRFLOW-1272, and
>> > https://issues.apache.org/jira/browse/AIRFLOW-1271. We'll follow up
>> with
>> > pull requests in the coming weeks.
>> >
>> > On Thu, Jun 1, 2017 at 10:44 AM Peter Dolan <pe...@google.com>
>> wrote:
>> >
>> >> Hi Bolke,
>> >>
>> >> Great! And yes, we will be able to maintain these operators after the
>> >> contributions. We further would always insist on strong test coverage
>> as
>> >> well.
>> >>
>> >> I'll open some JIRA tickets to track their implementation.
>> >>
>> >> Thanks,
>> >> Peter
>> >>
>> >> On Thu, Jun 1, 2017 at 6:05 AM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>> >>
>> >>> Hi Peter,
>> >>>
>> >>> That sounds great! I think the main criteria for this is will you
>> >>> maintain the code afterwards? The contrib section is slowly but
>> steadily
>> >>> growing and with operators/hooks we are particularly dependent on the
>> >>> community as not all (or even none in some case) of the committers use
>> >>> these themselves.
>> >>>
>> >>> In any case test coverage is required, but that is a given I think.
>> >>>
>> >>> Kind regards,
>> >>> Bolke
>> >>>
>> >>> > On 31 May 2017, at 21:10, Peter Dolan <peterdolan@google.com.INVALID
>> >
>> >>> wrote:
>> >>> >
>> >>> > Hello developers,
>> >>> >
>> >>> > I work with Google Cloud ML, and my team and I are interested in
>> >>> contributing a set of Operators to support working with the Cloud ML
>> >>> platform. The platform supports using the TensorFlow deep neural
>> network
>> >>> framework as a managed system.
>> >>> >
>> >>> > In particular, we would like to contribute
>> >>> > * CloudMLTrainingOperator, which would launch and monitor a Cloud
>> ML
>> >>> Training Job (https://cloud.google.com/ml-
>> engine/docs/how-tos/training-
>> >>> jobs <https://cloud.google.com/ml-engine/docs/how-tos/training-jobs
>> >),
>> >>> > * CloudMLBatchPredictionOperator, which would launch and monitor a
>> >>> Cloud ML Batch Prediction Job (https://cloud.google.com/ml-
>> >>> engine/docs/how-tos/batch-predict <https://cloud.google.com/ml-
>> >>> engine/docs/how-tos/batch-predict>), and
>> >>> > * CloudMLVersionOperator, which can create, update, and delete
>> >>> TensorFlow model versions (https://cloud.google.com/ml-
>> >>> engine/docs/how-tos/managing-models-jobs <
>> https://cloud.google.com/ml-
>> >>> engine/docs/how-tos/managing-models-jobs>)
>> >>> >
>> >>> > I'm eager to hear if the Airflow project is open to these
>> >>> contributions, and if any changes are suggested. We have working
>> prototype
>> >>> versions of all of them.
>> >>> >
>> >>> > Thanks in advance,
>> >>> > Peter
>> >>>
>> >>>
>>
>
Re: Cloud ML Operators
Posted by Peter Dolan <pe...@google.com.INVALID>.
Hey Chris,
We sent our first pull request out for review last week, let me know if you
or someone else has time to take a look at it -
https://github.com/apache/incubator-airflow/pull/2390. Definitely
appreciate quick feedback as this is our first PR and I'm sure we're making
some novice errors.
Thanks,
Peter
On Mon, Jun 5, 2017 at 1:32 PM Chris Riccomini <cr...@apache.org>
wrote:
> Awesome, feel free to ping for code reviews if you don't get them in a
> timely manner.
>
> On Fri, Jun 2, 2017 at 2:46 PM, Peter Dolan <peterdolan@google.com.invalid
> >
> wrote:
>
> > Just created https://issues.apache.org/jira/browse/AIRFLOW-1273,
> > https://issues.apache.org/jira/browse/AIRFLOW-1272, and
> > https://issues.apache.org/jira/browse/AIRFLOW-1271. We'll follow up with
> > pull requests in the coming weeks.
> >
> > On Thu, Jun 1, 2017 at 10:44 AM Peter Dolan <pe...@google.com>
> wrote:
> >
> >> Hi Bolke,
> >>
> >> Great! And yes, we will be able to maintain these operators after the
> >> contributions. We further would always insist on strong test coverage as
> >> well.
> >>
> >> I'll open some JIRA tickets to track their implementation.
> >>
> >> Thanks,
> >> Peter
> >>
> >> On Thu, Jun 1, 2017 at 6:05 AM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >>> Hi Peter,
> >>>
> >>> That sounds great! I think the main criteria for this is will you
> >>> maintain the code afterwards? The contrib section is slowly but
> steadily
> >>> growing and with operators/hooks we are particularly dependent on the
> >>> community as not all (or even none in some case) of the committers use
> >>> these themselves.
> >>>
> >>> In any case test coverage is required, but that is a given I think.
> >>>
> >>> Kind regards,
> >>> Bolke
> >>>
> >>> > On 31 May 2017, at 21:10, Peter Dolan <peterdolan@google.com.INVALID
> >
> >>> wrote:
> >>> >
> >>> > Hello developers,
> >>> >
> >>> > I work with Google Cloud ML, and my team and I are interested in
> >>> contributing a set of Operators to support working with the Cloud ML
> >>> platform. The platform supports using the TensorFlow deep neural
> network
> >>> framework as a managed system.
> >>> >
> >>> > In particular, we would like to contribute
> >>> > * CloudMLTrainingOperator, which would launch and monitor a Cloud ML
> >>> Training Job (
> https://cloud.google.com/ml-engine/docs/how-tos/training-
> >>> jobs <https://cloud.google.com/ml-engine/docs/how-tos/training-jobs>),
> >>> > * CloudMLBatchPredictionOperator, which would launch and monitor a
> >>> Cloud ML Batch Prediction Job (https://cloud.google.com/ml-
> >>> engine/docs/how-tos/batch-predict <https://cloud.google.com/ml-
> >>> engine/docs/how-tos/batch-predict>), and
> >>> > * CloudMLVersionOperator, which can create, update, and delete
> >>> TensorFlow model versions (https://cloud.google.com/ml-
> >>> engine/docs/how-tos/managing-models-jobs <https://cloud.google.com/ml-
> >>> engine/docs/how-tos/managing-models-jobs>)
> >>> >
> >>> > I'm eager to hear if the Airflow project is open to these
> >>> contributions, and if any changes are suggested. We have working
> prototype
> >>> versions of all of them.
> >>> >
> >>> > Thanks in advance,
> >>> > Peter
> >>>
> >>>
>
Re: Cloud ML Operators
Posted by Chris Riccomini <cr...@apache.org>.
Awesome, feel free to ping for code reviews if you don't get them in a
timely manner.
On Fri, Jun 2, 2017 at 2:46 PM, Peter Dolan <pe...@google.com.invalid>
wrote:
> Just created https://issues.apache.org/jira/browse/AIRFLOW-1273,
> https://issues.apache.org/jira/browse/AIRFLOW-1272, and
> https://issues.apache.org/jira/browse/AIRFLOW-1271. We'll follow up with
> pull requests in the coming weeks.
>
> On Thu, Jun 1, 2017 at 10:44 AM Peter Dolan <pe...@google.com> wrote:
>
>> Hi Bolke,
>>
>> Great! And yes, we will be able to maintain these operators after the
>> contributions. We further would always insist on strong test coverage as
>> well.
>>
>> I'll open some JIRA tickets to track their implementation.
>>
>> Thanks,
>> Peter
>>
>> On Thu, Jun 1, 2017 at 6:05 AM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>>> Hi Peter,
>>>
>>> That sounds great! I think the main criteria for this is will you
>>> maintain the code afterwards? The contrib section is slowly but steadily
>>> growing and with operators/hooks we are particularly dependent on the
>>> community as not all (or even none in some case) of the committers use
>>> these themselves.
>>>
>>> In any case test coverage is required, but that is a given I think.
>>>
>>> Kind regards,
>>> Bolke
>>>
>>> > On 31 May 2017, at 21:10, Peter Dolan <pe...@google.com.INVALID>
>>> wrote:
>>> >
>>> > Hello developers,
>>> >
>>> > I work with Google Cloud ML, and my team and I are interested in
>>> contributing a set of Operators to support working with the Cloud ML
>>> platform. The platform supports using the TensorFlow deep neural network
>>> framework as a managed system.
>>> >
>>> > In particular, we would like to contribute
>>> > * CloudMLTrainingOperator, which would launch and monitor a Cloud ML
>>> Training Job (https://cloud.google.com/ml-engine/docs/how-tos/training-
>>> jobs <https://cloud.google.com/ml-engine/docs/how-tos/training-jobs>),
>>> > * CloudMLBatchPredictionOperator, which would launch and monitor a
>>> Cloud ML Batch Prediction Job (https://cloud.google.com/ml-
>>> engine/docs/how-tos/batch-predict <https://cloud.google.com/ml-
>>> engine/docs/how-tos/batch-predict>), and
>>> > * CloudMLVersionOperator, which can create, update, and delete
>>> TensorFlow model versions (https://cloud.google.com/ml-
>>> engine/docs/how-tos/managing-models-jobs <https://cloud.google.com/ml-
>>> engine/docs/how-tos/managing-models-jobs>)
>>> >
>>> > I'm eager to hear if the Airflow project is open to these
>>> contributions, and if any changes are suggested. We have working prototype
>>> versions of all of them.
>>> >
>>> > Thanks in advance,
>>> > Peter
>>>
>>>
Re: Cloud ML Operators
Posted by Peter Dolan <pe...@google.com.INVALID>.
Just created https://issues.apache.org/jira/browse/AIRFLOW-1273,
https://issues.apache.org/jira/browse/AIRFLOW-1272, and
https://issues.apache.org/jira/browse/AIRFLOW-1271. We'll follow up with
pull requests in the coming weeks.
On Thu, Jun 1, 2017 at 10:44 AM Peter Dolan <pe...@google.com> wrote:
> Hi Bolke,
>
> Great! And yes, we will be able to maintain these operators after the
> contributions. We further would always insist on strong test coverage as
> well.
>
> I'll open some JIRA tickets to track their implementation.
>
> Thanks,
> Peter
>
> On Thu, Jun 1, 2017 at 6:05 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Hi Peter,
>>
>> That sounds great! I think the main criteria for this is will you
>> maintain the code afterwards? The contrib section is slowly but steadily
>> growing and with operators/hooks we are particularly dependent on the
>> community as not all (or even none in some case) of the committers use
>> these themselves.
>>
>> In any case test coverage is required, but that is a given I think.
>>
>> Kind regards,
>> Bolke
>>
>> > On 31 May 2017, at 21:10, Peter Dolan <pe...@google.com.INVALID>
>> wrote:
>> >
>> > Hello developers,
>> >
>> > I work with Google Cloud ML, and my team and I are interested in
>> contributing a set of Operators to support working with the Cloud ML
>> platform. The platform supports using the TensorFlow deep neural network
>> framework as a managed system.
>> >
>> > In particular, we would like to contribute
>> > * CloudMLTrainingOperator, which would launch and monitor a Cloud ML
>> Training Job (
>> https://cloud.google.com/ml-engine/docs/how-tos/training-jobs <
>> https://cloud.google.com/ml-engine/docs/how-tos/training-jobs>),
>> > * CloudMLBatchPredictionOperator, which would launch and monitor a
>> Cloud ML Batch Prediction Job (
>> https://cloud.google.com/ml-engine/docs/how-tos/batch-predict <
>> https://cloud.google.com/ml-engine/docs/how-tos/batch-predict>), and
>> > * CloudMLVersionOperator, which can create, update, and delete
>> TensorFlow model versions (
>> https://cloud.google.com/ml-engine/docs/how-tos/managing-models-jobs <
>> https://cloud.google.com/ml-engine/docs/how-tos/managing-models-jobs>)
>> >
>> > I'm eager to hear if the Airflow project is open to these
>> contributions, and if any changes are suggested. We have working prototype
>> versions of all of them.
>> >
>> > Thanks in advance,
>> > Peter
>>
>>
Re: Cloud ML Operators
Posted by Peter Dolan <pe...@google.com.INVALID>.
Hi Bolke,
Great! And yes, we will be able to maintain these operators after the
contributions. We further would always insist on strong test coverage as
well.
I'll open some JIRA tickets to track their implementation.
Thanks,
Peter
On Thu, Jun 1, 2017 at 6:05 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Hi Peter,
>
> That sounds great! I think the main criteria for this is will you maintain
> the code afterwards? The contrib section is slowly but steadily growing and
> with operators/hooks we are particularly dependent on the community as not
> all (or even none in some case) of the committers use these themselves.
>
> In any case test coverage is required, but that is a given I think.
>
> Kind regards,
> Bolke
>
> > On 31 May 2017, at 21:10, Peter Dolan <pe...@google.com.INVALID>
> wrote:
> >
> > Hello developers,
> >
> > I work with Google Cloud ML, and my team and I are interested in
> contributing a set of Operators to support working with the Cloud ML
> platform. The platform supports using the TensorFlow deep neural network
> framework as a managed system.
> >
> > In particular, we would like to contribute
> > * CloudMLTrainingOperator, which would launch and monitor a Cloud ML
> Training Job (
> https://cloud.google.com/ml-engine/docs/how-tos/training-jobs <
> https://cloud.google.com/ml-engine/docs/how-tos/training-jobs>),
> > * CloudMLBatchPredictionOperator, which would launch and monitor a
> Cloud ML Batch Prediction Job (
> https://cloud.google.com/ml-engine/docs/how-tos/batch-predict <
> https://cloud.google.com/ml-engine/docs/how-tos/batch-predict>), and
> > * CloudMLVersionOperator, which can create, update, and delete
> TensorFlow model versions (
> https://cloud.google.com/ml-engine/docs/how-tos/managing-models-jobs <
> https://cloud.google.com/ml-engine/docs/how-tos/managing-models-jobs>)
> >
> > I'm eager to hear if the Airflow project is open to these contributions,
> and if any changes are suggested. We have working prototype versions of all
> of them.
> >
> > Thanks in advance,
> > Peter
>
>
Re: Cloud ML Operators
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi Peter,
That sounds great! I think the main criteria for this is will you maintain the code afterwards? The contrib section is slowly but steadily growing and with operators/hooks we are particularly dependent on the community as not all (or even none in some case) of the committers use these themselves.
In any case test coverage is required, but that is a given I think.
Kind regards,
Bolke
> On 31 May 2017, at 21:10, Peter Dolan <pe...@google.com.INVALID> wrote:
>
> Hello developers,
>
> I work with Google Cloud ML, and my team and I are interested in contributing a set of Operators to support working with the Cloud ML platform. The platform supports using the TensorFlow deep neural network framework as a managed system.
>
> In particular, we would like to contribute
> * CloudMLTrainingOperator, which would launch and monitor a Cloud ML Training Job (https://cloud.google.com/ml-engine/docs/how-tos/training-jobs <https://cloud.google.com/ml-engine/docs/how-tos/training-jobs>),
> * CloudMLBatchPredictionOperator, which would launch and monitor a Cloud ML Batch Prediction Job (https://cloud.google.com/ml-engine/docs/how-tos/batch-predict <https://cloud.google.com/ml-engine/docs/how-tos/batch-predict>), and
> * CloudMLVersionOperator, which can create, update, and delete TensorFlow model versions (https://cloud.google.com/ml-engine/docs/how-tos/managing-models-jobs <https://cloud.google.com/ml-engine/docs/how-tos/managing-models-jobs>)
>
> I'm eager to hear if the Airflow project is open to these contributions, and if any changes are suggested. We have working prototype versions of all of them.
>
> Thanks in advance,
> Peter