You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@livy.apache.org by Ryan Skraba <ry...@skraba.com> on 2019/04/16 15:50:59 UTC
We're using Livy!
Hello! I haven't seen many "presentation" emails in the mailing list, but
I thought I'd introduce myself and share how we're using Livy in production.
I'm Ryan, a developer at Talend (not far from Alexey Romanenko and
Jean-Baptiste Onofre), and we've just released Talend Pipeline
Designer[1]. To simplify, we're a graphical front end in the cloud for
designing data pipelines, and an "agent" on an edge node for running them.
For us, Livy is a great match for running pipelines in that agent, and
we've been using it for quite some time now with success!
I believe we've pushed a few changes back to Livy (thanks Alexey!), but for
the most part we're living in a forked repo at some arbitrary commit, with
lots of modifications that are *mostly* only interesting to our jobs.
I'm pretty excited to hear about the release (belated congratulations!) and
I'm currently taking a look at rebasing our work and (1) identifying
features that might be useful (e.g. around docker deployment, job log
enhancements, some optional job argument preprocessing) and (2) getting off
of our fork by isolating our very specific code elsewhere.
I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly
interested in aligning back with the community!
All my best, Ryan
[1] For more info:
https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/
Re: We're using Livy!
Posted by Alexey Romanenko <ar...@gmail.com>.
Hey Ryan,
Great to hear from you and welcome to Livy community!
> On 16 Apr 2019, at 18:05, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi Ryan,
>
> Thanks for the introduction e-mail and welcome !
>
> While ago I already proposed some potential new features to Livy. Especially I proposed to "decouple" Livy from Spark to be able to interact with different execution engines.
>
> I'm planning to move forward on Livy PoCs.
>
> Looking forward to work together.
>
> Regards
> JB
>
> On 16/04/2019 17:50, Ryan Skraba wrote:
>> Hello! I haven't seen many "presentation" emails in the mailing list, but
>> I thought I'd introduce myself and share how we're using Livy in production.
>> I'm Ryan, a developer at Talend (not far from Alexey Romanenko and
>> Jean-Baptiste Onofre), and we've just released Talend Pipeline
>> Designer[1]. To simplify, we're a graphical front end in the cloud for
>> designing data pipelines, and an "agent" on an edge node for running them.
>> For us, Livy is a great match for running pipelines in that agent, and
>> we've been using it for quite some time now with success!
>> I believe we've pushed a few changes back to Livy (thanks Alexey!), but for
>> the most part we're living in a forked repo at some arbitrary commit, with
>> lots of modifications that are *mostly* only interesting to our jobs.
>> I'm pretty excited to hear about the release (belated congratulations!) and
>> I'm currently taking a look at rebasing our work and (1) identifying
>> features that might be useful (e.g. around docker deployment, job log
>> enhancements, some optional job argument preprocessing) and (2) getting off
>> of our fork by isolating our very specific code elsewhere.
>> I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly
>> interested in aligning back with the community!
>> All my best, Ryan
>> [1] For more info:
>> https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/
Re: We're using Livy!
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Jeff,
the idea is to provide kind of agnostic API and being able to leverage
Flink and eventually some Spark "flavors".
With a "Livy API", we can also imagine other engines to support Livy
(I'm thinking about Google Dataflow, etc).
Regards
JB
On 16/04/2019 18:19, Jeff Zhang wrote:
> Hi JB,
>
> I notice the ticket of `decoupling` livy from spark. Which alternative
> engine do you consider to adopt ?
>
>
> Jean-Baptiste Onofré <jb...@nanthrax.net> 于2019年4月17日周三 上午12:06写道：
>
>> Hi Ryan,
>>
>> Thanks for the introduction e-mail and welcome !
>>
>> While ago I already proposed some potential new features to Livy.
>> Especially I proposed to "decouple" Livy from Spark to be able to
>> interact with different execution engines.
>>
>> I'm planning to move forward on Livy PoCs.
>>
>> Looking forward to work together.
>>
>> Regards
>> JB
>>
>> On 16/04/2019 17:50, Ryan Skraba wrote:
>>> Hello! I haven't seen many "presentation" emails in the mailing list,
>> but
>>> I thought I'd introduce myself and share how we're using Livy in
>> production.
>>>
>>> I'm Ryan, a developer at Talend (not far from Alexey Romanenko and
>>> Jean-Baptiste Onofre), and we've just released Talend Pipeline
>>> Designer[1]. To simplify, we're a graphical front end in the cloud for
>>> designing data pipelines, and an "agent" on an edge node for running
>> them.
>>> For us, Livy is a great match for running pipelines in that agent, and
>>> we've been using it for quite some time now with success!
>>>
>>> I believe we've pushed a few changes back to Livy (thanks Alexey!), but
>> for
>>> the most part we're living in a forked repo at some arbitrary commit,
>> with
>>> lots of modifications that are *mostly* only interesting to our jobs.
>>>
>>> I'm pretty excited to hear about the release (belated congratulations!)
>> and
>>> I'm currently taking a look at rebasing our work and (1) identifying
>>> features that might be useful (e.g. around docker deployment, job log
>>> enhancements, some optional job argument preprocessing) and (2) getting
>> off
>>> of our fork by isolating our very specific code elsewhere.
>>>
>>> I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly
>>> interested in aligning back with the community!
>>>
>>> All my best, Ryan
>>>
>>> [1] For more info:
>>>
>> https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/
>>>
>>
>
>
Re: We're using Livy!
Posted by Jeff Zhang <zj...@gmail.com>.
Hi JB,
I notice the ticket of `decoupling` livy from spark. Which alternative
engine do you consider to adopt ?
Jean-Baptiste Onofré <jb...@nanthrax.net> 于2019年4月17日周三 上午12:06写道：
> Hi Ryan,
>
> Thanks for the introduction e-mail and welcome !
>
> While ago I already proposed some potential new features to Livy.
> Especially I proposed to "decouple" Livy from Spark to be able to
> interact with different execution engines.
>
> I'm planning to move forward on Livy PoCs.
>
> Looking forward to work together.
>
> Regards
> JB
>
> On 16/04/2019 17:50, Ryan Skraba wrote:
> > Hello! I haven't seen many "presentation" emails in the mailing list,
> but
> > I thought I'd introduce myself and share how we're using Livy in
> production.
> >
> > I'm Ryan, a developer at Talend (not far from Alexey Romanenko and
> > Jean-Baptiste Onofre), and we've just released Talend Pipeline
> > Designer[1]. To simplify, we're a graphical front end in the cloud for
> > designing data pipelines, and an "agent" on an edge node for running
> them.
> > For us, Livy is a great match for running pipelines in that agent, and
> > we've been using it for quite some time now with success!
> >
> > I believe we've pushed a few changes back to Livy (thanks Alexey!), but
> for
> > the most part we're living in a forked repo at some arbitrary commit,
> with
> > lots of modifications that are *mostly* only interesting to our jobs.
> >
> > I'm pretty excited to hear about the release (belated congratulations!)
> and
> > I'm currently taking a look at rebasing our work and (1) identifying
> > features that might be useful (e.g. around docker deployment, job log
> > enhancements, some optional job argument preprocessing) and (2) getting
> off
> > of our fork by isolating our very specific code elsewhere.
> >
> > I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly
> > interested in aligning back with the community!
> >
> > All my best, Ryan
> >
> > [1] For more info:
> >
> https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/
> >
>
--
Best Regards
Jeff Zhang
Re: We're using Livy!
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Ryan,
Thanks for the introduction e-mail and welcome !
While ago I already proposed some potential new features to Livy.
Especially I proposed to "decouple" Livy from Spark to be able to
interact with different execution engines.
I'm planning to move forward on Livy PoCs.
Looking forward to work together.
Regards
JB
On 16/04/2019 17:50, Ryan Skraba wrote:
> Hello! I haven't seen many "presentation" emails in the mailing list, but
> I thought I'd introduce myself and share how we're using Livy in production.
>
> I'm Ryan, a developer at Talend (not far from Alexey Romanenko and
> Jean-Baptiste Onofre), and we've just released Talend Pipeline
> Designer[1]. To simplify, we're a graphical front end in the cloud for
> designing data pipelines, and an "agent" on an edge node for running them.
> For us, Livy is a great match for running pipelines in that agent, and
> we've been using it for quite some time now with success!
>
> I believe we've pushed a few changes back to Livy (thanks Alexey!), but for
> the most part we're living in a forked repo at some arbitrary commit, with
> lots of modifications that are *mostly* only interesting to our jobs.
>
> I'm pretty excited to hear about the release (belated congratulations!) and
> I'm currently taking a look at rebasing our work and (1) identifying
> features that might be useful (e.g. around docker deployment, job log
> enhancements, some optional job argument preprocessing) and (2) getting off
> of our fork by isolating our very specific code elsewhere.
>
> I also see a LOT of interesting work in the PRs and JIRA, so I'm keenly
> interested in aligning back with the community!
>
> All my best, Ryan
>
> [1] For more info:
> https://www.talend.com/blog/2019/04/16/introducing-pipeline-designer-reinventing-data-integration/
>