You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Luciano Resende <lu...@gmail.com> on 2020/11/02 04:59:50 UTC
A Visual Editor for Airflow pipelines
Hi All,
As mentioned in the user list [1] we are working on a visual editor
for pipelines and adding Airflow as one of the supported backends.
https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
As you are the Airflow devs, we would invite you to help us implement
the best integration possible, in two steps:
1) Getting a solid integration for building and running pipelines with
python scripts and jupyter notebooks
2) Expand the available list component types and enable more generic operators
One of the questions raised in the original e-mail is related to how
to best submit the pipeline dag to be executed by the Airflow runtime,
we have tried a few different options, starting from the experimental
REST API, S3 bucket syncs and these seem to not be the ideal solution,
will be looking into git-sync next, but would really appreciate some
suggestions on the best options, particularly if someone has already
done some external integration similar to this.
Feel free to create issues for discussion and or more details
https://github.com/elyra-ai/elyra/issues
Or use this thread for suggestions
[1] https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: A Visual Editor for Airflow pipelines
Posted by Sumit Maheshwari <su...@gmail.com>.
Is there any open PR in Elyra, where any of the mentioned approach (api,
s3, git-sync etc) has been tried out? I think it's important for Airflow
devs to know the underlying architecture of Elyra to suggest something.
By a quick look at the .pipeline JSON file, it seems like you would need to
do a mapping of it with a serialized DAG's JSON. However, it would solve
only Scheduler & webserver part, for workers some special mechanism would
be needed.
I can join the call as well, if it is scheduled between 8am-11pm India
time.
On Tue, Nov 3, 2020 at 6:21 AM Alan K Chin <ak...@us.ibm.com> wrote:
> Hi All,
>
> Thanks for responding! We are pretty excited about using Airflow with
> Elyra. Regarding a time to meet and chat this week, let's shoot for
> sometime between Wednesday and Friday. We can set something more concrete
> tomorrow once we figure out everyone's timezones and availability.
>
> Look forward to hearing from all of you!
>
> --
> Alan Chin
> CODAIT, San Francisco
> Email - akchin@us.ibm.com
>
>
> ----- Original message -----
> From: Jarek Potiuk <Ja...@polidea.com>
> To: dev@airflow.apache.org
> Cc: luckbr1975@gmail.com, akchin@us.ibm.com
> Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
> Date: Mon, Nov 2, 2020 1:06 PM
>
> Cool!. I also think it's an interesting one:). But it would be great to
> have such integration possible from Elyra :). Let us know what comes out of
> it :).
>
> J.
>
>
> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry...@rywalker.com> wrote:
>
> Hi Luciano -
>
> Elyra looks like an interesting project — we'd love to connect and talk
> through the opportunity.
>
> You can compare your cal to mine and grab a slot here:
> https://calendly.com/ryw/60min
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_ryw_60min&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=aOWVqGrjKRUgW0EWaWXLRLpndFjU1PzNSlI6oWopbsU&e=>
> — and I'll be sure to get a few of the Airflow PMC members to join as well.
>
>
> -Ry
>
> Ry Walker
> Founder/CTO of Astronomer + Airflow Committer
>
>
> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
> wrote:
>
> Hi All,
>
> As mentioned in the user list [1] we are working on a visual editor
> for pipelines and adding Airflow as one of the supported backends.
>
> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__elyra.readthedocs.io_en_latest_user-5Fguide_pipelines.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=8f8kamB87CAFy88IyKQf-8ncdas2jx4HXTXIWlerVfo&e=>
>
> As you are the Airflow devs, we would invite you to help us implement
> the best integration possible, in two steps:
>
> 1) Getting a solid integration for building and running pipelines with
> python scripts and jupyter notebooks
>
> 2) Expand the available list component types and enable more generic
> operators
>
> One of the questions raised in the original e-mail is related to how
> to best submit the pipeline dag to be executed by the Airflow runtime,
> we have tried a few different options, starting from the experimental
> REST API, S3 bucket syncs and these seem to not be the ideal solution,
> will be looking into git-sync next, but would really appreciate some
> suggestions on the best options, particularly if someone has already
> done some external integration similar to this.
>
> Feel free to create issues for discussion and or more details
>
> https://github.com/elyra-ai/elyra/issues
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elyra-2Dai_elyra_issues&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=SGo8ml8UvVVZb6o0hTZmzmZO1PrtNwTnyS2BgY5JE4w&e=>
>
> Or use this thread for suggestions
>
> [1]
> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9-2540-253Cusers.airflow.apache.org-253E&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=FsDxHEyjQeUPKhX6SKQrp3sxoLm6O5JgmohQ1DTcDX8&e=>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=rrHOp_HyFiYz0ObxnOfzX6A_fudgvIJBcptsbNWlZPU&e=>
> http://lresende.blogspot.com/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=licSt0rU7lNnqEB7hRm-In85VLVg-i7MsjTzfj-feQs&e=>
>
>
>
> --
>
> Jarek Potiuk
> Polidea
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=jl0naplQ--zX_Ab1cCycs6qLUgEt6qjWRUHGSONmZ80&e=> |
> Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=jl0naplQ--zX_Ab1cCycs6qLUgEt6qjWRUHGSONmZ80&e=>
>
>
>
>
RE: A Visual Editor for Airflow pipelines
Posted by Alan K Chin <ak...@us.ibm.com>.
Hi All,
Thanks for responding! We are pretty excited about using Airflow with Elyra.
Regarding a time to meet and chat this week, let's shoot for sometime between
Wednesday and Friday. We can set something more concrete tomorrow once we
figure out everyone's timezones and availability.
Look forward to hearing from all of you!
\--
Alan Chin
CODAIT, San Francisco
Email - akchin@us.ibm.com
> \----- Original message -----
> From: Jarek Potiuk <Ja...@polidea.com>
> To: dev@airflow.apache.org
> Cc: luckbr1975@gmail.com, akchin@us.ibm.com
> Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
> Date: Mon, Nov 2, 2020 1:06 PM
>
>
>
> Cool!. I also think it's an interesting one:). But it would be great to have
such integration possible from Elyra :). Let us know what comes out of it :).
>
>
>
> J.
>
>
>
>
>
> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker
<[ry@rywalker.com](mailto:ry@rywalker.com)> wrote:
>
>> Hi Luciano -
>>
>>
>>
>> Elyra looks like an interesting project — we'd love to connect and talk
through the opportunity.
>>
>>
>>
>> You can compare your cal to mine and grab a slot here:
[https://calendly.com/ryw/60min](https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_ryw_60min&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=aOWVqGrjKRUgW0EWaWXLRLpndFjU1PzNSlI6oWopbsU&e=) — and
I'll be sure to get a few of the Airflow PMC members to join as well.
>>
>>
>>
>>
>>
>> -Ry
>>
>>
>>
>> Ry Walker
>>
>> Founder/CTO of Astronomer \+ Airflow Committer
>>
>>
>>
>>
>>
>> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende
<[luckbr1975@gmail.com](mailto:luckbr1975@gmail.com)> wrote:
>>
>>> Hi All,
>
> As mentioned in the user list [1] we are working on a visual editor
> for pipelines and adding Airflow as one of the supported backends.
>
>
[https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html](https://urldefense.proofpoint.com/v2/url?u=https-3A__elyra.readthedocs.io_en_latest_user-5Fguide_pipelines.html&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=8f8kamB87CAFy88IyKQf-8ncdas2jx4HXTXIWlerVfo&e=)
>
> As you are the Airflow devs, we would invite you to help us implement
> the best integration possible, in two steps:
>
> 1) Getting a solid integration for building and running pipelines with
> python scripts and jupyter notebooks
>
> 2) Expand the available list component types and enable more generic
operators
>
> One of the questions raised in the original e-mail is related to how
> to best submit the pipeline dag to be executed by the Airflow runtime,
> we have tried a few different options, starting from the experimental
> REST API, S3 bucket syncs and these seem to not be the ideal solution,
> will be looking into git-sync next, but would really appreciate some
> suggestions on the best options, particularly if someone has already
> done some external integration similar to this.
>
> Feel free to create issues for discussion and or more details
>
> [https://github.com/elyra-
ai/elyra/issues](https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elyra-2Dai_elyra_issues&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=SGo8ml8UvVVZb6o0hTZmzmZO1PrtNwTnyS2BgY5JE4w&e=)
>
> Or use this thread for suggestions
>
> [1]
[https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E](https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9-2540-253Cusers.airflow.apache.org-253E&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=FsDxHEyjQeUPKhX6SKQrp3sxoLm6O5JgmohQ1DTcDX8&e=)
>
>
> \--
> Luciano Resende
>
[http://twitter.com/lresende1975](https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=rrHOp_HyFiYz0ObxnOfzX6A_fudgvIJBcptsbNWlZPU&e=)
>
[http://lresende.blogspot.com/](https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=licSt0rU7lNnqEB7hRm-In85VLVg-i7MsjTzfj-feQs&e=)
>
>
>
>
>
> \--
>
> |
>
> Jarek Potiuk
>
[Polidea](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=jl0naplQ--zX_Ab1cCycs6qLUgEt6qjWRUHGSONmZ80&e=) |
Principal Software Engineer
>
> ---|---
>
> M: [+48 660 796 129](tel:+48660796129)
>
[](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-
GPBbvbR7Shp6MWghSk&s=jl0naplQ--zX_Ab1cCycs6qLUgEt6qjWRUHGSONmZ80&e=)
>
> ---
Re: A Visual Editor for Airflow pipelines
Posted by Tao Feng <fe...@gmail.com>.
Hey Max,
I just saw https://github.com/elyra-ai/elyra . May worth check out for your
use case.
On Wed, Mar 17, 2021 at 10:50 AM Max Payton <mp...@lyft.com.invalid>
wrote:
> We at Lyft are starting to think about developing pipelines using Jupyter,
> and was wondering if there were any notes from this meeting, or follow up
> discussions that might be relevant. We're interested in prior art and
> seeing what's already been done
> *Max Payton*
> He/Him/His
> Software Engineer
> 202.441.7757 <+12024417757>
> [image: Lyft] <http://www.lyft.com/>
>
>
> On Wed, Nov 4, 2020 at 9:29 PM QP Hou <qp...@scribd.com.invalid> wrote:
>
>> On Tue, Nov 3, 2020 at 12:05 PM Alan K Chin <ak...@us.ibm.com> wrote:
>> > @Jarek - It sounds like git-sync is or rather should be the default way
>> users add/modify DAGs. With that said, have you had any experience with
>> customers syncing their dags to other forms of dag storage (S3 etc.) and
>> what the outcomes were?
>> >
>>
>> We have been using S3 for DAG sync in production for more than a year
>> now. The biggest benefit is we basically never have to worry about
>> scalability or availability compared to other solutions. Access
>> control can be managed through IAM roles, which is entirely
>> transparent to application code. On top of that, the S3 event delivery
>> feature can be leveraged to avoid the pulling loop to make sync almost
>> real time. Only downside is you need to set up a CI/CD pipeline to
>> publish DAG changes to S3. I wrote about our implementation in our
>> tech blog at
>> https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html.
>>
>
Re: A Visual Editor for Airflow pipelines
Posted by Max Payton <mp...@lyft.com.INVALID>.
We at Lyft are starting to think about developing pipelines using Jupyter,
and was wondering if there were any notes from this meeting, or follow up
discussions that might be relevant. We're interested in prior art and
seeing what's already been done
*Max Payton*
He/Him/His
Software Engineer
202.441.7757 <+12024417757>
[image: Lyft] <http://www.lyft.com/>
On Wed, Nov 4, 2020 at 9:29 PM QP Hou <qp...@scribd.com.invalid> wrote:
> On Tue, Nov 3, 2020 at 12:05 PM Alan K Chin <ak...@us.ibm.com> wrote:
> > @Jarek - It sounds like git-sync is or rather should be the default way
> users add/modify DAGs. With that said, have you had any experience with
> customers syncing their dags to other forms of dag storage (S3 etc.) and
> what the outcomes were?
> >
>
> We have been using S3 for DAG sync in production for more than a year
> now. The biggest benefit is we basically never have to worry about
> scalability or availability compared to other solutions. Access
> control can be managed through IAM roles, which is entirely
> transparent to application code. On top of that, the S3 event delivery
> feature can be leveraged to avoid the pulling loop to make sync almost
> real time. Only downside is you need to set up a CI/CD pipeline to
> publish DAG changes to S3. I wrote about our implementation in our
> tech blog at
> https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html.
>
Re: A Visual Editor for Airflow pipelines
Posted by QP Hou <qp...@scribd.com.INVALID>.
On Tue, Nov 3, 2020 at 12:05 PM Alan K Chin <ak...@us.ibm.com> wrote:
> @Jarek - It sounds like git-sync is or rather should be the default way users add/modify DAGs. With that said, have you had any experience with customers syncing their dags to other forms of dag storage (S3 etc.) and what the outcomes were?
>
We have been using S3 for DAG sync in production for more than a year
now. The biggest benefit is we basically never have to worry about
scalability or availability compared to other solutions. Access
control can be managed through IAM roles, which is entirely
transparent to application code. On top of that, the S3 event delivery
feature can be leveraged to avoid the pulling loop to make sync almost
real time. Only downside is you need to set up a CI/CD pipeline to
publish DAG changes to S3. I wrote about our implementation in our
tech blog at https://tech.scribd.com/blog/2020/breaking-up-the-dag-repo.html.
Re: A Visual Editor for Airflow pipelines
Posted by Jarek Potiuk <Ja...@polidea.com>.
Which time zone 1pm Friday :) ?
Answering the question - I think there are lots of horror stories about
using the EFS (regarding the quota, guaranteed throughput). I've not tried
NFS, but with the lack of cloud-nativeness and characteristics, problems
might be the same. From the engineering POV, those are exactly the kind of
problems you can expect when you synchronize a bunch of files in the hope
that they will stay consistent across a folder. There is a rather bad
experience about using GCS-fuse in a similar setup (but mostly anecdotal).
The main problem with those is very little control you have over the
transfer and delays, some scenarios which are resulting in huge delays and
lack of consistency. Git solves pretty much all of it. Nearly atomic DAG
sync (all files have to be present locally to be checked out, so while
checkout is not transactional atomic, it is as close as it can be when it
comes to remote file sync). Minimizing the traffic based on git diff
analysis also helps. So I think this is pretty nice characteristics for DAG
sync overall
J.
On Tue, Nov 3, 2020 at 9:55 PM Ry Walker <ry...@rywalker.com> wrote:
> Let’s go w/ Friday option
>
> On Tue, Nov 3, 2020 at 3:05 PM Alan K Chin <ak...@us.ibm.com> wrote:
>
>> @Tomek - Thanks for the link to the Airflow talk ! Checking it out now.
>>
>> @Jarek - It sounds like git-sync is or rather should be the default way
>> users add/modify DAGs. With that said, have you had any experience with
>> customers syncing their dags to other forms of dag storage (S3 etc.) and
>> what the outcomes were?
>>
>> I spoke with Luciano and we're both available after anytime after noon on
>> Thursday and Friday to chat about this effort.
>> @Ry - Looking at your calendar Thurs@12:30 and Fri@1pm both look open,
>> which one would fit your schedule best?
>>
>> Look forward to chatting.
>>
>> --
>> Alan Chin
>> CODAIT, San Francisco
>> Email - akchin@us.ibm.com
>>
>>
>> ----- Original message -----
>> From: Tomasz Urbaszek <tu...@apache.org>
>> To: dev@airflow.apache.org
>> Cc: "luckbr1975@gmail.com" <lu...@gmail.com>, "akchin@us.ibm.com" <
>> akchin@us.ibm.com>
>> Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
>> Date: Tue, Nov 3, 2020 3:29 AM
>>
>> I think the visual DAG editor would be a thing!
>>
>> Not sure if you are aware of this Airflow Summit talk about visual DAG
>> editor and the integration between Airflow and CWL:
>>
>> https://www.youtube.com/watch?v=I4nFCqEnOJc&list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj&index=19
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DI4nFCqEnOJc-26list-3DPLGudixcDaxY3RGLSlWoN-5FcEEXhIT1OPmj-26index-3D19&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=RZHlZzsdqSjQy2fMfUR1RBROmSi6vC1jw7g-ulLp1KI&e=>
>>
>> Cheers,
>> Tomek
>>
>> On Tue, Nov 3, 2020 at 11:42 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> Agree - maybe 2.1 or 2.2 :).
>>
>> After some experiences with big customers deployments, I personally think
>> GitSync at least for now is the best approach out there. It requires git
>> repo + authorization, but this has all the added benefits of code change
>> tracking, it is a very standard interface, most of the git repos provide
>> some ways of manual review if needed and most have some kind of integration
>> with CI/automated code analysis.
>>
>> I personally think it should be the default, for any serious deployment
>> as it provides so many benefits with very limited extra. You just need an
>> extra "box" - git repo (which is pretty much a given in any organization).
>> It uses a standard interface that is highly customizable (branches/folder
>> structures, whatnot) and we already have git-sync container support in the
>> helm chart.
>>
>> J.
>>
>>
>> On Tue, Nov 3, 2020 at 11:27 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> Wishfull thinking at the moment Gerard -- the task execution still needs
>> files on disk to run the tasks.
>>
>> This was always in my long term plan for DAG serialization, but we aren't
>> there yet. And Custom operators makes this a non-straight forward problem
>> to solve.
>>
>> -ash
>>
>> On Nov 3 2020, at 12:18 am, Gerard Casas Saez
>> <gc...@twitter.com.INVALID> wrote:
>>
>> Would be interested to also know possible ways to do what Luciano
>> described. Hopefully w the serialized DAG and the new API we can start just
>> pushing the DAG to the DB (wishful thinking)?
>>
>> Gerard Casas Saez
>> Twitter | Cortex | @casassaez
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_casassaez&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=TRFzqZAGPxdwbYD4a2k_L9devsCrVoHDra_jpjBOe5c&e=>
>>
>> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> Cool!. I also think it's an interesting one:). But it would be great to
>> have such integration possible from Elyra :). Let us know what comes out of
>> it :).
>>
>> J.
>>
>>
>> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry...@rywalker.com> wrote:
>>
>> Hi Luciano -
>>
>> Elyra looks like an interesting project — we'd love to connect and talk
>> through the opportunity.
>>
>> You can compare your cal to mine and grab a slot here:
>> https://calendly.com/ryw/60min
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_ryw_60min&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=f8ijoSoMQ0xgE7EeFcRY9HG0dvXeDY2t5PStPcQI0NU&e=> —
>> and I'll be sure to get a few of the Airflow PMC members to join as well.
>>
>> -Ry
>>
>> Ry Walker
>> Founder/CTO of Astronomer + Airflow Committer
>>
>>
>> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> As mentioned in the user list [1] we are working on a visual editor
>> for pipelines and adding Airflow as one of the supported backends.
>>
>> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__elyra.readthedocs.io_en_latest_user-5Fguide_pipelines.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Zc2Gn-FE20z6ShPCPYrUo7lgTmpV6izvmUDL-VcoKPw&e=>
>>
>> As you are the Airflow devs, we would invite you to help us implement
>> the best integration possible, in two steps:
>>
>> 1) Getting a solid integration for building and running pipelines with
>> python scripts and jupyter notebooks
>>
>> 2) Expand the available list component types and enable more generic
>> operators
>>
>> One of the questions raised in the original e-mail is related to how
>> to best submit the pipeline dag to be executed by the Airflow runtime,
>> we have tried a few different options, starting from the experimental
>> REST API, S3 bucket syncs and these seem to not be the ideal solution,
>> will be looking into git-sync next, but would really appreciate some
>> suggestions on the best options, particularly if someone has already
>> done some external integration similar to this.
>>
>> Feel free to create issues for discussion and or more details
>>
>> https://github.com/elyra-ai/elyra/issues
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elyra-2Dai_elyra_issues&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=JmRvVcY5ICgzHm299LUrcP10D6Yw92A8fjC-MNKR8-0&e=>
>>
>> Or use this thread for suggestions
>>
>> [1]
>> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9-2540-253Cusers.airflow.apache.org-253E&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=51idLXv3vU7jUPVsRZaXNb_2CY8dadrqUs-IQdk9mPc&e=>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=O7PBvuGPX4hCZer2guxof7kqzYbfnaZRQWRb3XbUJQ8&e=>
>> http://lresende.blogspot.com/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=IrrouExrZxjMSpsrnRQXBqngSBPfdFFD2C521sVzGEI&e=>
>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=> |
>> Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=>
>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=> |
>> Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=>
>>
>>
>>
>> --
> Sent from Gmail Mobile
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: A Visual Editor for Airflow pipelines
Posted by Ry Walker <ry...@rywalker.com>.
Let’s go w/ Friday option
On Tue, Nov 3, 2020 at 3:05 PM Alan K Chin <ak...@us.ibm.com> wrote:
> @Tomek - Thanks for the link to the Airflow talk ! Checking it out now.
>
> @Jarek - It sounds like git-sync is or rather should be the default way
> users add/modify DAGs. With that said, have you had any experience with
> customers syncing their dags to other forms of dag storage (S3 etc.) and
> what the outcomes were?
>
> I spoke with Luciano and we're both available after anytime after noon on
> Thursday and Friday to chat about this effort.
> @Ry - Looking at your calendar Thurs@12:30 and Fri@1pm both look open,
> which one would fit your schedule best?
>
> Look forward to chatting.
>
> --
> Alan Chin
> CODAIT, San Francisco
> Email - akchin@us.ibm.com
>
>
> ----- Original message -----
> From: Tomasz Urbaszek <tu...@apache.org>
> To: dev@airflow.apache.org
> Cc: "luckbr1975@gmail.com" <lu...@gmail.com>, "akchin@us.ibm.com" <
> akchin@us.ibm.com>
> Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
> Date: Tue, Nov 3, 2020 3:29 AM
>
> I think the visual DAG editor would be a thing!
>
> Not sure if you are aware of this Airflow Summit talk about visual DAG
> editor and the integration between Airflow and CWL:
>
> https://www.youtube.com/watch?v=I4nFCqEnOJc&list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj&index=19
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DI4nFCqEnOJc-26list-3DPLGudixcDaxY3RGLSlWoN-5FcEEXhIT1OPmj-26index-3D19&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=RZHlZzsdqSjQy2fMfUR1RBROmSi6vC1jw7g-ulLp1KI&e=>
>
> Cheers,
> Tomek
>
> On Tue, Nov 3, 2020 at 11:42 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> Agree - maybe 2.1 or 2.2 :).
>
> After some experiences with big customers deployments, I personally think
> GitSync at least for now is the best approach out there. It requires git
> repo + authorization, but this has all the added benefits of code change
> tracking, it is a very standard interface, most of the git repos provide
> some ways of manual review if needed and most have some kind of integration
> with CI/automated code analysis.
>
> I personally think it should be the default, for any serious deployment as
> it provides so many benefits with very limited extra. You just need an
> extra "box" - git repo (which is pretty much a given in any organization).
> It uses a standard interface that is highly customizable (branches/folder
> structures, whatnot) and we already have git-sync container support in the
> helm chart.
>
> J.
>
>
> On Tue, Nov 3, 2020 at 11:27 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Wishfull thinking at the moment Gerard -- the task execution still needs
> files on disk to run the tasks.
>
> This was always in my long term plan for DAG serialization, but we aren't
> there yet. And Custom operators makes this a non-straight forward problem
> to solve.
>
> -ash
>
> On Nov 3 2020, at 12:18 am, Gerard Casas Saez
> <gc...@twitter.com.INVALID> wrote:
>
> Would be interested to also know possible ways to do what Luciano
> described. Hopefully w the serialized DAG and the new API we can start just
> pushing the DAG to the DB (wishful thinking)?
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_casassaez&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=TRFzqZAGPxdwbYD4a2k_L9devsCrVoHDra_jpjBOe5c&e=>
>
> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> Cool!. I also think it's an interesting one:). But it would be great to
> have such integration possible from Elyra :). Let us know what comes out of
> it :).
>
> J.
>
>
> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry...@rywalker.com> wrote:
>
> Hi Luciano -
>
> Elyra looks like an interesting project — we'd love to connect and talk
> through the opportunity.
>
> You can compare your cal to mine and grab a slot here:
> https://calendly.com/ryw/60min
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_ryw_60min&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=f8ijoSoMQ0xgE7EeFcRY9HG0dvXeDY2t5PStPcQI0NU&e=> —
> and I'll be sure to get a few of the Airflow PMC members to join as well.
>
> -Ry
>
> Ry Walker
> Founder/CTO of Astronomer + Airflow Committer
>
>
> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
> wrote:
>
> Hi All,
>
> As mentioned in the user list [1] we are working on a visual editor
> for pipelines and adding Airflow as one of the supported backends.
>
> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__elyra.readthedocs.io_en_latest_user-5Fguide_pipelines.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Zc2Gn-FE20z6ShPCPYrUo7lgTmpV6izvmUDL-VcoKPw&e=>
>
> As you are the Airflow devs, we would invite you to help us implement
> the best integration possible, in two steps:
>
> 1) Getting a solid integration for building and running pipelines with
> python scripts and jupyter notebooks
>
> 2) Expand the available list component types and enable more generic
> operators
>
> One of the questions raised in the original e-mail is related to how
> to best submit the pipeline dag to be executed by the Airflow runtime,
> we have tried a few different options, starting from the experimental
> REST API, S3 bucket syncs and these seem to not be the ideal solution,
> will be looking into git-sync next, but would really appreciate some
> suggestions on the best options, particularly if someone has already
> done some external integration similar to this.
>
> Feel free to create issues for discussion and or more details
>
> https://github.com/elyra-ai/elyra/issues
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elyra-2Dai_elyra_issues&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=JmRvVcY5ICgzHm299LUrcP10D6Yw92A8fjC-MNKR8-0&e=>
>
> Or use this thread for suggestions
>
> [1]
> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9-2540-253Cusers.airflow.apache.org-253E&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=51idLXv3vU7jUPVsRZaXNb_2CY8dadrqUs-IQdk9mPc&e=>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=O7PBvuGPX4hCZer2guxof7kqzYbfnaZRQWRb3XbUJQ8&e=>
> http://lresende.blogspot.com/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=IrrouExrZxjMSpsrnRQXBqngSBPfdFFD2C521sVzGEI&e=>
>
>
>
> --
>
> Jarek Potiuk
> Polidea
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=> |
> Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=>
>
>
>
> --
>
> Jarek Potiuk
> Polidea
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=> |
> Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-iNEukXNEGzJ2Fl_s&e=>
>
>
>
> --
Sent from Gmail Mobile
RE: A Visual Editor for Airflow pipelines
Posted by Alan K Chin <ak...@us.ibm.com>.
@Tomek - Thanks for the link to the Airflow talk ! Checking it out now.
@Jarek - It sounds like git-sync is or rather should be the default way users
add/modify DAGs. With that said, have you had any experience with customers
syncing their dags to other forms of dag storage (S3 etc.) and what the
outcomes were?
I spoke with Luciano and we're both available after anytime after noon on
Thursday and Friday to chat about this effort.
@Ry - Looking at your calendar Thurs@12:30 and Fri@1pm both look open, which
one would fit your schedule best?
Look forward to chatting.
\--
Alan Chin
CODAIT, San Francisco
Email - akchin@us.ibm.com
> \----- Original message -----
> From: Tomasz Urbaszek <tu...@apache.org>
> To: dev@airflow.apache.org
> Cc: "luckbr1975@gmail.com" <lu...@gmail.com>, "akchin@us.ibm.com"
<ak...@us.ibm.com>
> Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
> Date: Tue, Nov 3, 2020 3:29 AM
>
>
>
> I think the visual DAG editor would be a thing!
>
> Not sure if you are aware of this Airflow Summit talk about visual DAG
editor and the integration between Airflow and CWL:
>
[https://www.youtube.com/watch?v=I4nFCqEnOJc&list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj&index=19](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DI4nFCqEnOJc-26list-3DPLGudixcDaxY3RGLSlWoN-5FcEEXhIT1OPmj-26index-3D19&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=RZHlZzsdqSjQy2fMfUR1RBROmSi6vC1jw7g-ulLp1KI&e=)
>
> Cheers,
> Tomek
>
>
>
> On Tue, Nov 3, 2020 at 11:42 AM Jarek Potiuk
<[Jarek.Potiuk@polidea.com](mailto:Jarek.Potiuk@polidea.com)> wrote:
>
>> Agree \- maybe 2.1 or 2.2 :).
>>
>>
>>
>> After some experiences with big customers deployments, I personally think
GitSync at least for now is the best approach out there. It requires git repo
\+ authorization, but this has all the added benefits of code change tracking,
it is a very standard interface, most of the git repos provide some ways of
manual review if needed and most have some kind of integration with
CI/automated code analysis.
>>
>>
>>
>> I personally think it should be the default, for any serious deployment as
it provides so many benefits with very limited extra. You just need an extra
"box" - git repo (which is pretty much a given in any organization). It uses a
standard interface that is highly customizable (branches/folder structures,
whatnot) and we already have git-sync container support in the helm chart.
>>
>>
>>
>> J.
>>
>>
>>
>>
>>
>> On Tue, Nov 3, 2020 at 11:27 AM Ash Berlin-Taylor
<[ash@apache.org](mailto:ash@apache.org)> wrote:
>>
>>> Wishfull thinking at the moment Gerard -- the task execution still needs
files on disk to run the tasks.
>>>
>>>
>>>
>>> This was always in my long term plan for DAG serialization, but we aren't
there yet. And Custom operators makes this a non-straight forward problem to
solve.
>>>
>>>
>>>
>>> -ash
>>>
>>>
>>>
>>> On Nov 3 2020, at 12:18 am, Gerard Casas Saez
<gc...@twitter.com.INVALID> wrote:
>>>
>>>> Would be interested to also know possible ways to do what Luciano
described. Hopefully w the serialized DAG and the new API we can start just
pushing the DAG to the DB (wishful thinking)?
>>>>
>>>>
>>>>
>>>> Gerard Casas Saez
>>>>
>>>> Twitter | Cortex |
[@casassaez](https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_casassaez&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=TRFzqZAGPxdwbYD4a2k_L9devsCrVoHDra_jpjBOe5c&e=
"http://twitter.com/casassaez")
>>>>
>>>>
>>>>
>>>> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk
<[Jarek.Potiuk@polidea.com](mailto:Jarek.Potiuk@polidea.com
"mailto:Jarek.Potiuk@polidea.com")> wrote:
>>>>
>>>>> Cool!. I also think it's an interesting one:). But it would be great to
have such integration possible from Elyra :). Let us know what comes out of it
:).
>>>>>
>>>>>
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker
<[ry@rywalker.com](mailto:ry@rywalker.com "mailto:ry@rywalker.com")> wrote:
>>>>>
>>>>>> Hi Luciano -
>>>>>>
>>>>>>
>>>>>>
>>>>>> Elyra looks like an interesting project — we'd love to connect and talk
through the opportunity.
>>>>>>
>>>>>>
>>>>>>
>>>>>> You can compare your cal to mine and grab a slot here:
[https://calendly.com/ryw/60min](https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_ryw_60min&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=f8ijoSoMQ0xgE7EeFcRY9HG0dvXeDY2t5PStPcQI0NU&e=
"https://calendly.com/ryw/60min") — and I'll be sure to get a few of the
Airflow PMC members to join as well.
>>>>>>
>>>>>>
>>>>>>
>>>>>> -Ry
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ry Walker
>>>>>>
>>>>>> Founder/CTO of Astronomer \+ Airflow Committer
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende
<[luckbr1975@gmail.com](mailto:luckbr1975@gmail.com
"mailto:luckbr1975@gmail.com")> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> As mentioned in the user list [1] we are working on a visual editor
>>>>>>>
>>>>>>> for pipelines and adding Airflow as one of the supported backends.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
[https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html](https://urldefense.proofpoint.com/v2/url?u=https-3A__elyra.readthedocs.io_en_latest_user-5Fguide_pipelines.html&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Zc2Gn-
FE20z6ShPCPYrUo7lgTmpV6izvmUDL-VcoKPw&e=
"https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html")
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> As you are the Airflow devs, we would invite you to help us implement
>>>>>>>
>>>>>>> the best integration possible, in two steps:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 1) Getting a solid integration for building and running pipelines with
>>>>>>>
>>>>>>> python scripts and jupyter notebooks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2) Expand the available list component types and enable more generic
operators
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> One of the questions raised in the original e-mail is related to how
>>>>>>>
>>>>>>> to best submit the pipeline dag to be executed by the Airflow runtime,
>>>>>>>
>>>>>>> we have tried a few different options, starting from the experimental
>>>>>>>
>>>>>>> REST API, S3 bucket syncs and these seem to not be the ideal solution,
>>>>>>>
>>>>>>> will be looking into git-sync next, but would really appreciate some
>>>>>>>
>>>>>>> suggestions on the best options, particularly if someone has already
>>>>>>>
>>>>>>> done some external integration similar to this.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Feel free to create issues for discussion and or more details
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [https://github.com/elyra-
ai/elyra/issues](https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elyra-2Dai_elyra_issues&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=JmRvVcY5ICgzHm299LUrcP10D6Yw92A8fjC-
MNKR8-0&e= "https://github.com/elyra-ai/elyra/issues")
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Or use this thread for suggestions
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [1]
[https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E](https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9-2540-253Cusers.airflow.apache.org-253E&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=51idLXv3vU7jUPVsRZaXNb_2CY8dadrqUs-
IQdk9mPc&e=
"https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E")
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> \--
>>>>>>>
>>>>>>> Luciano Resende
>>>>>>>
>>>>>>>
[http://twitter.com/lresende1975](https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=O7PBvuGPX4hCZer2guxof7kqzYbfnaZRQWRb3XbUJQ8&e=
"http://twitter.com/lresende1975")
>>>>>>>
>>>>>>>
[http://lresende.blogspot.com/](https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=IrrouExrZxjMSpsrnRQXBqngSBPfdFFD2C521sVzGEI&e=
"http://lresende.blogspot.com/")
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> \--
>>>>>
>>>>> |
>>>>>
>>>>> Jarek Potiuk
>
[Polidea](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=) | Principal Software Engineer
>>>>>
>>>>> ---|---
>>>>>
>>>>> M: [+48 660 796 129](tel:+48660796129)
>
[](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=)
>>>>>
>>>>> ---
>>
>>
>>
>>
>>
>> \--
>>
>> |
>>
>> Jarek Potiuk
>
[Polidea](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=) | Principal Software Engineer
>>
>> ---|---
>>
>> M: [+48 660 796 129](tel:+48660796129)
>
[](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=)
>>
>> ---
RE: A Visual Editor for Airflow pipelines
Posted by Alan K Chin <ak...@us.ibm.com>.
Hello All,
I've scheduled a 60 minute call on Friday, Nov 6th @ 1PM PST to chat about
this integration and have some Q and A.
The meeting link is : <https://astronomer.zoom.us/j/5134172163> (thank you Ry
for zoom link and usage)
\--
Alan Chin
CODAIT, San Francisco
Email - akchin@us.ibm.com
> \----- Original message -----
> From: Alan K Chin/Silicon Valley/IBM
> To: turbaszek@apache.org
> Cc: dev@airflow.apache.org, luckbr1975@gmail.com
> Subject: Re: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
> Date: Tue, Nov 3, 2020 11:33 AM
>
>
> @Tomek - Thanks for the link to the Airflow talk ! Checking it out now.
>
>
>
> @Jarek - It sounds like git-sync is or rather should be the default way
users add/modify DAGs. With that said, have you had any experience with
customers syncing their dags to other forms of dag storage (S3 etc.) and what
the outcomes were?
>
>
>
> I spoke with Luciano and we're both available after anytime after noon on
Thursday and Friday to chat about this effort.
>
> @Ry - Looking at your calendar Thurs@12:30 and Fri@1pm both look open, which
one would fit your schedule best?
>
>
>
> Look forward to chatting.
>
>
>
> \--
> Alan Chin
> CODAIT, San Francisco
> Email - akchin@us.ibm.com
>
>
>
>> \----- Original message -----
> From: Tomasz Urbaszek <tu...@apache.org>
> To: dev@airflow.apache.org
> Cc: "luckbr1975@gmail.com" <lu...@gmail.com>, "akchin@us.ibm.com"
<ak...@us.ibm.com>
> Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines
> Date: Tue, Nov 3, 2020 3:29 AM
>
>
>>
>> I think the visual DAG editor would be a thing!
>
> Not sure if you are aware of this Airflow Summit talk about visual DAG
editor and the integration between Airflow and CWL:
>
[https://www.youtube.com/watch?v=I4nFCqEnOJc&list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj&index=19](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_watch-3Fv-3DI4nFCqEnOJc-26list-3DPLGudixcDaxY3RGLSlWoN-5FcEEXhIT1OPmj-26index-3D19&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=RZHlZzsdqSjQy2fMfUR1RBROmSi6vC1jw7g-ulLp1KI&e=)
>
> Cheers,
> Tomek
>>
>>
>>
>> On Tue, Nov 3, 2020 at 11:42 AM Jarek Potiuk
<[Jarek.Potiuk@polidea.com](mailto:Jarek.Potiuk@polidea.com)> wrote:
>>
>>> Agree \- maybe 2.1 or 2.2 :).
>>>
>>>
>>>
>>> After some experiences with big customers deployments, I personally think
GitSync at least for now is the best approach out there. It requires git repo
\+ authorization, but this has all the added benefits of code change tracking,
it is a very standard interface, most of the git repos provide some ways of
manual review if needed and most have some kind of integration with
CI/automated code analysis.
>>>
>>>
>>>
>>> I personally think it should be the default, for any serious deployment as
it provides so many benefits with very limited extra. You just need an extra
"box" - git repo (which is pretty much a given in any organization). It uses a
standard interface that is highly customizable (branches/folder structures,
whatnot) and we already have git-sync container support in the helm chart.
>>>
>>>
>>>
>>> J.
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Nov 3, 2020 at 11:27 AM Ash Berlin-Taylor
<[ash@apache.org](mailto:ash@apache.org)> wrote:
>>>
>>>> Wishfull thinking at the moment Gerard -- the task execution still needs
files on disk to run the tasks.
>>>>
>>>>
>>>>
>>>> This was always in my long term plan for DAG serialization, but we aren't
there yet. And Custom operators makes this a non-straight forward problem to
solve.
>>>>
>>>>
>>>>
>>>> -ash
>>>>
>>>>
>>>>
>>>> On Nov 3 2020, at 12:18 am, Gerard Casas Saez
<gc...@twitter.com.INVALID> wrote:
>>>>
>>>>> Would be interested to also know possible ways to do what Luciano
described. Hopefully w the serialized DAG and the new API we can start just
pushing the DAG to the DB (wishful thinking)?
>>>>>
>>>>>
>>>>>
>>>>> Gerard Casas Saez
>>>>>
>>>>> Twitter | Cortex |
[@casassaez](https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_casassaez&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=TRFzqZAGPxdwbYD4a2k_L9devsCrVoHDra_jpjBOe5c&e=
"http://twitter.com/casassaez")
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk
<[Jarek.Potiuk@polidea.com](mailto:Jarek.Potiuk@polidea.com
"mailto:Jarek.Potiuk@polidea.com")> wrote:
>>>>>
>>>>>> Cool!. I also think it's an interesting one:). But it would be great to
have such integration possible from Elyra :). Let us know what comes out of it
:).
>>>>>>
>>>>>>
>>>>>>
>>>>>> J.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker
<[ry@rywalker.com](mailto:ry@rywalker.com "mailto:ry@rywalker.com")> wrote:
>>>>>>
>>>>>>> Hi Luciano -
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Elyra looks like an interesting project — we'd love to connect and
talk through the opportunity.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> You can compare your cal to mine and grab a slot here:
[https://calendly.com/ryw/60min](https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_ryw_60min&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=f8ijoSoMQ0xgE7EeFcRY9HG0dvXeDY2t5PStPcQI0NU&e=
"https://calendly.com/ryw/60min") — and I'll be sure to get a few of the
Airflow PMC members to join as well.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Ry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ry Walker
>>>>>>>
>>>>>>> Founder/CTO of Astronomer \+ Airflow Committer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende
<[luckbr1975@gmail.com](mailto:luckbr1975@gmail.com
"mailto:luckbr1975@gmail.com")> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> As mentioned in the user list [1] we are working on a visual editor
>>>>>>>>
>>>>>>>> for pipelines and adding Airflow as one of the supported backends.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
[https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html](https://urldefense.proofpoint.com/v2/url?u=https-3A__elyra.readthedocs.io_en_latest_user-5Fguide_pipelines.html&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Zc2Gn-
FE20z6ShPCPYrUo7lgTmpV6izvmUDL-VcoKPw&e=
"https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html")
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> As you are the Airflow devs, we would invite you to help us implement
>>>>>>>>
>>>>>>>> the best integration possible, in two steps:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 1) Getting a solid integration for building and running pipelines
with
>>>>>>>>
>>>>>>>> python scripts and jupyter notebooks
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2) Expand the available list component types and enable more generic
operators
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> One of the questions raised in the original e-mail is related to how
>>>>>>>>
>>>>>>>> to best submit the pipeline dag to be executed by the Airflow
runtime,
>>>>>>>>
>>>>>>>> we have tried a few different options, starting from the experimental
>>>>>>>>
>>>>>>>> REST API, S3 bucket syncs and these seem to not be the ideal
solution,
>>>>>>>>
>>>>>>>> will be looking into git-sync next, but would really appreciate some
>>>>>>>>
>>>>>>>> suggestions on the best options, particularly if someone has already
>>>>>>>>
>>>>>>>> done some external integration similar to this.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Feel free to create issues for discussion and or more details
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [https://github.com/elyra-
ai/elyra/issues](https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elyra-2Dai_elyra_issues&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=JmRvVcY5ICgzHm299LUrcP10D6Yw92A8fjC-
MNKR8-0&e= "https://github.com/elyra-ai/elyra/issues")
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Or use this thread for suggestions
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
[https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E](https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9-2540-253Cusers.airflow.apache.org-253E&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=51idLXv3vU7jUPVsRZaXNb_2CY8dadrqUs-
IQdk9mPc&e=
"https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E")
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> \--
>>>>>>>>
>>>>>>>> Luciano Resende
>>>>>>>>
>>>>>>>>
[http://twitter.com/lresende1975](https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=O7PBvuGPX4hCZer2guxof7kqzYbfnaZRQWRb3XbUJQ8&e=
"http://twitter.com/lresende1975")
>>>>>>>>
>>>>>>>>
[http://lresende.blogspot.com/](https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=IrrouExrZxjMSpsrnRQXBqngSBPfdFFD2C521sVzGEI&e=
"http://lresende.blogspot.com/")
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> \--
>>>>>>
>>>>>> |
>>>>>>
>>>>>> Jarek Potiuk
>
[Polidea](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=) | Principal Software Engineer
>>>>>>
>>>>>> ---|---
>>>>>>
>>>>>> M: [+48 660 796 129](tel:+48660796129)
>
[](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=)
>>>>>>
>>>>>> ---
>>>
>>>
>>>
>>>
>>>
>>> \--
>>>
>>> |
>>>
>>> Jarek Potiuk
>
[Polidea](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=) | Principal Software Engineer
>>>
>>> ---|---
>>>
>>> M: [+48 660 796 129](tel:+48660796129)
>
[](https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-
siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=2XxJ21Px6dUbjMcXagRxWE1CbiyJeVeQJ88qfDCfFNk&s=Bti6AwFi8_yNtjHW5JYqHnkfXz-
iNEukXNEGzJ2Fl_s&e=)
>>>
>>> ---
>
>
Re: A Visual Editor for Airflow pipelines
Posted by Tomasz Urbaszek <tu...@apache.org>.
I think the visual DAG editor would be a thing!
Not sure if you are aware of this Airflow Summit talk about visual DAG
editor and the integration between Airflow and CWL:
https://www.youtube.com/watch?v=I4nFCqEnOJc&list=PLGudixcDaxY3RGLSlWoN_cEEXhIT1OPmj&index=19
Cheers,
Tomek
On Tue, Nov 3, 2020 at 11:42 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Agree - maybe 2.1 or 2.2 :).
>
> After some experiences with big customers deployments, I personally think
> GitSync at least for now is the best approach out there. It requires git
> repo + authorization, but this has all the added benefits of code change
> tracking, it is a very standard interface, most of the git repos provide
> some ways of manual review if needed and most have some kind of integration
> with CI/automated code analysis.
>
> I personally think it should be the default, for any serious deployment as
> it provides so many benefits with very limited extra. You just need an
> extra "box" - git repo (which is pretty much a given in any organization).
> It uses a standard interface that is highly customizable (branches/folder
> structures, whatnot) and we already have git-sync container support in the
> helm chart.
>
> J.
>
>
> On Tue, Nov 3, 2020 at 11:27 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> Wishfull thinking at the moment Gerard -- the task execution still needs
>> files on disk to run the tasks.
>>
>> This was always in my long term plan for DAG serialization, but we aren't
>> there yet. And Custom operators makes this a non-straight forward problem
>> to solve.
>>
>> -ash
>>
>> On Nov 3 2020, at 12:18 am, Gerard Casas Saez
>> <gc...@twitter.com.INVALID> wrote:
>>
>> Would be interested to also know possible ways to do what Luciano
>> described. Hopefully w the serialized DAG and the new API we can start just
>> pushing the DAG to the DB (wishful thinking)?
>>
>> Gerard Casas Saez
>> Twitter | Cortex | @casassaez <http://twitter.com/casassaez>
>>
>>
>> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> Cool!. I also think it's an interesting one:). But it would be great to
>> have such integration possible from Elyra :). Let us know what comes out of
>> it :).
>>
>> J.
>>
>>
>> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry...@rywalker.com> wrote:
>>
>> Hi Luciano -
>>
>> Elyra looks like an interesting project — we'd love to connect and talk
>> through the opportunity.
>>
>> You can compare your cal to mine and grab a slot here:
>> https://calendly.com/ryw/60min — and I'll be sure to get a few of the
>> Airflow PMC members to join as well.
>>
>> -Ry
>>
>> Ry Walker
>> Founder/CTO of Astronomer + Airflow Committer
>>
>>
>> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> As mentioned in the user list [1] we are working on a visual editor
>> for pipelines and adding Airflow as one of the supported backends.
>>
>> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
>>
>> As you are the Airflow devs, we would invite you to help us implement
>> the best integration possible, in two steps:
>>
>> 1) Getting a solid integration for building and running pipelines with
>> python scripts and jupyter notebooks
>>
>> 2) Expand the available list component types and enable more generic
>> operators
>>
>> One of the questions raised in the original e-mail is related to how
>> to best submit the pipeline dag to be executed by the Airflow runtime,
>> we have tried a few different options, starting from the experimental
>> REST API, S3 bucket syncs and these seem to not be the ideal solution,
>> will be looking into git-sync next, but would really appreciate some
>> suggestions on the best options, particularly if someone has already
>> done some external integration similar to this.
>>
>> Feel free to create issues for discussion and or more details
>>
>> https://github.com/elyra-ai/elyra/issues
>>
>> Or use this thread for suggestions
>>
>> [1]
>> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: A Visual Editor for Airflow pipelines
Posted by Jarek Potiuk <Ja...@polidea.com>.
Agree - maybe 2.1 or 2.2 :).
After some experiences with big customers deployments, I personally think
GitSync at least for now is the best approach out there. It requires git
repo + authorization, but this has all the added benefits of code change
tracking, it is a very standard interface, most of the git repos provide
some ways of manual review if needed and most have some kind of integration
with CI/automated code analysis.
I personally think it should be the default, for any serious deployment as
it provides so many benefits with very limited extra. You just need an
extra "box" - git repo (which is pretty much a given in any organization).
It uses a standard interface that is highly customizable (branches/folder
structures, whatnot) and we already have git-sync container support in the
helm chart.
J.
On Tue, Nov 3, 2020 at 11:27 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Wishfull thinking at the moment Gerard -- the task execution still needs
> files on disk to run the tasks.
>
> This was always in my long term plan for DAG serialization, but we aren't
> there yet. And Custom operators makes this a non-straight forward problem
> to solve.
>
> -ash
>
> On Nov 3 2020, at 12:18 am, Gerard Casas Saez
> <gc...@twitter.com.INVALID> wrote:
>
> Would be interested to also know possible ways to do what Luciano
> described. Hopefully w the serialized DAG and the new API we can start just
> pushing the DAG to the DB (wishful thinking)?
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez <http://twitter.com/casassaez>
>
>
> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> Cool!. I also think it's an interesting one:). But it would be great to
> have such integration possible from Elyra :). Let us know what comes out of
> it :).
>
> J.
>
>
> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry...@rywalker.com> wrote:
>
> Hi Luciano -
>
> Elyra looks like an interesting project — we'd love to connect and talk
> through the opportunity.
>
> You can compare your cal to mine and grab a slot here:
> https://calendly.com/ryw/60min — and I'll be sure to get a few of the
> Airflow PMC members to join as well.
>
> -Ry
>
> Ry Walker
> Founder/CTO of Astronomer + Airflow Committer
>
>
> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
> wrote:
>
> Hi All,
>
> As mentioned in the user list [1] we are working on a visual editor
> for pipelines and adding Airflow as one of the supported backends.
>
> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
>
> As you are the Airflow devs, we would invite you to help us implement
> the best integration possible, in two steps:
>
> 1) Getting a solid integration for building and running pipelines with
> python scripts and jupyter notebooks
>
> 2) Expand the available list component types and enable more generic
> operators
>
> One of the questions raised in the original e-mail is related to how
> to best submit the pipeline dag to be executed by the Airflow runtime,
> we have tried a few different options, starting from the experimental
> REST API, S3 bucket syncs and these seem to not be the ideal solution,
> will be looking into git-sync next, but would really appreciate some
> suggestions on the best options, particularly if someone has already
> done some external integration similar to this.
>
> Feel free to create issues for discussion and or more details
>
> https://github.com/elyra-ai/elyra/issues
>
> Or use this thread for suggestions
>
> [1]
> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: A Visual Editor for Airflow pipelines
Posted by Ash Berlin-Taylor <as...@apache.org>.
Wishfull thinking at the moment Gerard -- the task execution still needs files on disk to run the tasks.
This was always in my long term plan for DAG serialization, but we aren't there yet. And Custom operators makes this a non-straight forward problem to solve.
-ash
On Nov 3 2020, at 12:18 am, Gerard Casas Saez <gc...@twitter.com.INVALID> wrote:
> Would be interested to also know possible ways to do what Luciano described. Hopefully w the serialized DAG and the new API we can start just pushing the DAG to the DB (wishful thinking)?
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez (http://twitter.com/casassaez)
>
>
>
>
>
> On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <Jarek.Potiuk@polidea.com (mailto:Jarek.Potiuk@polidea.com)> wrote:
> > Cool!. I also think it's an interesting one:). But it would be great to have such integration possible from Elyra :). Let us know what comes out of it :).
> >
> > J.
> >
> >
> > On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry@rywalker.com (mailto:ry@rywalker.com)> wrote:
> > > Hi Luciano -
> > >
> > > Elyra looks like an interesting project — we'd love to connect and talk through the opportunity.
> > >
> > > You can compare your cal to mine and grab a slot here: https://calendly.com/ryw/60min — and I'll be sure to get a few of the Airflow PMC members to join as well.
> > >
> > > -Ry
> > >
> > > Ry Walker
> > > Founder/CTO of Astronomer + Airflow Committer
> > >
> > >
> > > On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <luckbr1975@gmail.com (mailto:luckbr1975@gmail.com)> wrote:
> > > > Hi All,
> > > >
> > > > As mentioned in the user list [1] we are working on a visual editor
> > > > for pipelines and adding Airflow as one of the supported backends.
> > > >
> > > > https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
> > > > As you are the Airflow devs, we would invite you to help us implement
> > > > the best integration possible, in two steps:
> > > >
> > > > 1) Getting a solid integration for building and running pipelines with
> > > > python scripts and jupyter notebooks
> > > >
> > > > 2) Expand the available list component types and enable more generic operators
> > > > One of the questions raised in the original e-mail is related to how
> > > > to best submit the pipeline dag to be executed by the Airflow runtime,
> > > > we have tried a few different options, starting from the experimental
> > > > REST API, S3 bucket syncs and these seem to not be the ideal solution,
> > > > will be looking into git-sync next, but would really appreciate some
> > > > suggestions on the best options, particularly if someone has already
> > > > done some external integration similar to this.
> > > >
> > > > Feel free to create issues for discussion and or more details
> > > > https://github.com/elyra-ai/elyra/issues
> > > > Or use this thread for suggestions
> > > > [1] https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
> > > >
> > > > --
> > > > Luciano Resende
> > > > http://twitter.com/lresende1975
> > > > http://lresende.blogspot.com/
> > >
> > >
> > >
> >
> >
> >
> >
> > --
> >
> >
> > Jarek Potiuk
> > Polidea (https://www.polidea.com/) | Principal Software Engineer
> >
> >
> >
> >
> >
> >
> >
> > M: +48 660 796 129 (tel:+48660796129)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
Re: A Visual Editor for Airflow pipelines
Posted by Gerard Casas Saez <gc...@twitter.com.INVALID>.
Would be interested to also know possible ways to do what Luciano
described. Hopefully w the serialized DAG and the new API we can start just
pushing the DAG to the DB (wishful thinking)?
Gerard Casas Saez
Twitter | Cortex | @casassaez <http://twitter.com/casassaez>
On Mon, Nov 2, 2020 at 2:06 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Cool!. I also think it's an interesting one:). But it would be great to
> have such integration possible from Elyra :). Let us know what comes out of
> it :).
>
> J.
>
>
> On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry...@rywalker.com> wrote:
>
>> Hi Luciano -
>>
>> Elyra looks like an interesting project — we'd love to connect and talk
>> through the opportunity.
>>
>> You can compare your cal to mine and grab a slot here:
>> https://calendly.com/ryw/60min — and I'll be sure to get a few of the
>> Airflow PMC members to join as well.
>>
>> -Ry
>>
>> Ry Walker
>> Founder/CTO of Astronomer + Airflow Committer
>>
>>
>> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> As mentioned in the user list [1] we are working on a visual editor
>>> for pipelines and adding Airflow as one of the supported backends.
>>>
>>> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
>>>
>>> As you are the Airflow devs, we would invite you to help us implement
>>> the best integration possible, in two steps:
>>>
>>> 1) Getting a solid integration for building and running pipelines with
>>> python scripts and jupyter notebooks
>>>
>>> 2) Expand the available list component types and enable more generic
>>> operators
>>>
>>> One of the questions raised in the original e-mail is related to how
>>> to best submit the pipeline dag to be executed by the Airflow runtime,
>>> we have tried a few different options, starting from the experimental
>>> REST API, S3 bucket syncs and these seem to not be the ideal solution,
>>> will be looking into git-sync next, but would really appreciate some
>>> suggestions on the best options, particularly if someone has already
>>> done some external integration similar to this.
>>>
>>> Feel free to create issues for discussion and or more details
>>>
>>> https://github.com/elyra-ai/elyra/issues
>>>
>>> Or use this thread for suggestions
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
>>>
>>>
>>> --
>>> Luciano Resende
>>> http://twitter.com/lresende1975
>>> http://lresende.blogspot.com/
>>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: A Visual Editor for Airflow pipelines
Posted by Jarek Potiuk <Ja...@polidea.com>.
Cool!. I also think it's an interesting one:). But it would be great to
have such integration possible from Elyra :). Let us know what comes out of
it :).
J.
On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <ry...@rywalker.com> wrote:
> Hi Luciano -
>
> Elyra looks like an interesting project — we'd love to connect and talk
> through the opportunity.
>
> You can compare your cal to mine and grab a slot here:
> https://calendly.com/ryw/60min — and I'll be sure to get a few of the
> Airflow PMC members to join as well.
>
> -Ry
>
> Ry Walker
> Founder/CTO of Astronomer + Airflow Committer
>
>
> On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> As mentioned in the user list [1] we are working on a visual editor
>> for pipelines and adding Airflow as one of the supported backends.
>>
>> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
>>
>> As you are the Airflow devs, we would invite you to help us implement
>> the best integration possible, in two steps:
>>
>> 1) Getting a solid integration for building and running pipelines with
>> python scripts and jupyter notebooks
>>
>> 2) Expand the available list component types and enable more generic
>> operators
>>
>> One of the questions raised in the original e-mail is related to how
>> to best submit the pipeline dag to be executed by the Airflow runtime,
>> we have tried a few different options, starting from the experimental
>> REST API, S3 bucket syncs and these seem to not be the ideal solution,
>> will be looking into git-sync next, but would really appreciate some
>> suggestions on the best options, particularly if someone has already
>> done some external integration similar to this.
>>
>> Feel free to create issues for discussion and or more details
>>
>> https://github.com/elyra-ai/elyra/issues
>>
>> Or use this thread for suggestions
>>
>> [1]
>> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: A Visual Editor for Airflow pipelines
Posted by Ry Walker <ry...@rywalker.com>.
Hi Luciano -
Elyra looks like an interesting project — we'd love to connect and talk
through the opportunity.
You can compare your cal to mine and grab a slot here:
https://calendly.com/ryw/60min — and I'll be sure to get a few of the
Airflow PMC members to join as well.
-Ry
Ry Walker
Founder/CTO of Astronomer + Airflow Committer
On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <lu...@gmail.com>
wrote:
> Hi All,
>
> As mentioned in the user list [1] we are working on a visual editor
> for pipelines and adding Airflow as one of the supported backends.
>
> https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html
>
> As you are the Airflow devs, we would invite you to help us implement
> the best integration possible, in two steps:
>
> 1) Getting a solid integration for building and running pipelines with
> python scripts and jupyter notebooks
>
> 2) Expand the available list component types and enable more generic
> operators
>
> One of the questions raised in the original e-mail is related to how
> to best submit the pipeline dag to be executed by the Airflow runtime,
> we have tried a few different options, starting from the experimental
> REST API, S3 bucket syncs and these seem to not be the ideal solution,
> will be looking into git-sync next, but would really appreciate some
> suggestions on the best options, particularly if someone has already
> done some external integration similar to this.
>
> Feel free to create issues for discussion and or more details
>
> https://github.com/elyra-ai/elyra/issues
>
> Or use this thread for suggestions
>
> [1]
> https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>