You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2019/11/11 09:54:21 UTC
Re: Donating code to add Common Workflow Language import to Airflow
Hello Andrey,
I think both myself and Maxime - we asked some important questions. If you
want to proceed with the donation, I think it would be great if you let us
know what do you think about the issues we mentioned. I know also Michael
whom I met at the workshops in Berlin - was very interested in this - so
maybe you can take part in the discussion. If you are willing to donate
the code and continue the discussion on it , I think we have to start well
... discussing :).
I just copied our point below to make it easier to answer both of us at the
same time:
Jarek:
1) is the CWL package more of a converter of CWL to Python DAG files (that
can then be scheduled as usual) or whether it is running alongside of the
scheduler and schedules tasks and operators separately using different
scheduling engine?. As a reference there is an
https://github.com/GoogleCloudPlatform/oozie-to-airflow converter from
Oozie XML to airflow DAGs. I think the biggest advantage of Airflow is
being able to modify and iterate quickly using python code so having
aPython Dag generated from CWL might be a good idea - even if it is not
perfect, user can still modify it and extend later manually rather than
relaying that all the features of CWL are implemented.
2 I'd also like to understand what dependencies it introduces on Airflow -
whether it relies on certain internals of Airflow that could make Airflow's
evolution more difficult? Also we have a roadmap for Airflow 2.0 already
and there are certain incompatibilities implemented, more is planned
already (and more to come not planned yet). Is the CWL importer 1.10
compatible or both 1.10 and (current state of) 2.0? Have you been
following some of the discussions with 2.0 and are you aware of some
potential incompatibilities?
3) What are the benefits you see to have Airflow CWL package managed by
the Airflow community rather than CWL one? It could work both ways - it
could be managed by either of the communities (as usual in case of such
imports), but I think it has to be carefully weighted who maintains it
eventually - it all depends on how much one could rely on other, what is
the release cycle of CWL new versions vs. Airflow versions etc. Could you
share your thought process and why you think it should be part of Airflow ?
Maxime:
4) Personally I like the idea of an ecosystem of packages (and repos)
managed
and maintained by their specialist. That way they can have their own CI,
their own release processes and cycles, and "namespaced" notifications. If
anything I'd rather push in the direction of breaking Airflow into many
smaller packages (core, scheduler, web, ...) as opposed to tacking other
projects on top of it.
5) Also arguably Airflow's DSL may be more "common" than CWL. Clearly CWL
has
more focussed intentions around creating something universal, but to me
that doesn't necessarily make it more legitimate or common than other specs
(Oozie, Azkaban , Informatica, ...) and should be treated similarly (would
we want to include extensions to all these as part of Airflow?).
6) I also prefer the codegen/migration approach (I think the
`oozie-to-airflow` tool does that) to allow a path that resolves the common
denominator lmitations. How can this tooling expose features that are
proper to Airflow (pools, priority weights, xcoms, callbacks!, ...)?
J.
On Thu, Oct 31, 2019 at 1:57 AM Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> As someone who has spent a lot of time acting as a maintainer, a code
> "donation" seems like dangerous gift to accept.
>
> Personally I like the idea of an ecosystem of packages (and repos) managed
> and maintained by their specialist. That way they can have their own CI,
> their own release processes and cycles, and "namespaced" notifications. If
> anything I'd rather push in the direction of breaking Airflow into many
> smaller packages (core, scheduler, web, ...) as opposed to tacking other
> projects on top of it.
>
> Also arguably Airflow's DSL may be more "common" than CWL. Clearly CWL has
> more focussed intentions around creating something universal, but to me
> that doesn't necessarily make it more legitimate or common than other specs
> (Oozie, Azkaban , Informatica, ...) and should be treated similarly (would
> we want to include extensions to all these as part of Airflow?).
>
> I also prefer the codegen/migration approach (I think the
> `oozie-to-airflow` tool does that) to allow a path that resolves the common
> denominator lmitations. How can this tooling expose features that are
> proper to Airflow (pools, priority weights, xcoms, callbacks!, ...)?
>
> Max
>
> On Wed, Oct 30, 2019 at 12:32 PM Andrey Kartashov <po...@porter.st>
> wrote:
>
> > My name is Andrey and I'm developer behind CWL-Airflow.
> > This message is follow up slack conversation. I copy past some messages
> > from there here.
> >
> >
> > >> Slack chat:
> >
> > When I've met CWL team there were no pipeline managers to support it.
> I've
> > picked up Airflow to just prove the concept that it is possible.
> >
> > The same time I was looking for a pipeline manager to use for
> > bioinformatic analysis and asked tons of questions from Airflow team as a
> > result special note in documentation: "Beyond the Horizon".
> Nevertheless, I
> > adopted Airflow for our bioinformatic use
> >
> > There are more than 200 different pipeline managers, and to believe that
> > in nearest future there will the only one and perfect one sounds
> > impossible. So, to exchange pipeline logic between different pipeline
> > managers and people it is good to have a standard (CWL is a a perfect
> fit)
> > like JavaScript standard and different executers, browsers...
> >
> > Apache taverna (pipeline manager) is working on adopting CWL for a while
> > now, we have code it is already working.
> >
> > So yes, CWL-Airflow is developed and the use is simple it extends Airflow
> > DAG class. However it is still required to put .py file with DAG (CWLDAG
> in
> > our case) to the dag directory. I would like just to put .cwl file into
> DAG
> > directory to simplify the usage
> >
> > I'm ready to develop what is necessary, but I'm not quite sure (I'm not a
> > big expert in airflow code) which way to go, plugin or some native core
> > code, or ...
> >
> > The project by itself lives https://github.com/Barski-lab/cwl-airflow,
> > there are tons of CWL tests
> > https://ci.commonwl.org/job/airflow-conformance/
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Andrey Kartashov <po...@porter.st>.
On 2019/11/13 23:28:22, Jarek Potiuk <Ja...@polidea.com> wrote:
> On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov <po...@porter.st> wrote:
>
> > Previous email gives more info why I believe CWL should be part of
> > Airflow. Or should I elaborate more?
> >
>
> Well yes. I still miss answers to the basic questions (we keep on asking
> them with Maxime):
>
> 1) We still need to understand more why do you insist so strongly on the
> code being taken over (or written) by Airflow team rather than maintained
> by CWL team?
>
I probably could not give you the whole answer. However, at least my understanding, which might be wrong :). CWL was invented and developed by teams like airflow team. They have their own pipeline managers. CWL team’s idea was not to write CWL for every existing pipeline manager (in java, go, c/c++, python, other languages). The idea was to create a standard simple and complete enough to satisfy 90% of existing pipelines. I by myself and Michael Kotliar are bioinformaticians in a biology wet lab, main task is data analysis, statistical methods etc. We most likely would support CWL-Airflow till our last breath, but we are not “immortal”, and users who would like to use Airflow with CWL feel that.
CWL - has a strong community and followers
Airflow - has an excellent community, followers and fast growing
CWL-Airflow - two free time developers?
> 2) What's wrong with the current model where you provide converters to
> Airflow and maintain it as separate packages.
>
I like current model and we use it in production. We install our solution with sequential executer on laptops/desktops, with local on a server, celery on (workers) cluster. For me with good experience in programming, software architecture, admin skills it is not a big deal to install Airflow, CWL-Airflow then add a dag into dag directory, wrap CWL into CWLDAG put cwl-wrapper python into the same dag directory. For a novice student/PhD/professor :) with biology background it would be difficult, a more natural way to put a CWL file without wrappers into dag directory.
> 3) As we wrote before - we are happy to promote CWL as one of the "sources"
> of workflow imports through our website and make easy to install
> cwl-airflow via [cwl] extra. What do you want to achieve by having code in
> Airflow repo/donated to Apache community?
>
For me it is more like a heritage, the kid will finally join the family.
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
On Wed, Nov 13, 2019 at 11:33 PM Andrey Kartashov <po...@porter.st> wrote:
> Previous email gives more info why I believe CWL should be part of
> Airflow. Or should I elaborate more?
>
Well yes. I still miss answers to the basic questions (we keep on asking
them with Maxime):
1) We still need to understand more why do you insist so strongly on the
code being taken over (or written) by Airflow team rather than maintained
by CWL team?
2) What's wrong with the current model where you provide converters to
Airflow and maintain it as separate packages.
3) As we wrote before - we are happy to promote CWL as one of the "sources"
of workflow imports through our website and make easy to install
cwl-airflow via [cwl] extra. What do you want to achieve by having code in
Airflow repo/donated to Apache community?
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Andrey Kartashov <po...@porter.st>.
Previous email gives more info why I believe CWL should be part of Airflow. Or should I elaborate more?
On 2019/11/13 21:09:52, Maxime Beauchemin <ma...@gmail.com> wrote:
> The big question is why can't it just be on its own Github repository and
> in its own PyPI package? Why does it have to be packaged with our PyPI
> package or live in the Airflow repo?
>
> Max
>
> On Wed, Nov 13, 2019 at 12:42 PM Andrey Kartashov <po...@porter.st> wrote:
>
> > I don't quite get what this example should to prove? I agree XML is too
> > much for everything but CWL is not XML. And our point is not to bring a new
> > pipeline manager we don't have any other but Airflow. We do not do
> > conversion because we have to change CWL file to send them to others
> > Universities/Hospitals/Companies to reproduce result and get improved CWL
> > back to run it on our lovely Airflow. If we convert CWL to python and
> > change it to python we later would have to convert it to CWL back. Our
> > CWLDAG is just an Airflow's DAG and can be used as usual within any
> > Airflow's pipelines
> > with task.set_upstream or task.set_downstream to integrate CWL pipeline
> > into bigger Airflow pipeline.
> >
> > >
> > > Speaking of oozie-2-airflow - as one of the creators of it. Here is the
> > > repo: https://github.com/GoogleCloudPlatform/oozie-to-airflow
> > >
> > > And some basic high-level concepts for O2A:
> > >
> > > - The world becomes a better place - one XML less at a time. This was
> > > our motto. From what we know - Ooozie XML workflows are pretty much
> > > universally hated (well maybe just disliked) in the data processing
> > world.
> > > But there are people who have 1000s of oozie workflows, so we wanted
> > to
> > > give people a tool to easily converts many XMLs to Airflow DAGs.
> > > - It's one-way conversion. The aim is to get Oozie XML workflows as
> > > source, convert it to DAGs and continue working on the generated DAGs.
> > > - We accept are not perfect. We know we can convert many Ooozie
> > > Worfkflows and they will work but we have a number of limitations (
> > >
> > https://github.com/GoogleCloudPlatform/oozie-to-airflow#common-known-limitations
> > ).
> > > We have also a list of issues:
> > > https://github.com/GoogleCloudPlatform/oozie-to-airflow/issues that
> > are
> > > still open and need solving for some more sophisticated features of
> > Ooozie.
> > > But this is OK. We accept that we are not perfect - the DAGs
> > generated by
> > > O2A can be further manually modified and evolved as Python DAGs to add
> > > missing features.
> > > - The Python code generated by O2A are generated with the focus of
> > > readability. Those python files are well written, formatted and
> > structured
> > > in a way that resembles as if a human programmer wrote it so that it
> > is
> > > easy to take them forward and modify them manually. For example we
> > can
> > > then add all the Airflow-specific features Maxime mentioned - queues,
> > xcom,
> > > callbacks, etc. etc. ). So we do not have to be perfect.
> >
> >
>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Maxime Beauchemin <ma...@gmail.com>.
The big question is why can't it just be on its own Github repository and
in its own PyPI package? Why does it have to be packaged with our PyPI
package or live in the Airflow repo?
Max
On Wed, Nov 13, 2019 at 12:42 PM Andrey Kartashov <po...@porter.st> wrote:
> I don't quite get what this example should to prove? I agree XML is too
> much for everything but CWL is not XML. And our point is not to bring a new
> pipeline manager we don't have any other but Airflow. We do not do
> conversion because we have to change CWL file to send them to others
> Universities/Hospitals/Companies to reproduce result and get improved CWL
> back to run it on our lovely Airflow. If we convert CWL to python and
> change it to python we later would have to convert it to CWL back. Our
> CWLDAG is just an Airflow's DAG and can be used as usual within any
> Airflow's pipelines
> with task.set_upstream or task.set_downstream to integrate CWL pipeline
> into bigger Airflow pipeline.
>
> >
> > Speaking of oozie-2-airflow - as one of the creators of it. Here is the
> > repo: https://github.com/GoogleCloudPlatform/oozie-to-airflow
> >
> > And some basic high-level concepts for O2A:
> >
> > - The world becomes a better place - one XML less at a time. This was
> > our motto. From what we know - Ooozie XML workflows are pretty much
> > universally hated (well maybe just disliked) in the data processing
> world.
> > But there are people who have 1000s of oozie workflows, so we wanted
> to
> > give people a tool to easily converts many XMLs to Airflow DAGs.
> > - It's one-way conversion. The aim is to get Oozie XML workflows as
> > source, convert it to DAGs and continue working on the generated DAGs.
> > - We accept are not perfect. We know we can convert many Ooozie
> > Worfkflows and they will work but we have a number of limitations (
> >
> https://github.com/GoogleCloudPlatform/oozie-to-airflow#common-known-limitations
> ).
> > We have also a list of issues:
> > https://github.com/GoogleCloudPlatform/oozie-to-airflow/issues that
> are
> > still open and need solving for some more sophisticated features of
> Ooozie.
> > But this is OK. We accept that we are not perfect - the DAGs
> generated by
> > O2A can be further manually modified and evolved as Python DAGs to add
> > missing features.
> > - The Python code generated by O2A are generated with the focus of
> > readability. Those python files are well written, formatted and
> structured
> > in a way that resembles as if a human programmer wrote it so that it
> is
> > easy to take them forward and modify them manually. For example we
> can
> > then add all the Airflow-specific features Maxime mentioned - queues,
> xcom,
> > callbacks, etc. etc. ). So we do not have to be perfect.
>
>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
On Thu, Nov 14, 2019 at 2:58 AM Andrey Kartashov <po...@porter.st> wrote:
> I'm sorry I have not used quotes. But now, I do not understand at all what
> we are discussing. The email system mixed time and order. Can we all have a
> phone/skype/hangouts/jeet.si call where we can quickly reduce the problem
> and start a new thread? To answer 10 questions from one email and then kind
> of repeat the same answers because of similar question in other email?
>
The thing here is that I am just one of the committers/PMCs - so I cannot
even make decisions - i can just present my opinions. It's the community
that has to decide and you have to convince community to your case not me
:).
Devlist is the place where others can see such discussion and chime in as
needed. Unless someone else from PMC would like to join the discussion on
the phone, there is little I can do for donation purpose. I think it's best
we discuss all the details here and I would like other PMC especially to
express their opinions here. I think we are mixing time and order but I
have a feeling we are slowly getting to the real use case you want to solve
and why you want Airflow to take over the code.
>
> > J.
> >
> >
> > > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Andrey Kartashov <po...@porter.st>.
I'm sorry I have not used quotes. But now, I do not understand at all what we are discussing. The email system mixed time and order. Can we all have a phone/skype/hangouts/jeet.si call where we can quickly reduce the problem and start a new thread? To answer 10 questions from one email and then kind of repeat the same answers because of similar question in other email?
>
> That's fine as long as it is one of the ways to get the DAG, but primary
> use for Airflow is to use DAGs. If you can produce Airflow DAGs from CWL
> and you do not expect that Airflow DAGs are converted back to CWL (which
> might be impossible) that's all fine and nice way to execute CWL dags via
> Airflow. I think we have no argument here.
>
> J.
>
>
> > Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Andrey,
I think you should read the "quotes" in the email. For devlists it's
generally good practice to quote the parts of emails you refer to and
provide your answer (most mailers support it). I try to do it every time I
respond..
On Wed, Nov 13, 2019 at 9:42 PM Andrey Kartashov <po...@porter.st> wrote:
> I don't quite get what this example should to prove? I agree XML is too
> much for everything but CWL is not XML. And our point is not to bring a new
> pipeline manager we don't have any other but Airflow.
The XML comment was about Oozie-2-Airflow to explain the principles and
assumptions we made (that was a question asked by Misha " I need to get
more information about oozie-to-airflow approach.". I provided it so that
you get all the context and understand how we've done that. I do not say
you have to follow the same patterns - I just explained it for comparison.
Oozie has i's own specification of DAG
<https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html>
(it is also an executor) and oozie-2-airflow is really converting the
specification of Oozie to specification of Airflow (it does not replicate
the pipline manager - just converts Oozie specification (XML) to Airflow
one (Python).
> We do not do conversion because we have to change CWL file to send them to
> others Universities/Hospitals/Companies to reproduce result and get
> improved CWL back to run it on our lovely Airflow. If we convert CWL to
> python and change it to python we later would have to convert it to CWL
> back. Our CWLDAG is just an Airflow's DAG and can be used as usual within
> any Airflow's pipelines
> with task.set_upstream or task.set_downstream to integrate CWL pipeline
> into bigger Airflow pipeline.
That's fine as long as it is one of the ways to get the DAG, but primary
use for Airflow is to use DAGs. If you can produce Airflow DAGs from CWL
and you do not expect that Airflow DAGs are converted back to CWL (which
might be impossible) that's all fine and nice way to execute CWL dags via
Airflow. I think we have no argument here.
J.
> Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Andrey Kartashov <po...@porter.st>.
I don't quite get what this example should to prove? I agree XML is too much for everything but CWL is not XML. And our point is not to bring a new pipeline manager we don't have any other but Airflow. We do not do conversion because we have to change CWL file to send them to others Universities/Hospitals/Companies to reproduce result and get improved CWL back to run it on our lovely Airflow. If we convert CWL to python and change it to python we later would have to convert it to CWL back. Our CWLDAG is just an Airflow's DAG and can be used as usual within any Airflow's pipelines
with task.set_upstream or task.set_downstream to integrate CWL pipeline into bigger Airflow pipeline.
>
> Speaking of oozie-2-airflow - as one of the creators of it. Here is the
> repo: https://github.com/GoogleCloudPlatform/oozie-to-airflow
>
> And some basic high-level concepts for O2A:
>
> - The world becomes a better place - one XML less at a time. This was
> our motto. From what we know - Ooozie XML workflows are pretty much
> universally hated (well maybe just disliked) in the data processing world.
> But there are people who have 1000s of oozie workflows, so we wanted to
> give people a tool to easily converts many XMLs to Airflow DAGs.
> - It's one-way conversion. The aim is to get Oozie XML workflows as
> source, convert it to DAGs and continue working on the generated DAGs.
> - We accept are not perfect. We know we can convert many Ooozie
> Worfkflows and they will work but we have a number of limitations (
> https://github.com/GoogleCloudPlatform/oozie-to-airflow#common-known-limitations).
> We have also a list of issues:
> https://github.com/GoogleCloudPlatform/oozie-to-airflow/issues that are
> still open and need solving for some more sophisticated features of Ooozie.
> But this is OK. We accept that we are not perfect - the DAGs generated by
> O2A can be further manually modified and evolved as Python DAGs to add
> missing features.
> - The Python code generated by O2A are generated with the focus of
> readability. Those python files are well written, formatted and structured
> in a way that resembles as if a human programmer wrote it so that it is
> easy to take them forward and modify them manually. For example we can
> then add all the Airflow-specific features Maxime mentioned - queues, xcom,
> callbacks, etc. etc. ). So we do not have to be perfect.
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
>
> I hope ideas in that treads and questions do not interfere with current
> discussions. I have not mentioned anything about `cwl descriptor + job`
> (Michael K probably did :(), but in current implementation CWL Description
> is static and lives long live :) and then one can attach a sensor to the
> top of CWLDAG to check files in a directory or as we do we trigger that
> dag `airflow trigger_dag --conf "{\"job\":$(cat ./hg19.job)}"
> "bowtie-index"` so we starting it with kind of xcom data filled in.
>
I looked at the
https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside
to
understand what CWL is and that's where I took the descriptor + job (in Key
Concepts).
> I never thought about rewriting all the python DAGs code :) into CWL. I
> did say that there are people who decided to use CWL and they already
> limited to environment they have like toil, IBM HPC and others who picked
> up Airflow for them CWL is the solution.
OK. So as I understand finally the problem you want to solve - "To make
Airflow more accessible to people who already use CWL or who will find it
easier to write dags in CWL". I still think this does not necessarily have
to be solved by donating CWL code to Airflow (see below).
I do not believe that it is possible to convert anybody (so I separate
> existing users and new users), but I strongly believe that there is a huge
> biology, chemistry, astronomers, physics, etc community which would
> appreciate CWL in Airflow
>
So possibly the solution is that we make it easy to configure Airflow with
the CWL installed - but not necessarily donating code to Airflow. We have
various ways of installing Airflow and we are just now working on a new
Airflow Website, landing page that describes various ways of installing
Airflow. We are also going to have an official helm chart updated with
production image of Airflow and it already has various ways of installation
(Celery/Kubernetes executor etc). What I see can be done there is to have a
dedicated section on how to install Airflow locally with CWL enabled (you
could provide a description on how to do it) and have an option in helm
chart to support installation with cwl-airflow installed as separate POD in
kubernetes. This way you achieve the goal of making Airflow available for
CWL users without donating CWL code. I am happy to help with this - as I am
involved with both tasks.
> > What's the problem you want to solve by
> > donating the code to Airflow team ?
> >
>
> Because I do it in free time, and at some point I just might have no free
> time.
>
Ok. So what you basically say is that you think Airflow community has more
capacity than CWL community to maintain CWL converter. I am not so sure
about it (precisely because of the lost opportunities). But maybe a better
solution is to ask in the airflow community whether there are people who
could join the CWL-airflow converter and increase the community there.
I would not say for the whole community, but I would not feel comfortable
as a community to take responsibility on the converter without prior
knowledge and understanding CWL in detail. Especially that it is rather for
small group of users (at least initially). But I find CWL as an idea very
interesting and maybe there are some people in the community who would love
to contribute to your project? Suggestion - maybe just ask - here and in
slack - if there is enough interest in contributing to CWL-Airflow, rather
than donating the code to Airflow ? Just promote your project in the
community and ask for help.
I can see this as the best of both worlds - if you find a few people who
would like to help and get familiar with it and they are also part of the
Airflow community and we get collective knowledge about it - then
eventually it might lead to incorporating it to Airflow itself if our
community gets more familiar with CWL. I think this is the best way to
achieve the final goal of finally incorporating CWL as part of Airflow.
In the meantime - I am happy to help to make Airflow more "CWL friendly"
for the users - both from documentation and Helm chart POV.
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Andrey Kartashov <po...@porter.st>.
Hi J,
You always start with hello Andrey (so no I feel uncomfortable :)
> I'd like to start by saying I'm a big big fan of Airflow, otherwise there
> > wouldn’t be CWL-Airflow :). I like everything about Airflow development.
> > Especially the way of adding extra packages to Airflow like Kubernetes for
> > example with just pip install 'apache-airflow[kubernetes]'. I believe it
> > would be nice to have pip install 'apache-airflow[cwl]' too.
>
>
> This can be easily done as Maxime said - have a separate cwl-airflow
> package in pypi and have it configured as dependency in 'cwl' extra. I see
> no problem with that. I assume you have a perfect converter that always
> produces good and schedulable .dag file. I imagine the users will run (on
> demand) the converter to convert the cwl descriptor + job to a RESULT.py in
> the dag folder and they otherwise run a pretty standard airflow to process
> it. Do you imagine a more "involved" integration with Airlfow? If so - how
> do you imagine the use case? Could you explain how do you envision the
> life-cycyle of such workflow?
>
I found some old threads on google groups.
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/airbnb_airflow/_muNQ7jTAkQ/qeWCaLHiAAAJ
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/airbnb_airflow/nk2GAciOADI/gW1pWTwRDwAJ
I hope ideas in that treads and questions do not interfere with current discussions. I have not mentioned anything about `cwl descriptor + job` (Michael K probably did :(), but in current implementation CWL Description is static and lives long live :) and then one can attach a sensor to the top of CWLDAG to check files in a directory or as we do we trigger that dag `airflow trigger_dag --conf "{\"job\":$(cat ./hg19.job)}" "bowtie-index"` so we starting it with kind of xcom data filled in.
> >
> > Here I'm citing from http://commonwl.org: "The Common Workflow Language
> > (CWL) is an open standard for describing analysis workflows and tools in a
> > way that makes them portable and scalable across a variety of software and
> > hardware environment". So CWL is a standard and it is more about how
> > describe a workflow in a way that everybody will understand and be able to
> > reproduce. It might be not yet widely implemented and lacks some features,
> > but it’s definitely not about favorite programming language or pipeline
> > manager. Of course, this CWL specification brings some limitations and they
> > exist because of CWL attempts to formalize the most common features of any
> > pipeline.
> >
>
> Do you imagine that ALL airflow users starts using CWL as their main
> workflow language? For the reasons described above (Python use/Data
> Scientist approach) I think this is not going to happen for Airflow, I can
> understand that you want to run a CWL workflows using Airflow, but I do not
> see Airflow DAG developers switching to use CWL as their main workflow
> language. And for people who have 100s or 1000s of DAGs there is no easy
> way to convert them to CWL (is there?).
>
I never thought about rewriting all the python DAGs code :) into CWL. I did say that there are people who decided to use CWL and they already limited to environment they have like toil, IBM HPC and others who picked up Airflow for them CWL is the solution.
>
> > We believe that it’s worth moving towards CWL standard based on the
> > growing interest of people and big companies who run scientific pipelines.
> > There are a lot of published scientific papers. Projects such as Toil,
> > Arvados, Galaxy, Taverna and others already have solutions to run CWL
> > pipelines. People are interested in pipelines that are easy to share. Even
> > IBM released CWL HPC executer
> >
>
> I understand there are papers/ I also understand that there is a group of
> people who would like to have easily shareable pipelines. Why do you think
> this is important for Airflow users to have them?
>
I do not believe that it is possible to convert anybody (so I separate existing users and new users), but I strongly believe that there is a huge biology, chemistry, astronomers, physics, etc community which would appreciate CWL in Airflow.
> > CWL-Airflow creates Airflow DAG with all its steps on the fly on every
> > dagbag refresh. I think this behavior can be considered as converter. More
> > details can be found here
> > https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside
> >
>
> It's not a question of time, but rather the question of focus (and missed
> opportunities) - we already have a full roadmap for 2.0 and beyond and we
> are all working hard on it. Since you have already working solution - why
> do you not want to maintain it? What's the problem you want to solve by
> donating the code to Airflow team ?
>
Because I do it in free time, and at some point I just might have no free time.
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hello Andrey,
I have nothing about emotions, I actually think emotions are super
important and drive our passion to do stuff, but I'd love if you can answer
the questions we ask by quoting and answering them.
I'd like to start by saying I'm a big big fan of Airflow, otherwise there
> wouldn’t be CWL-Airflow :). I like everything about Airflow development.
> Especially the way of adding extra packages to Airflow like Kubernetes for
> example with just pip install 'apache-airflow[kubernetes]'. I believe it
> would be nice to have pip install 'apache-airflow[cwl]' too.
This can be easily done as Maxime said - have a separate cwl-airflow
package in pypi and have it configured as dependency in 'cwl' extra. I see
no problem with that. I assume you have a perfect converter that always
produces good and schedulable .dag file. I imagine the users will run (on
demand) the converter to convert the cwl descriptor + job to a RESULT.py in
the dag folder and they otherwise run a pretty standard airflow to process
it. Do you imagine a more "involved" integration with Airlfow? If so - how
do you imagine the use case? Could you explain how do you envision the
life-cycyle of such workflow?
>
> Here I'm citing from http://commonwl.org: "The Common Workflow Language
> (CWL) is an open standard for describing analysis workflows and tools in a
> way that makes them portable and scalable across a variety of software and
> hardware environment". So CWL is a standard and it is more about how
> describe a workflow in a way that everybody will understand and be able to
> reproduce. It might be not yet widely implemented and lacks some features,
> but it’s definitely not about favorite programming language or pipeline
> manager. Of course, this CWL specification brings some limitations and they
> exist because of CWL attempts to formalize the most common features of any
> pipeline.
>
Do you imagine that ALL airflow users starts using CWL as their main
workflow language? For the reasons described above (Python use/Data
Scientist approach) I think this is not going to happen for Airflow, I can
understand that you want to run a CWL workflows using Airflow, but I do not
see Airflow DAG developers switching to use CWL as their main workflow
language. And for people who have 100s or 1000s of DAGs there is no easy
way to convert them to CWL (is there?).
> We believe that it’s worth moving towards CWL standard based on the
> growing interest of people and big companies who run scientific pipelines.
> There are a lot of published scientific papers. Projects such as Toil,
> Arvados, Galaxy, Taverna and others already have solutions to run CWL
> pipelines. People are interested in pipelines that are easy to share. Even
> IBM released CWL HPC executer
>
I understand there are papers/ I also understand that there is a group of
people who would like to have easily shareable pipelines. Why do you think
this is important for Airflow users to have them?
>
> The main idea that I want to express is that CWL is not another pipeline
> manager or executer, it is a specification, a standard. So, we cannot
> compare it with Oozie or Azkaban etc. We agree that there are more than 200
> pipeline managers and no standard/specification.
That's fine. I understand it's a standard. My point is - as long as you
just want to treat airflow as executor of CWL and provide converter to make
a python DAG out of that - that's perfectly fine (for someone who already
uses CWL). Do you need anything more?.
>
>
> That’s why we are trying to show that CWL is not a burden but a solution
> to make at least some pipelines easily shareable and reproducible.
>
Is there anything lacking in the current "converter" that makes it not good
in this job? From what I understand if someone wants to write a shareable
and reproducible workflow, they can write it in CWL and use your converter
and run it via Airflow (or HPC or whatever other pipeline managers). Is
there anything else that you think deeper Airflow integration can help
with here ?
I believe that it will not take too much time for Airflow team to implement
> CWL-reader within Airflow structure, if you don’t like donations. I
> understand that CWL-Airflow depends on cwltool package which in its turn
> depends on other libs like: ruamel.yaml,rdflib, shellescape, schema-salad,
> psutil, scandir, pathlib, …, but it can be easily simplified if necessary.
> CWL-Airflow creates Airflow DAG with all its steps on the fly on every
> dagbag refresh. I think this behavior can be considered as converter. More
> details can be found here
> https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside
>
It's not a question of time, but rather the question of focus (and missed
opportunities) - we already have a full roadmap for 2.0 and beyond and we
are all working hard on it. Since you have already working solution - why
do you not want to maintain it? What's the problem you want to solve by
donating the code to Airflow team ?
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Andrey Kartashov <po...@porter.st>.
I believe there are just two big questions:
- what is CWL/why CWL-Airflow exists/should be CWL or CWL-Airflow be a part of Apache-Airflow
-technical implementation of CWL-Airflow
I'd like to start by saying I'm a big big fan of Airflow, otherwise there wouldn’t be CWL-Airflow :). I like everything about Airflow development. Especially the way of adding extra packages to Airflow like Kubernetes for example with just pip install 'apache-airflow[kubernetes]'. I believe it would be nice to have pip install 'apache-airflow[cwl]' too.
1)
My thoughts below might seem emotional, but they express the idea of CWL in simple words.
Here I'm citing from http://commonwl.org: "The Common Workflow Language (CWL) is an open standard for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environment". So CWL is a standard and it is more about how describe a workflow in a way that everybody will understand and be able to reproduce. It might be not yet widely implemented and lacks some features, but it’s definitely not about favorite programming language or pipeline manager. Of course, this CWL specification brings some limitations and they exist because of CWL attempts to formalize the most common features of any pipeline.
We believe that it’s worth moving towards CWL standard based on the growing interest of people and big companies who run scientific pipelines. There are a lot of published scientific papers. Projects such as Toil, Arvados, Galaxy, Taverna and others already have solutions to run CWL pipelines. People are interested in pipelines that are easy to share. Even IBM released CWL HPC executer
The main idea that I want to express is that CWL is not another pipeline manager or executer, it is a specification, a standard. So, we cannot compare it with Oozie or Azkaban etc. We agree that there are more than 200 pipeline managers and no standard/specification.
That’s why we are trying to show that CWL is not a burden but a solution to make at least some pipelines easily shareable and reproducible.
2)
I believe that it will not take too much time for Airflow team to implement CWL-reader within Airflow structure, if you don’t like donations. I understand that CWL-Airflow depends on cwltool package which in its turn depends on other libs like: ruamel.yaml,rdflib, shellescape, schema-salad, psutil, scandir, pathlib, …, but it can be easily simplified if necessary. CWL-Airflow creates Airflow DAG with all its steps on the fly on every dagbag refresh. I think this behavior can be considered as converter. More details can be found here https://cwl-airflow.readthedocs.io/en/1.0.18/readme/how_it_works.html#what-s-inside
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
I think it would be great to hear from others what they think - but here
are my thoughts:
1) CWL-Airflow package is more of a converter. We created a special class
> CWLDAG (inherited from DAG), which allows us to set the path to the CWL
> file (a separate YAML file). Once parsed by Airflow the newly created DAG
> will have the structure similar to the workflow described in CWL file. The
> actual dag file that you keep in the dags folder is still a python file,
> but it would be nice if we could parse CWL files directly from the dags
> folder without any intermediate python files.
>
I think leaving it as converted to .py files is far better idea from
Airflow POV. Using Python as "source of truth" is the core of Airflow and I
think if we would like something that you describe would require two
things: 1) the conversion will have to be perfect and produce always
perfect 1-1 equivalent DAG (more on that in oozie-2-airflow below and why
O2A intentionally is not perfect converter). 2) It would require some kind
of abstract representation of the workflow.
We are working on some abstract representation of DAG in the DAG
serialization effort, but it is still very much python-driven (it's a
product of running python code) and I think we would not want to be tied to
CWL definition so keeping python code as common source of truth seems
better for us.
> 2) We are trying to keep our package as simple as possible, so it
> shouldn’t cause any dependency issues in the future. We more depend on the
> changes in the CWL standard and its reference implementation (cwltool)
> rather than changes in Airflow. We are actively using XCom to transfer some
> metadata between the tasks, but, as far as I understand, it’s one of the
> “core ideas” of Airflow, so it shouldn’t be dramatically changed in the
> future version.
>
That's why also keeping python code as common "interface" seems like a good
idea. That's the most "stable" part I think of Airflow definition of
workflows because the goal is to keep the minimum number of changes for the
users already having plenty of DAGs. So even if we introduce some
incompatibilities in 2.0, maybe we will even provide some tools to
automatically convert 1.10 dags into 2.0 dags (something I thought about
already but haven't yet discussed in community). The internal/more abstract
representations of DAGs are prone to change (and they will change for sure
in 2.0).
> 3) I would like to see CWL-Airflow be actively used by other people.
> Currently, CWL community is more centered around scientific pipelines,
> whereas Airflow is closer to commercial or business-related workflows. I
> believe that CWL as a standard can be used in both of these areas.
> Additionally, CWL standard is more suitable for data-driven workflow
> management model, which, as far as I understand, Airflow lacks at the
> moment. As for the release cycle of CWL and Airflow, I'm not ready to
> answer it now.
I still do not see what would be the benefits of maintaining CWL converter
by Airflow team :). I think we will always think in "Python" way. That's
one of the main benefits and I think the advantages of Airflow - especially
in a world of rapid prototyping and fast iterating on workflows. People
writing the DAGs (data scientists) do not have to learn
yet-another-standard but they use familiar language - Python - and
ready-to-use building blocks - Operators to define their workflows. I think
the whole community of Airflow is centered around that concept. I do not
think Airflow community would like to swap that paradigm with something
else (I actually love that paradigm). But I might be wrong - of course - so
I will let others chime-in.
> 4) Agree, it makes sense, but then we will discuss how to become a part of
> that ecosystem of packages:)
From what I see now I think what we can do - rather than donating the code
to Airflow- we are currently working on a new "landing page" of Airflow
(Current prototype here: https://airflow-website-s5z26d2t7a-ew.a.run.app/)
so maybe we can simply add a page there to add "integration with other
workflow engines" and have Oozie2Airflow, CWLToAirflow and any future "To
Airflow" converters. I can talk to the team that works on it and I think
it's generally a good idea to have such page.
5) I think every spec has it's pros and cons and it will take ages to
> include all of them. Moreover, it will become a nightmare for those who
> will try to use that system. That's why we started with CWL as the most
> promising one. Of course, it's only from our point of view:) Our main
> intention was to choose that standard that will be the easiest for not
> programmers (in our case biologists) to create new pipelines by themselves.
Yeah. That's I think is the biggest concept difference. Our main group of
users are Data Scientists who actually use Python as their primary way to
talk to computers. I think this is a huge paradigm difference and where
combining CWL and Airflow paradigm is next-to-impossible because we simply
have different foundational assumptions about where the DAGs "source of
truth" is. I think using Python as main workflow language is just great for
those people and I don't think it will go away.
> 6) Unfortunately, I cannot answer this question now. I need to get more
> information about oozie-to-airflow approach.
>
Speaking of oozie-2-airflow - as one of the creators of it. Here is the
repo: https://github.com/GoogleCloudPlatform/oozie-to-airflow
And some basic high-level concepts for O2A:
- The world becomes a better place - one XML less at a time. This was
our motto. From what we know - Ooozie XML workflows are pretty much
universally hated (well maybe just disliked) in the data processing world.
But there are people who have 1000s of oozie workflows, so we wanted to
give people a tool to easily converts many XMLs to Airflow DAGs.
- It's one-way conversion. The aim is to get Oozie XML workflows as
source, convert it to DAGs and continue working on the generated DAGs.
- We accept are not perfect. We know we can convert many Ooozie
Worfkflows and they will work but we have a number of limitations (
https://github.com/GoogleCloudPlatform/oozie-to-airflow#common-known-limitations).
We have also a list of issues:
https://github.com/GoogleCloudPlatform/oozie-to-airflow/issues that are
still open and need solving for some more sophisticated features of Ooozie.
But this is OK. We accept that we are not perfect - the DAGs generated by
O2A can be further manually modified and evolved as Python DAGs to add
missing features.
- The Python code generated by O2A are generated with the focus of
readability. Those python files are well written, formatted and structured
in a way that resembles as if a human programmer wrote it so that it is
easy to take them forward and modify them manually. For example we can
then add all the Airflow-specific features Maxime mentioned - queues, xcom,
callbacks, etc. etc. ). So we do not have to be perfect.
J.
> Best regards,
> Michael
>
> On 2019/11/11 09:54:21, Jarek Potiuk <Ja...@polidea.com> wrote:
> > Hello Andrey,
> >
> > I think both myself and Maxime - we asked some important questions. If
> you
> > want to proceed with the donation, I think it would be great if you let
> us
> > know what do you think about the issues we mentioned. I know also Michael
> > whom I met at the workshops in Berlin - was very interested in this - so
> > maybe you can take part in the discussion. If you are willing to donate
> > the code and continue the discussion on it , I think we have to start
> well
> > ... discussing :).
> >
> > I just copied our point below to make it easier to answer both of us at
> the
> > same time:
> >
> > Jarek:
> >
> > 1) is the CWL package more of a converter of CWL to Python DAG files
> (that
> > can then be scheduled as usual) or whether it is running alongside of the
> > scheduler and schedules tasks and operators separately using different
> > scheduling engine?. As a reference there is an
> > https://github.com/GoogleCloudPlatform/oozie-to-airflow converter from
> > Oozie XML to airflow DAGs. I think the biggest advantage of Airflow is
> > being able to modify and iterate quickly using python code so having
> > aPython Dag generated from CWL might be a good idea - even if it is not
> > perfect, user can still modify it and extend later manually rather than
> > relaying that all the features of CWL are implemented.
> >
> > 2 I'd also like to understand what dependencies it introduces on Airflow
> -
> > whether it relies on certain internals of Airflow that could make
> Airflow's
> > evolution more difficult? Also we have a roadmap for Airflow 2.0 already
> > and there are certain incompatibilities implemented, more is planned
> > already (and more to come not planned yet). Is the CWL importer 1.10
> > compatible or both 1.10 and (current state of) 2.0? Have you been
> > following some of the discussions with 2.0 and are you aware of some
> > potential incompatibilities?
> >
> > 3) What are the benefits you see to have Airflow CWL package managed by
> > the Airflow community rather than CWL one? It could work both ways - it
> > could be managed by either of the communities (as usual in case of such
> > imports), but I think it has to be carefully weighted who maintains it
> > eventually - it all depends on how much one could rely on other, what is
> > the release cycle of CWL new versions vs. Airflow versions etc. Could
> you
> > share your thought process and why you think it should be part of
> Airflow ?
> >
> > Maxime:
> >
> > 4) Personally I like the idea of an ecosystem of packages (and repos)
> > managed
> > and maintained by their specialist. That way they can have their own CI,
> > their own release processes and cycles, and "namespaced" notifications.
> If
> > anything I'd rather push in the direction of breaking Airflow into many
> > smaller packages (core, scheduler, web, ...) as opposed to tacking other
> > projects on top of it.
> >
> > 5) Also arguably Airflow's DSL may be more "common" than CWL. Clearly CWL
> > has
> > more focussed intentions around creating something universal, but to me
> > that doesn't necessarily make it more legitimate or common than other
> specs
> > (Oozie, Azkaban , Informatica, ...) and should be treated similarly
> (would
> > we want to include extensions to all these as part of Airflow?).
> >
> > 6) I also prefer the codegen/migration approach (I think the
> > `oozie-to-airflow` tool does that) to allow a path that resolves the
> common
> > denominator lmitations. How can this tooling expose features that are
> > proper to Airflow (pools, priority weights, xcoms, callbacks!, ...)?
> >
> > J.
> >
> > On Thu, Oct 31, 2019 at 1:57 AM Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> > > As someone who has spent a lot of time acting as a maintainer, a code
> > > "donation" seems like dangerous gift to accept.
> > >
> > > Personally I like the idea of an ecosystem of packages (and repos)
> managed
> > > and maintained by their specialist. That way they can have their own
> CI,
> > > their own release processes and cycles, and "namespaced"
> notifications. If
> > > anything I'd rather push in the direction of breaking Airflow into many
> > > smaller packages (core, scheduler, web, ...) as opposed to tacking
> other
> > > projects on top of it.
> > >
> > > Also arguably Airflow's DSL may be more "common" than CWL. Clearly CWL
> has
> > > more focussed intentions around creating something universal, but to me
> > > that doesn't necessarily make it more legitimate or common than other
> specs
> > > (Oozie, Azkaban , Informatica, ...) and should be treated similarly
> (would
> > > we want to include extensions to all these as part of Airflow?).
> > >
> > > I also prefer the codegen/migration approach (I think the
> > > `oozie-to-airflow` tool does that) to allow a path that resolves the
> common
> > > denominator lmitations. How can this tooling expose features that are
> > > proper to Airflow (pools, priority weights, xcoms, callbacks!, ...)?
> > >
> > > Max
> > >
> > > On Wed, Oct 30, 2019 at 12:32 PM Andrey Kartashov <po...@porter.st>
> > > wrote:
> > >
> > > > My name is Andrey and I'm developer behind CWL-Airflow.
> > > > This message is follow up slack conversation. I copy past some
> messages
> > > > from there here.
> > > >
> > > >
> > > > >> Slack chat:
> > > >
> > > > When I've met CWL team there were no pipeline managers to support it.
> > > I've
> > > > picked up Airflow to just prove the concept that it is possible.
> > > >
> > > > The same time I was looking for a pipeline manager to use for
> > > > bioinformatic analysis and asked tons of questions from Airflow team
> as a
> > > > result special note in documentation: "Beyond the Horizon".
> > > Nevertheless, I
> > > > adopted Airflow for our bioinformatic use
> > > >
> > > > There are more than 200 different pipeline managers, and to believe
> that
> > > > in nearest future there will the only one and perfect one sounds
> > > > impossible. So, to exchange pipeline logic between different pipeline
> > > > managers and people it is good to have a standard (CWL is a a perfect
> > > fit)
> > > > like JavaScript standard and different executers, browsers...
> > > >
> > > > Apache taverna (pipeline manager) is working on adopting CWL for a
> while
> > > > now, we have code it is already working.
> > > >
> > > > So yes, CWL-Airflow is developed and the use is simple it extends
> Airflow
> > > > DAG class. However it is still required to put .py file with DAG
> (CWLDAG
> > > in
> > > > our case) to the dag directory. I would like just to put .cwl file
> into
> > > DAG
> > > > directory to simplify the usage
> > > >
> > > > I'm ready to develop what is necessary, but I'm not quite sure (I'm
> not a
> > > > big expert in airflow code) which way to go, plugin or some native
> core
> > > > code, or ...
> > > >
> > > > The project by itself lives
> https://github.com/Barski-lab/cwl-airflow,
> > > > there are tons of CWL tests
> > > > https://ci.commonwl.org/job/airflow-conformance/
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Donating code to add Common Workflow Language import to Airflow
Posted by Misha Kotliar <mi...@gmail.com>.
Hi everyone,
Sorry for the late response:) I'm Michael. I wrote the post that Michael R. Crusoe mentioned at the very beginning of this conversation. Both Andrey and I work on CWL-Airflow. I will try to answer some of your questions.
1) CWL-Airflow package is more of a converter. We created a special class CWLDAG (inherited from DAG), which allows us to set the path to the CWL file (a separate YAML file). Once parsed by Airflow the newly created DAG will have the structure similar to the workflow described in CWL file. The actual dag file that you keep in the dags folder is still a python file, but it would be nice if we could parse CWL files directly from the dags folder without any intermediate python files.
2) We are trying to keep our package as simple as possible, so it shouldn’t cause any dependency issues in the future. We more depend on the changes in the CWL standard and its reference implementation (cwltool) rather than changes in Airflow. We are actively using XCom to transfer some metadata between the tasks, but, as far as I understand, it’s one of the “core ideas” of Airflow, so it shouldn’t be dramatically changed in the future version.
3) I would like to see CWL-Airflow be actively used by other people. Currently, CWL community is more centered around scientific pipelines, whereas Airflow is closer to commercial or business-related workflows. I believe that CWL as a standard can be used in both of these areas. Additionally, CWL standard is more suitable for data-driven workflow management model, which, as far as I understand, Airflow lacks at the moment. As for the release cycle of CWL and Airflow, I'm not ready to answer it now.
4) Agree, it makes sense, but then we will discuss how to become a part of that ecosystem of packages:)
5) I think every spec has it's pros and cons and it will take ages to include all of them. Moreover, it will become a nightmare for those who will try to use that system. That's why we started with CWL as the most promising one. Of course, it's only from our point of view:) Our main intention was to choose that standard that will be the easiest for not programmers (in our case biologists) to create new pipelines by themselves.
6) Unfortunately, I cannot answer this question now. I need to get more information about oozie-to-airflow approach.
Best regards,
Michael
On 2019/11/11 09:54:21, Jarek Potiuk <Ja...@polidea.com> wrote:
> Hello Andrey,
>
> I think both myself and Maxime - we asked some important questions. If you
> want to proceed with the donation, I think it would be great if you let us
> know what do you think about the issues we mentioned. I know also Michael
> whom I met at the workshops in Berlin - was very interested in this - so
> maybe you can take part in the discussion. If you are willing to donate
> the code and continue the discussion on it , I think we have to start well
> ... discussing :).
>
> I just copied our point below to make it easier to answer both of us at the
> same time:
>
> Jarek:
>
> 1) is the CWL package more of a converter of CWL to Python DAG files (that
> can then be scheduled as usual) or whether it is running alongside of the
> scheduler and schedules tasks and operators separately using different
> scheduling engine?. As a reference there is an
> https://github.com/GoogleCloudPlatform/oozie-to-airflow converter from
> Oozie XML to airflow DAGs. I think the biggest advantage of Airflow is
> being able to modify and iterate quickly using python code so having
> aPython Dag generated from CWL might be a good idea - even if it is not
> perfect, user can still modify it and extend later manually rather than
> relaying that all the features of CWL are implemented.
>
> 2 I'd also like to understand what dependencies it introduces on Airflow -
> whether it relies on certain internals of Airflow that could make Airflow's
> evolution more difficult? Also we have a roadmap for Airflow 2.0 already
> and there are certain incompatibilities implemented, more is planned
> already (and more to come not planned yet). Is the CWL importer 1.10
> compatible or both 1.10 and (current state of) 2.0? Have you been
> following some of the discussions with 2.0 and are you aware of some
> potential incompatibilities?
>
> 3) What are the benefits you see to have Airflow CWL package managed by
> the Airflow community rather than CWL one? It could work both ways - it
> could be managed by either of the communities (as usual in case of such
> imports), but I think it has to be carefully weighted who maintains it
> eventually - it all depends on how much one could rely on other, what is
> the release cycle of CWL new versions vs. Airflow versions etc. Could you
> share your thought process and why you think it should be part of Airflow ?
>
> Maxime:
>
> 4) Personally I like the idea of an ecosystem of packages (and repos)
> managed
> and maintained by their specialist. That way they can have their own CI,
> their own release processes and cycles, and "namespaced" notifications. If
> anything I'd rather push in the direction of breaking Airflow into many
> smaller packages (core, scheduler, web, ...) as opposed to tacking other
> projects on top of it.
>
> 5) Also arguably Airflow's DSL may be more "common" than CWL. Clearly CWL
> has
> more focussed intentions around creating something universal, but to me
> that doesn't necessarily make it more legitimate or common than other specs
> (Oozie, Azkaban , Informatica, ...) and should be treated similarly (would
> we want to include extensions to all these as part of Airflow?).
>
> 6) I also prefer the codegen/migration approach (I think the
> `oozie-to-airflow` tool does that) to allow a path that resolves the common
> denominator lmitations. How can this tooling expose features that are
> proper to Airflow (pools, priority weights, xcoms, callbacks!, ...)?
>
> J.
>
> On Thu, Oct 31, 2019 at 1:57 AM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > As someone who has spent a lot of time acting as a maintainer, a code
> > "donation" seems like dangerous gift to accept.
> >
> > Personally I like the idea of an ecosystem of packages (and repos) managed
> > and maintained by their specialist. That way they can have their own CI,
> > their own release processes and cycles, and "namespaced" notifications. If
> > anything I'd rather push in the direction of breaking Airflow into many
> > smaller packages (core, scheduler, web, ...) as opposed to tacking other
> > projects on top of it.
> >
> > Also arguably Airflow's DSL may be more "common" than CWL. Clearly CWL has
> > more focussed intentions around creating something universal, but to me
> > that doesn't necessarily make it more legitimate or common than other specs
> > (Oozie, Azkaban , Informatica, ...) and should be treated similarly (would
> > we want to include extensions to all these as part of Airflow?).
> >
> > I also prefer the codegen/migration approach (I think the
> > `oozie-to-airflow` tool does that) to allow a path that resolves the common
> > denominator lmitations. How can this tooling expose features that are
> > proper to Airflow (pools, priority weights, xcoms, callbacks!, ...)?
> >
> > Max
> >
> > On Wed, Oct 30, 2019 at 12:32 PM Andrey Kartashov <po...@porter.st>
> > wrote:
> >
> > > My name is Andrey and I'm developer behind CWL-Airflow.
> > > This message is follow up slack conversation. I copy past some messages
> > > from there here.
> > >
> > >
> > > >> Slack chat:
> > >
> > > When I've met CWL team there were no pipeline managers to support it.
> > I've
> > > picked up Airflow to just prove the concept that it is possible.
> > >
> > > The same time I was looking for a pipeline manager to use for
> > > bioinformatic analysis and asked tons of questions from Airflow team as a
> > > result special note in documentation: "Beyond the Horizon".
> > Nevertheless, I
> > > adopted Airflow for our bioinformatic use
> > >
> > > There are more than 200 different pipeline managers, and to believe that
> > > in nearest future there will the only one and perfect one sounds
> > > impossible. So, to exchange pipeline logic between different pipeline
> > > managers and people it is good to have a standard (CWL is a a perfect
> > fit)
> > > like JavaScript standard and different executers, browsers...
> > >
> > > Apache taverna (pipeline manager) is working on adopting CWL for a while
> > > now, we have code it is already working.
> > >
> > > So yes, CWL-Airflow is developed and the use is simple it extends Airflow
> > > DAG class. However it is still required to put .py file with DAG (CWLDAG
> > in
> > > our case) to the dag directory. I would like just to put .cwl file into
> > DAG
> > > directory to simplify the usage
> > >
> > > I'm ready to develop what is necessary, but I'm not quite sure (I'm not a
> > > big expert in airflow code) which way to go, plugin or some native core
> > > code, or ...
> > >
> > > The project by itself lives https://github.com/Barski-lab/cwl-airflow,
> > > there are tons of CWL tests
> > > https://ci.commonwl.org/job/airflow-conformance/
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>