You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "philiprbrenan (via GitHub)" <gi...@apache.org> on 2024/02/22 03:01:35 UTC
[PR] Modernized front page scenario [airflow]
philiprbrenan opened a new pull request, #37609:
URL: https://github.com/apache/airflow/pull/37609
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
Used the dag and task decorators to document a more compelling front page which makes Airflow's strengths more immediately visible to potential users.
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "philiprbrenan (via GitHub)" <gi...@apache.org>.
philiprbrenan commented on PR #37609:
URL: https://github.com/apache/airflow/pull/37609#issuecomment-1959648739
Simplified the initial demo example by replacing the call to ``expand`` with explicit tasks. I tried using ``>>`` to organize @task decorated methods but this is not permitted requiring the use of parameters instead. This means that the old way of doing things and the new decorated way of doing things cannot be mixed. The new decorated technique is more much more compelling because it is compact and thus easy to understand, reason about and build upon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37609:
URL: https://github.com/apache/airflow/pull/37609#issuecomment-1966554895
Looks like Chat-GPT generated response. No time to read it, so I shortened it with Chat GPT:
```
Initially skeptical of Airflow, I found it cumbersome and lacking conceptual integrity, akin to scattered building blocks without a coherent design. However, a revelation came when I unearthed a pure Python solution buried deep within Airflow's documentation, utilizing only the @dag and @task decorators. This discovery transformed my perception, turning Airflow from an annoyance to an indispensable tool. Motivated by this experience, I crafted a new introductory approach, emphasizing the simplicity and power of the decorators. Despite suggestions to include the >> operator, I found it confounding and incongruent with typical programming flow. Presenting both approaches risks confusing new users with unnecessary choices, particularly through a GUI interface. Therefore, I advocate for a streamlined introduction focused solely on the decorators, aligning with Airflow's architectural elegance and ensuring a smoother entry for newcomers without unnecessary complexity.
```
This has been discussed multiple times and your point of view is only of yours. Airflow has multiple different kind of users - both experienced Pythonistas and people who are just data scientists and don't care too much about python - they want to put their DAGs from existing building blocks (i.e. operators)
That's why - what we are asking you to make the first page in the way that appeal to both kind of users - and the tab switching of different ways of programming with Airflow is a good idea. You have to accept the fact of life that other people have different experiences and think differently - and Airflow does not have to respond to your mindset only, but also to other mindsets and our direction guiding you there make it so. Even if not perfectly fitting either, making it appealing to both is our goal.
Kind request to adjust to those expectations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37609:
URL: https://github.com/apache/airflow/pull/37609#issuecomment-1961971507
cc: @BasPH - I know you have an experience and are interested in reviewing and "thinking" in the "first contact" kind of docs - I've done my share of comments and directions - curious about your review/comments. This is a super-important part of our documentation so I think we shoudl have at least few poeple who know/care about it to give their feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "BasPH (via GitHub)" <gi...@apache.org>.
BasPH commented on PR #37609:
URL: https://github.com/apache/airflow/pull/37609#issuecomment-1965339944
@philiprbrenan could you elaborate on how you see this change making Airflow's strengths more visible?
One thing I do suggest adding, as @potiuk mentioned, is both ways of writing DAGs (using tabs): (1) the object-oriented approach and (2) the functional approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37609:
URL: https://github.com/apache/airflow/pull/37609#discussion_r1499108113
##########
docs/apache-airflow/index.rst:
##########
@@ -19,56 +19,71 @@ What is Airflow™?
=========================================
`Apache Airflow™ <https://github.com/apache/airflow>`_ is an open-source platform for developing, scheduling,
-and monitoring batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows
+and monitoring distributable, batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows
connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is
-deployable in many ways, varying from a single process on your laptop to a distributed setup to support even
-the biggest workflows.
+deployable in many ways, varying from a single process on your laptop to a distributed setup across a network of many
+machines to support even the largest of workflows.
Workflows as code
=========================================
-The main characteristic of Airflow workflows is that all workflows are defined in Python code. "Workflows as
-code" serves several purposes:
+The main characteristic of Airflow workflows is that all workflows are defined in Python code as a
+`Directed Acyclic Graph <https://en.wikipedia.org/wiki/Directed_acyclic_graph>`__.
+
+"Workflows as code" serves several purposes:
- **Dynamic**: Airflow pipelines are configured as Python code, allowing for dynamic pipeline generation.
- **Extensible**: The Airflow™ framework contains operators to connect with numerous technologies. All Airflow components are extensible to easily adjust to your environment.
-- **Flexible**: Workflow parameterization is built-in leveraging the `Jinja <https://jinja.palletsprojects.com>`_ templating engine.
+- **Flexible**: Workflows can be easily parameterized by using the built-in `Jinja <https://jinja.palletsprojects.com>`_ templating engine.
Take a look at the following snippet of code:
.. code-block:: python
- from datetime import datetime
+ # Create tasks using a decorated DAG and pass values between those tasks.
Review Comment:
I like the proposal in general, but I have one big ask:
I think we should not make the mapped-task example as the first thing the user sees. This is really a bit more "Advanced" use case and not one that ss the main strength of airflow, also it introduces quite a few concepts at once and they might be difficult to grasp by a new user.
Immediate questions:
* shoudl I always do that even in a simple case ?
* what is that expand thing?
* what the hell those values are.
I think those **maybe** should folllow the basic example as a more advance case (which I would find very useful to add). This should be clearly marked with something like "while simple workflows are simple, you can also do much more complex tasks where - for example - you can replicate tasks to run in parallel, allowing airflow fo run workflow with semantics similar to MapReduce" - followed by the new example. That would be more appealing to the users who hit the page and have no idea about Airflow.
This should be similar in concept to the "architecture overview" - where we start with a basic, simplest deployment of Airflow and then follow that with more complex cases that focus on distribution and eventually security-focused deployments.
See https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html#architecture-diagrams
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #37609:
URL: https://github.com/apache/airflow/pull/37609#issuecomment-1958571210
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "philiprbrenan (via GitHub)" <gi...@apache.org>.
philiprbrenan commented on PR #37609:
URL: https://github.com/apache/airflow/pull/37609#issuecomment-1961813465
_put example to a separate section_
There are now two examples - a very simple one with **>>** operators, a more complex one with decorators , both in their own sections.
_make the example more compact and maybe do something with the screenshots to get them fit into smaller space so that they can be seen together on a single page_
Screenshots have been tightened up and reduced by 50%
_have two swappable variants of the DAG - classic and task flow one_
There are now two examples _classic_: a very simple one with >> operators, task with a more complex one with decorators. A screen shot has been added to show how to run the dags and the log results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "philiprbrenan (via GitHub)" <gi...@apache.org>.
philiprbrenan commented on PR #37609:
URL: https://github.com/apache/airflow/pull/37609#issuecomment-1965642832
I have had the benefit of being forced to use ``Airflow`` on an important project with real time pressure and monetary consequences.
I say _forced_ because the first time I saw ``Airflow`` I was underwhelmed. My initial take was: "_so what - this does not seem to advance the state of the art beyond anything that I can already do, so why would I want to take the time to learn how to use this_"? Indeed, using it seemed a retrograde step because of the need to define all the tasks in the ``DAG`` before the ``DAG`` could be run.
Thus, initially ``Airflow`` appeared to me as a confused heap of building blocks that had been left lying littered around in a corner of the Internet for people to stumble into and waste their time falling over them in the dark as opposed to something essential. Rather like the initial Soviet view of the intelligence flowing from Los Alamos. Why for example did we need a bash operator when there was plenty of capability to execute stuff from the command line in Python? Why did we need a special Python operator for determining the next task to run? I was already perfectly capable of sending an email or executing a bash command from Python and did not want to know that there was yet another way of doing these mundane tasks.
In particular, this amorphous mass of blocks seemed to lack the conceptual integrity considered so essential in system design as detailed in Chapter 4 of the "Mythical Man Month -- Brookes" thus:
_I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas._
The thing that changed my mindset completely, a Damascene conversion that transformed me from scathingly ``anti`` to completely ``pro``, was that eventually, after some considerable time and effort, I discovered, deep in the documentation, the pure Python solution using just the ``@dag`` and ``@task`` decorators. These decorators allowed me to mark up my code to get it run in parallel under ``Airflow`` with the smallest amount of change and disruption possible to the existing code. Suddenly, ``Airflow`` went from being extremely annoying to powerfully enabling, from almost ``useless`` to absolutely ``essential``.
Fired by my experience I wrote a proposed new intro to ``Airflow`` that would spare future users the tedious and expensive learning process that I had been forced to go through.
My first version of the improved intro used only the ``@dag`` and ``@task`` decorators to demonstrate a minimal pure Python ``DAG`` that illustrated the essential power of ``Airflow`` in a compelling and easily understood way.
The idea was that a skilled Python programmer could then easily fill in all the other details for themselves if they wanted to, or if they preferred, dive deeper into the other offerings available in ``Airflow``. Either way they would be in through the door and we would be in business.
I proposed my improved intro via a pull request. Helpful comments were made and the initial proposal improved further.
One comment that was made was that there should also be a demonstration of the ``>>`` operator as it was seen as an important user facing component.
Initially I was in favor of this and I modified my example to include usage of the ``>>`` operator. To my surprise, I found out you cannot use ``>>`` with ``@task`` . To overcome this problem, I produced a small separate ``DAG`` that used ``>>`` simply to illustrate this feature and placed it first before getting on with the main business of the ``@dag`` and ``@task`` decorators without further reference to ``>>``.
I have to say that I do not find it easy to explain the ``>>`` operator to other users. Programmers, in particular, want to see a flow of data through their code: you call this routine to get this value which you then hand to this routine to process it further. Stuff that is earlier in the code usually gets executed than stuff later in the code. But the ``>>`` seems to require one to redundantly repeat something that is obvious anyway while ignoring the flow of data. The power of the decorators is that all of these issues suddenly disappear. ``Airflow`` code starts to looks like normal code again with ``Airflow`` working invisibly behind the scenes to move tasks to different machines and figure out which tasks can be run in parallel because they lack dependencies rather than requiring the user to state what they think the dependencies might be. ``>>`` feels redundant and alien while ``@task`` feels comfortable and familiar.
Having invested a good deal of time in creating the proposed intro I produced a preceding ``>>`` example in high hopes of getting through to completion and acceptance of my main proposal. But really, all that having two examples in the intro does is confuse the new ``Airflow`` user by providing two wildly different ways of doing two different things. And such users will be confused even more by asking them to make an important decision, which approach they prefer, when they are least capable of making such a decision.
We are now proposing to dig this hole deeper by having a tab where the new user has to make that confusing initial decision with the help of a GUI, that must by its nature, give precedence to one example over the other and thus exert an unwanted bias on the unsuspecting new user hoping to be enlightened as to the essential essence of ``Airflow`` with as few clicks and as little bias as possible.
We should spare new users this conundrum. We should put our best foot forward from the get go. The decorators approach is compelling to new users in a way that the ``>>`` operator is not. ``>>`` is important infrastructure, but it is not architecture. The decorators approach will compel new users to enter the cathedral of ``Airflow`` to admire the beauty of its architectural simplicity and integrity, as recommended by Brookes. And if the new intro does not work as expected, we can easily revert to the existing intro and try again later in the light of the new experience gained.
May I therefore recommend that I be encouraged to resubmit my proposal minus the leading ``>>`` operator example in the sure and certain expectation of imminent publication?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Modernized front page scenario [airflow]
Posted by "philiprbrenan (via GitHub)" <gi...@apache.org>.
philiprbrenan closed pull request #37609: Modernized front page scenario
URL: https://github.com/apache/airflow/pull/37609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org