You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/07 21:07:15 UTC
[GitHub] [airflow] vikramkoka opened a new pull request #12170: Added a second example for TaskFlow API
vikramkoka opened a new pull request #12170:
URL: https://github.com/apache/airflow/pull/12170
Added a second example DAG for the TaskFlow API, this time using Pandas dataframes and files to be more realistic. This also illustrates the mix of task dependencies between task decorators and sensors.
@kaxil @turbaszek @casassg @ashb
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#issuecomment-723502681
The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519238841
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
+ # [END documentation]
+
+ # [START extract]
+ @dag.task()
+ def extract():
+ """
+ #### Extract task
+ A simple Extract task to get data ready for the rest of the data
+ pipeline. In this case, data is read from a file.
+ """
+ order_data_file = '/tmp/order_data.csv'
+
+ order_data_df = pd.read_csv(order_data_file)
+
+ # convert to JSON so that it be returned using xcom
+ order_data_df_str = order_data_df.to_json(orient='split')
Review comment:
@kaxil Yes, I was planning on adding the 1,2, 3 use cases to the tutorial document as soon as I finished the PR for the custom Xcom backend.
I had not thought about adding that to this example file as well, but could do so.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519231352
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
+ # [END documentation]
+
+ # [START extract]
+ @dag.task()
+ def extract():
+ """
+ #### Extract task
+ A simple Extract task to get data ready for the rest of the data
+ pipeline. In this case, data is read from a file.
+ """
+ order_data_file = '/tmp/order_data.csv'
+
+ order_data_df = pd.read_csv(order_data_file)
+
+ # convert to JSON so that it be returned using xcom
+ order_data_df_str = order_data_df.to_json(orient='split')
Review comment:
Definitely a valid concern, I think the plan was to add a notice / warning to show all the use cases but was missed:
1) Using XCom to pass Pandas Dataframe using TaskGroup for **small data** that fits DB (48 KB)
2) To use XCom Backend to store it on GCS / S3 and still pass pandas Dataframe for larger data
3) Using pickling vs json to (de)serialize XCom data.
The other nice to have would be to move the comments from the DAG file into docs similar to how you did in https://github.com/apache/airflow/pull/11308 @vikramkoka so that we can have **NOTES** / **WARNINGS** that are highlighted and more readable too. I can help out in this effort.
How does that sound @turbaszek ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519339494
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
Review comment:
This isn't needed any more I think - the decorator got changed to do this automatically I think.
Right @casassg ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#issuecomment-751294674
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#issuecomment-751323588
Absolutely Jarek, planning on this next week
On Fri, Dec 25, 2020 at 2:20 PM Jarek Potiuk <no...@github.com>
wrote:
> Should you please complete it :) @vikramkoka
> <https://github.com/vikramkoka> @casassg <https://github.com/casassg> ?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/pull/12170#issuecomment-751295606>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGAC72ITSADCTW5IQSNQ4ZLSWUFZHANCNFSM4TN4AEYA>
> .
>
--
Vikram Koka
*SVP Engineering*
Email: vikram@astronomer.io
Mobile: +1 408 966 2203
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r556044790
##########
File path: airflow/example_dags/tutorial_xcom_pandas.py
##########
@@ -0,0 +1,149 @@
+#
Review comment:
@kaxil there was discussion on devlist and I think we've never reached consensus. But in general we were against having custom XCom backend in code, but we may revisit it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519226884
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
+ # [END documentation]
+
+ # [START extract]
+ @dag.task()
+ def extract():
+ """
+ #### Extract task
+ A simple Extract task to get data ready for the rest of the data
+ pipeline. In this case, data is read from a file.
+ """
+ order_data_file = '/tmp/order_data.csv'
+
+ order_data_df = pd.read_csv(order_data_file)
+
+ # convert to JSON so that it be returned using xcom
+ order_data_df_str = order_data_df.to_json(orient='split')
Review comment:
@turbaszek Yeah, Vikram is going to do a follow-up to show usage of Custom XCom backends in more depth and how users can use it too :) For now this PR shows how you can do it without needing to write a Custom XCom backend.
> Imho this example would show the full capabilities of TaskFlow API if it would use custom XCom backend for persisting data frame on external storage like gcs or s3. I know that @marclamberti was working on something around that 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519229130
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
+ # [END documentation]
+
+ # [START extract]
+ @dag.task()
+ def extract():
+ """
+ #### Extract task
+ A simple Extract task to get data ready for the rest of the data
+ pipeline. In this case, data is read from a file.
+ """
+ order_data_file = '/tmp/order_data.csv'
+
+ order_data_df = pd.read_csv(order_data_file)
+
+ # convert to JSON so that it be returned using xcom
+ order_data_df_str = order_data_df.to_json(orient='split')
Review comment:
> For now this PR shows how you can do it without needing to write a Custom XCom backend.
@kaxil I think this PR shows "bad" example of using XCom - the dataframe can be arbitrary large and impact performance of Airflow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] casassg commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
casassg commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r520876869
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
Review comment:
We would need to use the `@dag` decorator for that. I would strongly suggest doing so instead as its more in the TaskFlow API
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519219813
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
+ # [END documentation]
+
+ # [START extract]
+ @dag.task()
+ def extract():
+ """
+ #### Extract task
+ A simple Extract task to get data ready for the rest of the data
+ pipeline. In this case, data is read from a file.
+ """
+ order_data_file = '/tmp/order_data.csv'
+
+ order_data_df = pd.read_csv(order_data_file)
+
+ # convert to JSON so that it be returned using xcom
+ order_data_df_str = order_data_df.to_json(orient='split')
Review comment:
Imho this example would show the full capabilities of TaskFlow API if it would use custom XCom backend for persisting data frame on external storage like gcs or s3. I know that @marclamberti was working on something around that 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#issuecomment-758884945
> In general this looks good to me. I'm not sure if using `dbm` is not an overkill (especially if we want explain what is crucial for custom XCom).
>
> Also, do we plan to add docs (I see the start/end anchors) or we just want to keep and example DAG?
I was planning on adding a doc after this was approved by everyone as being a reasonable approach.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ldacey commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
ldacey commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519226202
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
+ # [END documentation]
+
+ # [START extract]
+ @dag.task()
+ def extract():
+ """
+ #### Extract task
+ A simple Extract task to get data ready for the rest of the data
+ pipeline. In this case, data is read from a file.
+ """
+ order_data_file = '/tmp/order_data.csv'
+
+ order_data_df = pd.read_csv(order_data_file)
+
+ # convert to JSON so that it be returned using xcom
+ order_data_df_str = order_data_df.to_json(orient='split')
Review comment:
Looks cool. Yeah, I generally pass xcoms with blob paths or filters (for parquet datasets) in order pass data to other tasks and would not really ever pass an actual dataframe or even json version of the dataframe through xcom.
Is it feasible for Airflow to support generating tasks similar to .pipe() with pandas (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pipe.html)? Or how your write queries with Spark.
`etl = (extract().pipe(transform).pipe(load, var="total_order_value"))`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#issuecomment-751295606
Should you please complete it :) @vikramkoka @casassg ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r557206475
##########
File path: airflow/example_dags/tutorial_xcom_pandas.py
##########
@@ -0,0 +1,149 @@
+#
Review comment:
@vikramkoka I have no strong opinion. I would only consider adding `.airflowignore` to avoid parsing non-dag files in example directories. Also if the example is there in code, we should make sure that it can be used by anyone.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka closed pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
vikramkoka closed pull request #12170:
URL: https://github.com/apache/airflow/pull/12170
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r556042397
##########
File path: airflow/example_dags/tutorial_xcom_pandas.py
##########
@@ -0,0 +1,149 @@
+#
Review comment:
Any suggestion on where this file should live @turbaszek?
It can stay here too -- If we decide to keep it here, we need to apply the following diff as one of the tests is failing:
```diff
diff --git a/tests/jobs/test_scheduler_job.py b/tests/jobs/test_scheduler_job.py
index da27016a0..f8b6f3811 100644
--- a/tests/jobs/test_scheduler_job.py
+++ b/tests/jobs/test_scheduler_job.py
@@ -3235,6 +3235,7 @@ class TestSchedulerJob(unittest.TestCase):
ignored_files = {
'helper.py',
+ 'tutorial_xcom_pandas.py',
}
example_dag_folder = airflow.example_dags.__path__[0]
for root, _, files in os.walk(example_dag_folder): # pylint: disable=too-many-nested-blocks
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r556988034
##########
File path: airflow/example_dags/tutorial_xcom_pandas.py
##########
@@ -0,0 +1,149 @@
+#
Review comment:
@turbaszek It is possible that I misunderstood the email thread on the devlist, but I thought the conclusion from that thread was that including examples of custom XCom backend is good. And that is exactly what this attempts to be - an example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#issuecomment-762416739
Closing pull request and withdrawing change of custom xcom backends
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12170: Added a second example for TaskFlow API
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12170:
URL: https://github.com/apache/airflow/pull/12170#discussion_r519358811
##########
File path: airflow/example_dags/tutorial_taskflow_api_pandas.py
##########
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=missing-function-docstring
+"""
+### TaskFlow API Example using Pandas
+
+This is a simple ETL data pipeline example which demonstrates the use of the
+TaskFlow API using three simple tasks for Extract, Transform, and Load, while
+using Pandas and loading data from files.
+
+"""
+# [START tutorial]
+# [START import_module]
+import pandas as pd
+
+# The DAG object; we'll need this to instantiate a DAG
+from airflow import DAG
+from airflow.operators.python_operator import PythonOperator
+from airflow.sensors.filesystem import FileSensor
+from airflow.utils.dates import days_ago
+
+# [END import_module]
+
+
+def generate_data_file():
+ tmp_data_file = open('/tmp/order_data.csv', 'w')
+ tmp_data_file.write('order_id,order_value\n')
+ tmp_data_file.write('"1001", 301.27\n')
+ tmp_data_file.write('"1002", 433.21\n')
+ tmp_data_file.write('"1003", 502.22\n')
+ tmp_data_file.close()
+
+
+# [START default_args]
+# These args will get passed on to each operator
+# You can override them on a per-task basis during operator initialization
+default_args = {
+ 'owner': 'airflow',
+}
+# [END default_args]
+
+
+# [START instantiate_dag]
+with DAG(
+ 'tutorial_taskflow_api_pandas',
+ default_args=default_args,
+ description='TaskFlow API ETL Pandas',
+ schedule_interval=None,
+ start_date=days_ago(2),
+ tags=['example'],
+) as dag:
+ # [END instantiate_dag]
+
+ # [START documentation]
+ dag.doc_md = __doc__
+ # [END documentation]
+
+ # [START extract]
+ @dag.task()
+ def extract():
+ """
+ #### Extract task
+ A simple Extract task to get data ready for the rest of the data
+ pipeline. In this case, data is read from a file.
+ """
+ order_data_file = '/tmp/order_data.csv'
+
+ order_data_df = pd.read_csv(order_data_file)
+
+ # convert to JSON so that it be returned using xcom
+ order_data_df_str = order_data_df.to_json(orient='split')
Review comment:
I'm ok with 2 and 3 but 1 still concerns me as this may make users to demand higher memory allowance or just forget about it ("I did as it was in example and it doesn't work").
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org