You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/01/03 07:04:32 UTC
[GitHub] [airflow-site] chandulal opened a new pull request #231: Add blogs
for airflow DAGs testing
chandulal opened a new pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] chandulal commented on a change in pull request
#231: Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
chandulal commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362793178
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+linkTitle: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+author: "Chandu Kavar and Sarang Shinde"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the xComs and DAGs end-to-end"
+tags: ["Testing"]
+date: "2020-01-03"
+---
+In [part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c) of this blog, I explained DAG validation tests, DAG definition tests, and Unit tests. These tests help to verify the syntax and cyclicity in the DAG, upstream and downstream of each task, the logic of the custom operator and sensor, etc. I would recommend reading the first part of this post.
+
+In this part, I covered the integration and end-to-end pipeline tests.
+
+You can find this blog on the medium [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82).
Review comment:
I made part2 paywall-free on medium as well. The link will be the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #231:
Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362833802
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+linkTitle: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+author: "Chandu Kavar and Sarang Shinde"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the xComs and DAGs end-to-end"
+tags: ["Testing"]
+date: "2020-01-03"
+---
+In [part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c?source=friends_link&sk=3e1ec0b1e5fcc1e9c258e08d260c7152) of this blog, I explained DAG validation tests, DAG definition tests, and Unit tests. These tests help to verify the syntax and cyclicity in the DAG, upstream and downstream of each task, the logic of the custom operator and sensor, etc. I would recommend reading the first part of this post.
+
+In this part, I covered the integration and end-to-end pipeline tests.
+
+You can find this blog on the medium [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82?source=friends_link&sk=0841c4c3bf786a48e748f0e6c1fd189f).
Review comment:
Btw I saw that there were similar blog posts that were published. I have raised this issue and asked for feedback from the entire community.
Feel free to voice your opinion on the mailing list :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] chandulal commented on a change in pull request
#231: Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
chandulal commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362797692
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+linkTitle: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+author: "Chandu Kavar and Sarang Shinde"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the xComs and DAGs end-to-end"
+tags: ["Testing"]
+date: "2020-01-03"
+---
+In [part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c) of this blog, I explained DAG validation tests, DAG definition tests, and Unit tests. These tests help to verify the syntax and cyclicity in the DAG, upstream and downstream of each task, the logic of the custom operator and sensor, etc. I would recommend reading the first part of this post.
+
+In this part, I covered the integration and end-to-end pipeline tests.
+
+You can find this blog on the medium [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82).
Review comment:
I added a friend link for the safer side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] louisguitton commented on a change in pull request
#231: Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
louisguitton commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362767420
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+linkTitle: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+author: "Chandu Kavar and Sarang Shinde"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the xComs and DAGs end-to-end"
+tags: ["Testing"]
+date: "2020-01-03"
+---
+In [part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c) of this blog, I explained DAG validation tests, DAG definition tests, and Unit tests. These tests help to verify the syntax and cyclicity in the DAG, upstream and downstream of each task, the logic of the custom operator and sensor, etc. I would recommend reading the first part of this post.
+
+In this part, I covered the integration and end-to-end pipeline tests.
+
+You can find this blog on the medium [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82).
Review comment:
Just to mention that as a reader, I often ran into the Medium paywall until I converted recently.
For those people who are not paying Medium customers, I think there is a way for authors to get a paywall-free link to their articles. Is it the case here ?
Else, do you have that article on another (free) platform like for part 1 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] chandulal commented on a change in pull request
#231: Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
chandulal commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362794116
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+linkTitle: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+author: "Chandu Kavar and Sarang Shinde"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the xComs and DAGs end-to-end"
+tags: ["Testing"]
+date: "2020-01-03"
+---
+In [part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c) of this blog, I explained DAG validation tests, DAG definition tests, and Unit tests. These tests help to verify the syntax and cyclicity in the DAG, upstream and downstream of each task, the logic of the custom operator and sensor, etc. I would recommend reading the first part of this post.
+
+In this part, I covered the integration and end-to-end pipeline tests.
+
+You can find this blog on the medium [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82).
Review comment:
Let me update the free URL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] turbaszek closed pull request #231: Add blogs for
airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
turbaszek closed pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] chandulal commented on a change in pull request
#231: Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
chandulal commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362793178
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+linkTitle: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+author: "Chandu Kavar and Sarang Shinde"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the xComs and DAGs end-to-end"
+tags: ["Testing"]
+date: "2020-01-03"
+---
+In [part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c) of this blog, I explained DAG validation tests, DAG definition tests, and Unit tests. These tests help to verify the syntax and cyclicity in the DAG, upstream and downstream of each task, the logic of the custom operator and sensor, etc. I would recommend reading the first part of this post.
+
+In this part, I covered the integration and end-to-end pipeline tests.
+
+You can find this blog on the medium [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82).
Review comment:
I made part2 paywall-free on medium as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #231:
Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362821134
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 1 - DAG Validation Tests DAG, Definition Tests and Unit Tests"
+linkTitle: "Testing in Airflow Part 1 - DAG Validation Tests, DAG Definition Tests and Unit Tests"
+author: "Chandu Kavar"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the validity of DAGs, typos, cyclicity, upstream and downstream dependencies and custom operator/sensors."
+tags: ["Testing"]
+date: "2020-01-03"
+---
+Testing is an integral part of any software system to build confidence and increase the reliability of the system. Recently, I joined Grab and here at Grab, we are using Airflow to create and manage pipelines. But, we were facing issues with Airflow. I had a conversation with my engineering manager and discussed on how we could make Airflow reliable and testable.
+Before Grab, I worked with ThoughtWorks. They are using TDD (Test Driven Development) methodology in almost all the projects. As a matter of fact, everyone is very cognizant about testing. This motivated me to explore testing in Airflow.
+I spent a weekend Googling. But, I didn’t find any good articles on airflow test covering all the aspects, there are only a handful of articles available. I felt it’s worth to write and share a blog on airflow testing.
+
+You can find more details on how to write these tests with examples in [Testing in Airflow Part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c?source=friends_link&sk=3e1ec0b1e5fcc1e9c258e08d260c7152).
Review comment:
Sorry, the content needs to entirely on the Airflow website.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] turbaszek commented on issue #231: Add blogs for
airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#issuecomment-608517584
Closing due to no response from author
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] kaxil commented on a change in pull request #231:
Add blogs for airflow DAGs testing
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #231: Add blogs for airflow DAGs testing
URL: https://github.com/apache/airflow-site/pull/231#discussion_r362821213
##########
File path: landing-pages/site/content/en/blog/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests.md
##########
@@ -0,0 +1,16 @@
+---
+title: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+linkTitle: "Testing in Airflow Part 2 - Integration Tests and End-to-End Pipeline Tests"
+author: "Chandu Kavar and Sarang Shinde"
+twitter: "chandukavar"
+github: "chandulal"
+linkedin: "chandu-kavar-627a209b"
+description: "To know how to test the xComs and DAGs end-to-end"
+tags: ["Testing"]
+date: "2020-01-03"
+---
+In [part 1](https://blog.usejournal.com/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c?source=friends_link&sk=3e1ec0b1e5fcc1e9c258e08d260c7152) of this blog, I explained DAG validation tests, DAG definition tests, and Unit tests. These tests help to verify the syntax and cyclicity in the DAG, upstream and downstream of each task, the logic of the custom operator and sensor, etc. I would recommend reading the first part of this post.
+
+In this part, I covered the integration and end-to-end pipeline tests.
+
+You can find this blog on the medium [Testing in Airflow Part 2](https://medium.com/@chandukavar/testing-in-airflow-part-2-integration-tests-and-end-to-end-pipeline-tests-af0555cd1a82?source=friends_link&sk=0841c4c3bf786a48e748f0e6c1fd189f).
Review comment:
Same here: Sorry, the content needs to entirely on the Airflow website.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services