You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/20 10:21:39 UTC
[GitHub] [airflow] Sanchit112 opened a new pull request #16548: Updated; Added new pipeline example for the tutorial docs (Issue #11208)
Sanchit112 opened a new pull request #16548:
URL: https://github.com/apache/airflow/pull/16548
Added hooks and changed whats next position as per request from https://github.com/apache/airflow/pull/16084
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Sanchit112 commented on a change in pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
Sanchit112 commented on a change in pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#discussion_r659167596
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -1,23 +1,3 @@
- .. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- .. http://www.apache.org/licenses/LICENSE-2.0
-
- .. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-
-
Review comment:
ok ill add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #16548:
URL: https://github.com/apache/airflow/pull/16548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#issuecomment-869030240
YEAH. YET another random pylint problem. Hopefuly we get rid of it by Monday.
In the meantime just do what I did here: https://github.com/apache/airflow/pull/16674/files#diff-1d96ddfb92ff78838a2d87ff6cdbd5cf445e841fbe78ceea9de732744ffc7728R3653
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#discussion_r656010619
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -426,17 +411,18 @@ Let's break this down into 3 steps: get data, insert data, merge data:
for row in response.text.split("\n"):
file.write(row)
-Here we are passing a``GET`` request to get the data from the URL and save it in ``employees.csv`` file on our Airflow instance.
+Here we are passing a`GET` request to get the data from the URL and save it in `employees.csv` file on our Airflow instance.
Review comment:
This is reStructuredText, not markdown, so two backticks is correct for inline code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Sanchit112 commented on pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
Sanchit112 commented on pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#issuecomment-869020592
@kaxil can you help me with this? Not able to get this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#discussion_r656010619
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -426,17 +411,18 @@ Let's break this down into 3 steps: get data, insert data, merge data:
for row in response.text.split("\n"):
file.write(row)
-Here we are passing a``GET`` request to get the data from the URL and save it in ``employees.csv`` file on our Airflow instance.
+Here we are passing a`GET` request to get the data from the URL and save it in `employees.csv` file on our Airflow instance.
Review comment:
This is reStructuredText, not markdown, so two backticks is correct for inline code
```suggestion
Here we are passing a ``GET`` request to get the data from the URL and save it in ``employees.csv`` file on our Airflow instance.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#issuecomment-868987320
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Sanchit112 commented on pull request #16548: Updated; Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
Sanchit112 commented on pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#issuecomment-864531651
@mik-laj @kaxil
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#discussion_r654918088
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -426,17 +407,16 @@ Let's break this down into 3 steps: get data, insert data, merge data:
for row in response.text.split("\n"):
file.write(row)
-Here we are passing a``GET`` request to get the data from the URL and save it in ``employees.csv`` file on our Airflow instance.
+Here we are passing a`GET` request to get the data from the URL and save it in `employees.csv` file on our Airflow instance.
.. code-block:: python
@task
def insert_data():
- engine = create_engine(
- "postgresql+psycopg2://postgres:password@localhost:5432/postgres"
- )
+ postgres_hook = PostgresHook(postgres_conn_id="LOCAL")
+ conn = postgres_hook.get_conn()
df = pd.read_csv("/usr/local/airflow/dags/files/employees.csv")
- df.to_sql("Employees_temp", con=engine, if_exists="replace", chunksize=1000)
+ df.to_sql("Employees_temp", conn, if_exists="replace", chunksize=1000)
Review comment:
Are you sure if it works? Based on Pandas documentation, Dataframe supports [`sqlalchemy.engine.(Engine or Connection) or sqlite3.Connection`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html) only. Here, we have `psycopg2.extensions.connection`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#issuecomment-866611494
Static checks are red. Can you fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#issuecomment-864534584
I think the whole .csv file was added again ? That was unintentional ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#issuecomment-869030545
Going to ignore random Pylint issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#discussion_r654917551
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -358,25 +358,6 @@ which are used to populate the run schedule with task instances from this dag.
--start-date 2015-06-01 \
--end-date 2015-06-07
-What's Next?
--------------
-That's it, you have written, tested and backfilled your very first Airflow
-pipeline. Merging your code into a code repository that has a master scheduler
-running against it should get it to get triggered and run every day.
-
-Here's a few things you might want to do next:
-
-.. seealso::
- - Read the :doc:`/concepts/index` section for detailed explanation of Airflow concepts such as DAGs, Tasks, Operators, and more.
- - Take an in-depth tour of the UI - click all the things!
- - Keep reading the docs!
-
- - Review the :doc:`how-to guides<howto/index>`, which include a guide to writing your own operator
- - Review the :ref:`Command Line Interface Reference<cli>`
- - Review the :ref:`List of operators <pythonapi:operators>`
- - Review the :ref:`Macros reference<macros>`
- - Write your first pipeline!
-
Lets look at another example; we need to get some data from a file which is hosted online and need to insert into our local database. We also need to look at removing duplicate rows while inserting.
Review comment:
Can you add a new header here to separate two examples?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #16548: Updated: Added new pipeline example for the tutorial docs (Issue #11208)
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #16548:
URL: https://github.com/apache/airflow/pull/16548#discussion_r659163828
##########
File path: docs/apache-airflow/tutorial.rst
##########
@@ -1,23 +1,3 @@
- .. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- .. http://www.apache.org/licenses/LICENSE-2.0
-
- .. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-
-
Review comment:
We need the license @Sanchit112 at the top of the file :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org