You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by fo...@apache.org on 2018/01/20 09:05:26 UTC
incubator-airflow git commit: [AIRFLOW-2017][Airflow 2017] adding
query output to PostgresOperator
Repository: incubator-airflow
Updated Branches:
refs/heads/master 33c720421 -> 97ca9791c
[AIRFLOW-2017][Airflow 2017] adding query output to PostgresOperator
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow 2017]
(https://issues.apache.org/jira/browse/AIRFLOW-201
7/) issues and references them in the PR title.
For example, "[AIRFLOW-2017] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2017
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
Currently we're not getting the output logs of the
postgres operator that you would get otherwise if
you ran a psql command. It's because the postgres
conn has an attribute called [notices](http://init
d.org/psycopg/docs/connection.html#connection.noti
ces) which contains this information.
We need to just print the results of this to get
that output in the airflow logs, which makes it
easy to debug amongst other things.
I've included some images for before and after
pictures.
**BEFORE**
<img width="1146" alt="screen shot 2018-01-19 at 4
46 59 pm" src="https://user-images.githubuserconte
nt.com/10408007/35178405-6f6a1da8-fd3d-11e7-8f50-0
dbd567d8ab4.png">
**AFTER**
<img width="1147" alt="screen shot 2018-01-19 at 4
46 25 pm" src="https://user-images.githubuserconte
nt.com/10408007/35178406-74ea4ae6-fd3d-11e7-9551-6
31eac6bfe7b.png">
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
There isn't anything to test, there is nothing
changing to the current implementation besides an
addition of logging.
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #2959 from Acehaidrey/AIRFLOW-2017
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/97ca9791
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/97ca9791
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/97ca9791
Branch: refs/heads/master
Commit: 97ca9791c34e528e8072b3a01dcd96965685530e
Parents: 33c7204
Author: Ace Haidrey <ah...@pandora.com>
Authored: Sat Jan 20 10:05:16 2018 +0100
Committer: Fokko Driesprong <fo...@godatadriven.com>
Committed: Sat Jan 20 10:05:16 2018 +0100
----------------------------------------------------------------------
airflow/hooks/postgres_hook.py | 4 ++--
airflow/operators/postgres_operator.py | 2 ++
2 files changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/97ca9791/airflow/hooks/postgres_hook.py
----------------------------------------------------------------------
diff --git a/airflow/hooks/postgres_hook.py b/airflow/hooks/postgres_hook.py
index 81e10d7..1c016a9 100644
--- a/airflow/hooks/postgres_hook.py
+++ b/airflow/hooks/postgres_hook.py
@@ -51,8 +51,8 @@ class PostgresHook(DbApiHook):
'keepalives_idle']:
conn_args[arg_name] = arg_val
- psycopg2_conn = psycopg2.connect(**conn_args)
- return psycopg2_conn
+ self.conn = psycopg2.connect(**conn_args)
+ return self.conn
def copy_expert(self, sql, filename, open=open):
'''
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/97ca9791/airflow/operators/postgres_operator.py
----------------------------------------------------------------------
diff --git a/airflow/operators/postgres_operator.py b/airflow/operators/postgres_operator.py
index c93dc7b..e8a8a68 100644
--- a/airflow/operators/postgres_operator.py
+++ b/airflow/operators/postgres_operator.py
@@ -53,3 +53,5 @@ class PostgresOperator(BaseOperator):
self.hook = PostgresHook(postgres_conn_id=self.postgres_conn_id,
schema=self.database)
self.hook.run(self.sql, self.autocommit, parameters=self.parameters)
+ for output in self.hook.conn.notices:
+ self.log.info(output)