[jira] [Commented] (AIRFLOW-2253) Adding Airflow CLI instrumentation

["Kevin Campbell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467210#comment-16467210 ] 

Kevin Campbell commented on AIRFLOW-2253:
-----------------------------------------

This change results in a failure for the initdb method when run on an empty database
{code:java}
[2018-05-08 10:22:33,295] {cli_action_loggers.py:69} ERROR - Failed on pre-execution callback using <function default_action_log at 0x7f92b770d048>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "log" does not exist
LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 67, in on_pre_execution
cb(**kwargs)
File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 99, in default_action_log
session.commit()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 927, in commit
self.transaction.commit()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 467, in commit
self._prepare_impl()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2209, in flush
self._flush(objects)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2329, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2293, in _flush
flush_context.execute()
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
rec.execute(self)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
uow
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
mapper, table, insert)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements
execute(statement, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
context)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
context)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "log" does not exist
LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
^
[SQL: 'INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s, %(task_id)s, %(event)s, %(execution_date)s, %(owner)s, %(extra)s) RETURNING log.id'] [parameters: {'dttm': datetime.datetime(2018, 5, 8, 10, 22, 33, 280371, tzinfo=datetime.timezone.utc), 'dag_id': None, 'task_id': None, 'event': 'cli_initdb', 'execution_date': None, 'owner': 'airflow', 'extra':
'{"host_name": "web-74bd6d846f-8bdmn", "full_command": "[\'/usr/local/bin/airflow\', \'initdb\']"}'}]{code}

> Adding Airflow CLI instrumentation
> ----------------------------------
>
>                 Key: AIRFLOW-2253
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2253
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Jin Hyuk Chang
>            Assignee: Jin Hyuk Chang
>            Priority: Major
>
> We are trying to improve user experience on Airflow particularly on Airflow CLI where our user does most of testing in CLI. As a first step, we'd like to instrument user's activity on CLI.
> Currently, Airflow instruments user activity from web UI via www.utils.action_logger, but not on CLI and we are trying to instrument user activity from Airflow CLI as well. By default, we'd like to instrument same as what current behavior from Web UI.
> Should it need different instrumentation, we will provide a registry method to register callback so that customized instrument can be easily plugged in. (e.g: We have our customized event topic that already wired with ingestion pipeline. Having pluggable callback, making integration with customized instrumentation a breeze.)
> Thanks!
> Jin



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)