You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Alexander Petrovsky (JIRA)" <ji...@apache.org> on 2017/06/29 15:24:00 UTC
[jira] [Updated] (AIRFLOW-1358) SQL file renderer and
on_failure_callback
[ https://issues.apache.org/jira/browse/AIRFLOW-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Petrovsky updated AIRFLOW-1358:
-----------------------------------------
Issue Type: Bug (was: New Feature)
> SQL file renderer and on_failure_callback
> -----------------------------------------
>
> Key: AIRFLOW-1358
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1358
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
> Reporter: Alexander Petrovsky
>
> I have the follow dag:
> https://gist.github.com/juise/cc006584e17be72e78550aab53a1be9c
> When I broke my firs step (prepare), on_failure_callback function was called, but it's crashed too, due looks like in sql rendered something is broken, and he can't work with sql files (by example, by cleanup.sql file, see more info below). But, when I use flat sql instead of sql file, everything works perfectly.
> My cleanup.sql file:
> {code:java}
> $ cat cleanup.sql
> DROP TABLE IF EXISTS etl.{{ params.target_table }}_{{ ts_nodash }} CASCADE;
> {code:java}
> Log:
> {code:java}
> [2017-06-29 15:14:13,474] {models.py:168} INFO - Filling up the DagBag from /opt/airflow/dags/ga_full_stats_test/ga_full_stats.py
> [2017-06-29 15:14:13,852] {base_hook.py:67} INFO - Using connection to: v2.rtty.in
> [2017-06-29 15:14:13,858] {base_hook.py:67} INFO - Using connection to: vga.rtty.in
> [2017-06-29 15:14:13,893] {base_task_runner.py:112} INFO - Running: ['bash', '-c', u'airflow run ga_full_stats_test_v2 prepare 2017-06-06T10:27:00 --job_id 123798 --raw -sd DAGS_FOLDER/ga_full_stats_test/ga_full_stats.py']
> [2017-06-29 15:14:14,391] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:14,391] {__init__.py:57} INFO - Using executor CeleryExecutor
> [2017-06-29 15:14:14,924] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:14,924] {models.py:168} INFO - Filling up the DagBag from /opt/airflow/dags/ga_full_stats_test/ga_full_stats.py
> [2017-06-29 15:14:15,088] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,087] {base_hook.py:67} INFO - Using connection to: v2.rtty.in
> [2017-06-29 15:14:15,091] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,091] {base_hook.py:67} INFO - Using connection to: vga.rtty.in
> [2017-06-29 15:14:15,109] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,109] {models.py:1128} INFO - Dependencies all met for <TaskInstance: ga_full_stats_test_v2.prepare 2017-06-06 10:27:00 [queued]>
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,112] {models.py:1128} INFO - Dependencies all met for <TaskInstance: ga_full_stats_test_v2.prepare 2017-06-06 10:27:00 [queued]>
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,112] {models.py:1328} INFO -
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: Starting attempt 1 of 1
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
> [2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
> [2017-06-29 15:14:15,120] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,120] {models.py:1352} INFO - Executing <Task(VerticaOperator): prepare> on 2017-06-06 10:27:00
> [2017-06-29 15:14:15,138] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,137] {vertica_operator.py:45} INFO - Executing: CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;
> [2017-06-29 15:14:15,141] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,141] {base_hook.py:67} INFO - Using connection to: vga.rtty.in
> [2017-06-29 15:14:15,158] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,158] {dbapi_hook.py:167} INFO - CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;
> [2017-06-29 15:14:15,164] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,164] {models.py:1427} ERROR - Severity: ERROR, Message: Syntax error at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror, File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL: 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;'
> [2017-06-29 15:14:15,164] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1384, in run
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: result = task_copy.execute(context=context)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py", line 47, in execute
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: hook.run(self.sql)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in run
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: cur.execute(s)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line 111, in execute
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: raise errors.QueryError.from_error_response(self._message, operation)
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: VerticaSyntaxError: Severity: ERROR, Message: Syntax error at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror, File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL: 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;'
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,164] {models.py:1451} INFO - Marking task as FAILED.
> [2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,165] {vertica_operator.py:45} INFO - Executing: cleanup.sql
> [2017-06-29 15:14:15,169] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,169] {base_hook.py:67} INFO - Using connection to: vga.rtty.in
> [2017-06-29 15:14:15,181] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,181] {dbapi_hook.py:167} INFO - cleanup.sql
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,185] {models.py:1466} ERROR - Failed at executing callback
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,185] {models.py:1467} ERROR - Severity: ERROR, Message: Syntax error at or near "cleanup", Sqlstate: 42601, Position: 1, Routine: base_yyerror, File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL: 'cleanup.sql'
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1464, in handle_failure
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: task.on_failure_callback(context)
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File "/opt/airflow/dags/ga_full_stats_test/ga_full_stats.py", line 64, in cleanup_on_failure
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: cleanup.execute(context)
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py", line 47, in execute
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: hook.run(self.sql)
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in run
> [2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: cur.execute(s)
> [2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line 111, in execute
> [2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask: raise errors.QueryError.from_error_response(self._message, operation)
> [2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask: VerticaSyntaxError: Severity: ERROR, Message: Syntax error at or near "cleanup", Sqlstate: 42601, Position: 1, Routine: base_yyerror, File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL: 'cleanup.sql'
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: [2017-06-29 15:14:15,191] {models.py:1472} ERROR - Severity: ERROR, Message: Syntax error at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror, File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL: 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;'
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File "/usr/bin/airflow", line 28, in <module>
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: args.func(args)
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: pool=args.pool,
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: result = func(*args, **kwargs)
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1384, in run
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: result = task_copy.execute(context=context)
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py", line 47, in execute
> [2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: hook.run(self.sql)
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in run
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: cur.execute(s)
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: File "/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line 111, in execute
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: raise errors.QueryError.from_error_response(self._message, operation)
> [2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: vertica_python.errors.VerticaSyntaxError: Severity: ERROR, Message: Syntax error at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror, File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL: 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;'
> [2017-06-29 15:14:15,893] {jobs.py:2107} INFO - Task exited with return code 1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)