You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Dan Davydov (JIRA)" <ji...@apache.org> on 2017/04/14 18:00:43 UTC
[jira] [Updated] (AIRFLOW-1111) Make logs easier to read and
separate infra/task logs
[ https://issues.apache.org/jira/browse/AIRFLOW-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dan Davydov updated AIRFLOW-1111:
---------------------------------
Description:
Ideally there would be tabs for logs:
Dag-parsing (needs a better name)
1 (Short for "Run 1")
2
...
Dag-parsing would contain the output like:
{code}
[2017-04-14 15:24:54,341] {models.py:168} INFO - Filling up the DagBag from /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,342] {models.py:253} DEBUG - Importing /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_r>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_large>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_tiny>
[2017-04-14 15:24:54,357] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_platinum_medium>
[2017-04-14 15:24:54,553] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup memory in / since it already exists
[2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup airflow in /memory since it already exists
[2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup 2017-04-14 in /memory/airflow since it already exists
[2017-04-14 15:24:54,554] {cgroup_task_runner.py:77} DEBUG - Creating cgroup 82d42ce2-2126-11e7-ad73-22000a896f7d in /memory/airflow/2017-04-14
{code}
and "1" would contain
{code}
2017-04-14 15:24:56,146] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,146] {models.py:1344} INFO - Executing <Task(HiveOperator): insert_canary_row> on 2017-04-14 15:10:00
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,172] {hive_operator.py:98} INFO - Executing: USE datainfra;
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: CREATE TABLE IF NOT EXISTS datainfra.airflow_precious_canary_gold_large (key STRING);
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: INSERT OVERWRITE TABLE datainfra.airflow_precious_canary_gold_large
[2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: SELECT key FROM datainfra.one_row LIMIT 1;
[2017-04-14 15:24:56,187] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,186] {hive_hooks.py:207} INFO - hive -hiveconf airflow.ctx.dag.dag_id=airflow_precious_canary_gold_large -hiveconf airflow.ctx.dag_run.execution_date=2017-04-14T15:10:00 -hiveconf airflow.ctx.task_instance.execution_date=2017-04-14T15:10:00 -hiveconf airflow.ctx.task.task_id=insert_canary_row -hiveconf mapreduce.job.queuename=airflow_canary_gold -f /tmp/airflow_hiveop_ZixLMN/tmphgm9MR
{code}
(note that "Starting attempt X of Y has been removed from this log).
By default "1" would be opened if it exists when a user clicks into the log, otherwise dag-parsing would be opened.
was:
Ideally there would be tabs for logs:
Dag-parsing (needs a better name)
1 (Short for "Run 1")
2
...
Dag-parsing would contain the output like:
{code}
[2017-04-14 15:24:54,341] {models.py:168} INFO - Filling up the DagBag from /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,342] {models.py:253} DEBUG - Importing /srv/data/airflow/airflow_canaries/canaries.py
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_r>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_large>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_medium>
[2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_tiny>
[2017-04-14 15:24:54,357] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_platinum_medium>
{code}
By default "1" would be opened if it exists when a user clicks into the log, otherwise dag-parsing would be opened.
> Make logs easier to read and separate infra/task logs
> -----------------------------------------------------
>
> Key: AIRFLOW-1111
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1111
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Dan Davydov
>
> Ideally there would be tabs for logs:
> Dag-parsing (needs a better name)
> 1 (Short for "Run 1")
> 2
> ...
> Dag-parsing would contain the output like:
> {code}
> [2017-04-14 15:24:54,341] {models.py:168} INFO - Filling up the DagBag from /srv/data/airflow/airflow_canaries/canaries.py
> [2017-04-14 15:24:54,342] {models.py:253} DEBUG - Importing /srv/data/airflow/airflow_canaries/canaries.py
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_medium>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_silver_r>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_large>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_medium>
> [2017-04-14 15:24:54,356] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_gold_tiny>
> [2017-04-14 15:24:54,357] {models.py:373} DEBUG - Loaded DAG <DAG: airflow_precious_canary_platinum_medium>
> [2017-04-14 15:24:54,553] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup memory in / since it already exists
> [2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup airflow in /memory since it already exists
> [2017-04-14 15:24:54,554] {cgroup_task_runner.py:82} DEBUG - Not creating cgroup 2017-04-14 in /memory/airflow since it already exists
> [2017-04-14 15:24:54,554] {cgroup_task_runner.py:77} DEBUG - Creating cgroup 82d42ce2-2126-11e7-ad73-22000a896f7d in /memory/airflow/2017-04-14
> {code}
> and "1" would contain
> {code}
> 2017-04-14 15:24:56,146] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,146] {models.py:1344} INFO - Executing <Task(HiveOperator): insert_canary_row> on 2017-04-14 15:10:00
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,172] {hive_operator.py:98} INFO - Executing: USE datainfra;
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: CREATE TABLE IF NOT EXISTS datainfra.airflow_precious_canary_gold_large (key STRING);
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: INSERT OVERWRITE TABLE datainfra.airflow_precious_canary_gold_large
> [2017-04-14 15:24:56,173] {base_task_runner.py:95} INFO - Subtask: SELECT key FROM datainfra.one_row LIMIT 1;
> [2017-04-14 15:24:56,187] {base_task_runner.py:95} INFO - Subtask: [2017-04-14 15:24:56,186] {hive_hooks.py:207} INFO - hive -hiveconf airflow.ctx.dag.dag_id=airflow_precious_canary_gold_large -hiveconf airflow.ctx.dag_run.execution_date=2017-04-14T15:10:00 -hiveconf airflow.ctx.task_instance.execution_date=2017-04-14T15:10:00 -hiveconf airflow.ctx.task.task_id=insert_canary_row -hiveconf mapreduce.job.queuename=airflow_canary_gold -f /tmp/airflow_hiveop_ZixLMN/tmphgm9MR
> {code}
> (note that "Starting attempt X of Y has been removed from this log).
> By default "1" would be opened if it exists when a user clicks into the log, otherwise dag-parsing would be opened.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)