You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/08 01:34:32 UTC
[GitHub] [airflow] mik-laj opened a new issue #8187: Extend elastic DAG with
a binary tree, grid
mik-laj opened a new issue #8187: Extend elastic DAG with a binary tree, grid
URL: https://github.com/apache/airflow/issues/8187
**Description**
We have ["elastic DAG"](https://github.com/apache/airflow/blob/master/scripts/perf/dags/elastic_dag.py#L77) which we use in various performance tests. It allows you to generate any DAG using environment variables.
An example test that uses such a DAG looks like this
```python
with mock.patch.dict("os.environ", {
"PERF_DAGS_COUNT": str(dag_count),
"PERF_TASKS_COUNT": str(task_count),
"PERF_START_AGO": start_ago,
"PERF_SCHEDULE_INTERVAL": schedule_interval,
"PERF_SHAPE": shape,
}), conf_vars({
('scheduler', 'use_job_schedule'): 'True',
}):
dagbag = DagBag(dag_folder=ELASTIC_DAG_FILE, include_examples=False)
with assert_queries_count(expected_query_count):
processor = DagFileProcessor([], mock.MagicMock())
processor._process_dags(dagbag.dags.values())
```
Source:
https://github.com/apache/airflow/blob/6bd2e59/tests/jobs/test_scheduler_job.py#L1175
This DAG File allows you to generate DAGs that contain all the tasks connected in a linear manner or have no connections.
It would be useful to add more complex DAGs, especially binary trees or/and grids. This will allow us to test more complex cases more easily.
**Use case / motivation**
N/A
**Related Issues**
N/A
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #8187: Extend elastic DAG with
a binary tree, grid, star
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8187: Extend elastic DAG with a binary tree, grid, star
URL: https://github.com/apache/airflow/issues/8187#issuecomment-613029604
All children of task 1. This is needed to check if the tasks will be performed at the second iteration of the loop. Grid is needed to perform a stress test and at the same time not get lost in all dependencies. Binary tree will allow us to check cases more real to real situations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] alexandrecaze commented on issue #8187: Extend elastic
DAG with a binary tree, grid, star
Posted by GitBox <gi...@apache.org>.
alexandrecaze commented on issue #8187: Extend elastic DAG with a binary tree, grid, star
URL: https://github.com/apache/airflow/issues/8187#issuecomment-613026273
Would we imagine a second layer for the star ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] alexandrecaze commented on issue #8187: Extend elastic
DAG with a binary tree, grid, star
Posted by GitBox <gi...@apache.org>.
alexandrecaze commented on issue #8187: Extend elastic DAG with a binary tree, grid, star
URL: https://github.com/apache/airflow/issues/8187#issuecomment-613007188
I am interested in taking this issue @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] alexandrecaze commented on issue #8187: Extend elastic
DAG with a binary tree, grid, star
Posted by GitBox <gi...@apache.org>.
alexandrecaze commented on issue #8187: Extend elastic DAG with a binary tree, grid, star
URL: https://github.com/apache/airflow/issues/8187#issuecomment-613020676
@mik-laj What do you mean by grids and stars shaped DAGs ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #8187: Extend elastic DAG with
a binary tree, grid, star
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8187: Extend elastic DAG with a binary tree, grid, star
URL: https://github.com/apache/airflow/issues/8187#issuecomment-613023389
DAGs that look like the picture
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] alexandrecaze edited a comment on issue #8187: Extend
elastic DAG with a binary tree, grid, star
Posted by GitBox <gi...@apache.org>.
alexandrecaze edited a comment on issue #8187: Extend elastic DAG with a binary tree, grid, star
URL: https://github.com/apache/airflow/issues/8187#issuecomment-613026273
Would we have a second layer for the star ?
or all children of task 1 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services