You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "MkSafavi (via GitHub)" <gi...@apache.org> on 2023/03/07 16:12:06 UTC

[GitHub] [airflow] MkSafavi opened a new issue, #29959: expand dynamic mapped tasks in batches

MkSafavi opened a new issue, #29959:
URL: https://github.com/apache/airflow/issues/29959

   ### Description
   
   expanding tasks in batches to allow mapped tasks spawn more than 1024 processes.
   
   ### Use case/motivation
   
   Maximum length of a list is limited to 1024 by `max_map_length (AIRFLOW__CORE__MAX_MAP_LENGTH)`.
   during scheduling of the new tasks, an UPDATE query is ran that tries to set all the new tasks at once.  Increasing `max_map_length` more than 4K makes airflow scheduler completely unresponsive.
   
   Also, Postgres throws `stack depth limit exceeded` error which can be fixed by update to a newer version and setting `max_stack_depth`. But it doesn't really matter because airflow scheduler freezes up.
    
   As a workaround, I split the dag runs into subdag runs which works but it would be much nicer if we didn't have to worry about exceeding  `max_map_length`. 
   
   ### Related issues
   
   It was discussed here:
   [Increasing 'max_map_length' leads to SQL 'max_stack_depth' error with 5000 dags to be spawned #28478](https://github.com/apache/airflow/discussions/28478)
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #29959: expand dynamic mapped tasks in batches

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29959: expand dynamic mapped tasks in batches
URL: https://github.com/apache/airflow/issues/29959


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #29959: expand dynamic mapped tasks in batches

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29959:
URL: https://github.com/apache/airflow/issues/29959#issuecomment-1458438141

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hussein-awala commented on issue #29959: expand dynamic mapped tasks in batches

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #29959:
URL: https://github.com/apache/airflow/issues/29959#issuecomment-1490542661

   @dzhigimont yes I'm working on other tasks and didn't have time for this one, I assigned you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #29959: expand dynamic mapped tasks in batches

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29959:
URL: https://github.com/apache/airflow/issues/29959#issuecomment-1459068467

   Yep. That would be a good thing to have.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dzhigimont commented on issue #29959: expand dynamic mapped tasks in batches

Posted by "dzhigimont (via GitHub)" <gi...@apache.org>.
dzhigimont commented on issue #29959:
URL: https://github.com/apache/airflow/issues/29959#issuecomment-1490537303

   @hussein-awala No offense, but I didn't see any activities from your side with the issue, so I decided to start working on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org