You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/13 09:01:34 UTC
[GitHub] [airflow] potiuk opened a new issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
potiuk opened a new issue #20251:
URL: https://github.com/apache/airflow/issues/20251
At entry of the Breeze command we verify if there iss enouggh CPU/memory/disk space and print (coloured) information if the resources are not enough.
We should replicate that in Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1002974092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20251:
URL: https://github.com/apache/airflow/issues/20251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1003406916
Yeah - we want separate "tools" for the "in_container" - those should be only scripts that are available inside the container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1010219584
Closes with #20763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-995905389
@potiuk I would like to work on this. In Breeze installation, I will take as consideration: https://github.com/apache/airflow/blob/main/BREEZE.rst#resources-required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1003392464
We already have "tools" directory and some files. We have this structure:
| - scripts
| - init_container
| - run_resource_check
| - tools
| - list-integrations.py
"tools" is something more general, I think it can be left out, as is. Unless we have another "tools" folder just for init_container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1003392464
We already have a "tools" directory and some files. We have this structure:
| - scripts
----| - init_container
--------| - run_resource_check
----| - tools
--------| - list-integrations.py
"tools" is something more general, I think it can be left out, as is. Unless we have another "tools" folder just for init_container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1003406916
Yeah - we want separate "tools" for the "in_container" - those should be only scripts that are available (as tools) inside the container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn removed a comment on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
edithturn removed a comment on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1010219584
Closes with #20763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20251:
URL: https://github.com/apache/airflow/issues/20251
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1002858727
Ideal resources available to check:
* Memory available 16G.
* CPUs available 12.
* Disk available 41G.
Breeze has a line where call `_all_libs.sh` and `run_resource_check.sh` is part of one of these scripts.
How could we structure run_resource_check.py? this script is inside: scripts/in_container, we will change it?
I could say it is not part of CI, so we can consider keeping it in the same directory as an in_container of Breeze. @potiuk do you have a better idea for a good place for this script?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1002974092
I think we should rewrite this check in python entirely:
* make a python script equivalent to run_resource_check.sh (run_resource_check.py ) - but it should be run inside breeze
* replace calling of the run_resource_check.sh -> run_resource_check.py
* remove the .sh script
This way the run_resouce_check.py will be used also in the old "breeze". It should stay in "in_container" folder.
UPDATE: Maybe - for the "nicer" organization ot it, it should be added to 'scripts/in_container/tools` folder ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1002974092
I think we should rewrite this check in python entirely:
* make a python script equivalent to run_resource_check.sh (run_resource_check.py ) - but it should be run inside breeze
* replace calling of the run_resource_check.sh -> run_resource_py
* remove the .sh script
This way the run_resouce_check.py will be used also in the old "breeze".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #20251: Breeze: Verify if there are enough resources avaiilable in Breeze
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #20251:
URL: https://github.com/apache/airflow/issues/20251#issuecomment-1016432006
Had to revert merged PR as it was failing all jobs, so re-opening this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org