You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/08 05:57:12 UTC
[GitHub] [airflow] edithturn opened a new pull request #20763: [WIP] Verify enough resources for breeze
edithturn opened a new pull request #20763:
URL: https://github.com/apache/airflow/pull/20763
Check resources before install docker: Close issue: #20251
* Memory available
* CPUs available
* Disk available
Using **psutil** to calculate available resources in kilobytes, get_size function will convert in Gb.
I created a dictionary to add resources available and resources needed.
TODO:
* Verbose, colors on the console
* Check if script _in_container_script_init.sh
* Check if for disk this is taking the mounted on root pat "/"
* Check if _docker_engine_resources.sh is calling successful to run_resource_check.py
@potiuk I will appreciate your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786065823
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ MINIMUM_ALLOWED_MEMORY = 4
+ MINIMUM_ALLOWED_CPUS = 2
+ MINIMUM_ALLOWED_DISK = 20
+ print("\nChecking resources.\n")
+
+ # Memory current available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+
+ # Cpus current available
+ cpus_available = psutil.cpu_count(logical=True)
+
+ # Disk current available
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+
+ resources: Dict[str, Resource] = {
+ 'Memory': Resource(current=mem_available, minimumAllowed=MINIMUM_ALLOWED_MEMORY),
+ 'Cpus': Resource(current=cpus_available, minimumAllowed=MINIMUM_ALLOWED_CPUS),
+ 'Disk': Resource(current=disk_available, minimumAllowed=MINIMUM_ALLOWED_DISK),
+ }
+ return resources
+
+
+def resoure_validate():
+
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ for resource, capacity in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
Review comment:
I do not think we need the `resource == "..."` her e at all . This if could lbe reduced to simpy `if capacity.current < capacity.minimumAllowed:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016448859
It was not at all obvious, really I think the reason is this:
https://psutil.readthedocs.io/en/latest/#disks
| disk_partitions(all=False)
| If all parameter is False it tries to distinguish and return physical devices only (e.g. hard disks, cd-rom drives, USB keys) and ignore all others (e.g. pseudo, memory, duplicate, inaccessible filesystems)
Which precisely what happened.
I think you should list all partitions, and check the size of the partition that has "/" as mountpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011417552
Just rebase and I am ready to merge it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786340660
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
It's for readability. It's much clearer to see that it's 1024 ** 3 (you could also write it this way but multiplication is likely faster than power of).
Simple question: Do you know by heart that 1073741824 is 1024 ^ 3 when you look at it? Can you verify it when code reviewing without using calculator? How about 1024*1024*1024.
For me - I can immediately see that the second one is 1024 ** 3 as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786384657
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
Yes, this is definitely more understandable.
@potiuk some points I checked here:
**Point 01**: gsutil returns the size of resources in bytes of memory(available) and disk(free), and the function get_size takes bytes and returns Gb.
` 1 Gigabytes = 1024*1024*1024 = 1073741824 bytes`. Here is where I am taking just the integer value:
`value_gb = bytes // factor` if this is 4.605457306 I will take 4 because rounding to 5 is not convenient, it would be to complete the missing memory
**Point 02**: One of the things I realized in the previous code **run_resource_check.sh** is that in "human_readable_memory" we was storing the total memory in our system instead of Total Memory Available. Now I am using: `mem_available = get_size(svmem.available)`
**This is to my own understanding** :)
MemFree != MemAvailable. We are using MemAvailable which is the case.
- MemFree: The amount of physical RAM, in kilobytes, left unused by the system.
- MemAvailable: An estimate of how much memory is available for starting new applications, without swapping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782531535
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('Cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('Disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ resources.setdefault("Memory", []).append(4)
+ resources.setdefault("Cpus", []).append(2)
+ resources.setdefault("Disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
+ resource == "Memory"
+ and available[0] < 4
+ or resource == "Cpus"
+ and available[0] < 2
+ or resource == "Disk"
+ and available[0] < 20
+ ):
+ console.print(f"[yellow]WARNING!!!: Not enough {resource} available for Docker.")
+ print(f"At least {available[1]}{check} of {resource} required. You have {available[0]}{check}\n")
+ warning_resources = True
+ else:
+ console.print(f" * {resource} available {available[0]}{check}. [green]OK.\n")
Review comment:
```suggestion
console.print(f" * {resource} available {available[0]}{check}. [green]OK.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782253113
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -93,8 +95,16 @@ Breeze, is not globally accessible until your PATH is updated. Add <USER FOLDER>
pipx ensurepath
```
+<<<<<<< HEAD
Review comment:
I got it! It means I didn't execute pre-commit for this, if I had done this it would catch the errors before pushing them.
I should remember as a rule, run pre-commit always
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r780678128
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -93,8 +95,16 @@ Breeze, is not globally accessible until your PATH is updated. Add <USER FOLDER>
pipx ensurepath
```
+<<<<<<< HEAD
Review comment:
Some merge artifacts
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+import psutil
+
+os.system("sh _in_container_script_init.sh")
Review comment:
I think we do not really need to source it - it was mainly needed for the bash script to source some common functions but we can simply drop it.
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+import psutil
+
+os.system("sh _in_container_script_init.sh")
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("Checking resources.")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+
+ resources = resoure_check()
+
+ resources.setdefault("memory", []).append(4)
+ resources.setdefault("cpus", []).append(2)
+ resources.setdefault("disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ if (
+ resource == "memory"
+ and available[0] < 4
+ or resource == "cpus"
+ and available[0] < 2
+ or resource == "disk"
+ and available[0] < 20
+ ):
+ print(f"== {resource} ==")
+ print(f"WARNING!!!: Not enough {resource} available for Docker.")
+ print(f"At least {available[1]}GB of {resource} required. You have {available[0]}\n")
Review comment:
We need to add colors/rich to that :)
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -32,6 +32,8 @@
alt="Airflow Breeze - Development and Test Environment for Apache Airflow">
</div>
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
Review comment:
Copy * Paste mistake :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011406936
@potiuk this si how this script is showing the information with the job:
However, there is an error that I do not understand yet, related to:
The BREEZE variable is not set. Defaulting to a blank string.
ERROR: for airflow "Container" 00543b681b2e "is unhealthy.
Encountered errors while bringing up the project. "
In the action: https://github.com/apache/airflow/runs/4793789902?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011476143
I rebased it :fearful:
, I am not sure if this is good, because I still having BREEZE.md and the rebase --continue didn't ask me to squash.
I need to master rebase, I am just following these instructions: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#how-to-rebase-pr
Please Jarek, let me know if something wrong here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016448859
It was not at all obvious, really I think the reason is this:
https://psutil.readthedocs.io/en/latest/#disks
> disk_partitions(all=False)
> If all parameter is False it tries to distinguish and return physical devices only (e.g. hard disks, cd-rom drives, USB keys) and ignore all others (e.g. pseudo, memory, duplicate, inaccessible filesystems)
Which precisely happened.
I think you should list all partitions, and check the size of the partition that has "/" as mountpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016390336
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782442146
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+import psutil
+
+os.system("sh _in_container_script_init.sh")
Review comment:
Not needed - mostly of the time (or we already use stuff instead - like "run_command"). Anything that will be needed will end up as a shared Python code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786334200
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -0,0 +1,108 @@
+<!--
Review comment:
Thank you, I did this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016486598
BTW. The root cause is that In our self-hosted runners, Docker is mounted on "tmpfs" (big ram-disk) to make things faster. So the 'smart' disk_partitions() did not return any partition :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016503868
> Hey @edithturn - looks like the "self-hosted" runners (with a different filesystem - in memory) had some problems so you will need to re-create the PR and we should test it there as well :(
@jarek where did you get these logs? could you point o me, thanks in advance.
I didn't find it in the workflow after it was merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r783163958
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('Cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('Disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ resources.setdefault("Memory", []).append(4)
+ resources.setdefault("Cpus", []).append(2)
+ resources.setdefault("Disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
+ resource == "Memory"
+ and available[0] < 4
+ or resource == "Cpus"
+ and available[0] < 2
+ or resource == "Disk"
+ and available[0] < 20
+ ):
+ console.print(f"[yellow]WARNING!!!: Not enough {resource} available for Docker.")
+ print(f"At least {available[1]}{check} of {resource} required. You have {available[0]}{check}\n")
+ warning_resources = True
+ else:
+ console.print(f" * {resource} available {available[0]}{check}. [green]OK.\n")
Review comment:
Done :), definitely looks great without space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r783312075
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
Review comment:
I did it @potiuk, I tried to give to the variables an appropriate name.
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
Review comment:
I did it @potiuk, I tried to give the variables an appropriate name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r783189748
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('Cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('Disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ resources.setdefault("Memory", []).append(4)
+ resources.setdefault("Cpus", []).append(2)
+ resources.setdefault("Disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
+ resource == "Memory"
+ and available[0] < 4
Review comment:
Should I make the same with "Memory", "Cpus" and "Disk", they are used in both functions, the global variable will be good here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r780771334
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -32,6 +32,8 @@
alt="Airflow Breeze - Development and Test Environment for Apache Airflow">
</div>
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
Review comment:
Happens sometimes, when there som whitespace differences etc. I tend to get similar artifacts from time to time - repeated lines. Usually there some results of resolving conflicts slightly differently in two branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011476143
I rebased it :fearful:
I am not sure if this is good, because I still have BREEZE.md and the rebase --continue didn't ask me to squash.
I need to master rebase, I am just following these instructions: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#how-to-rebase-pr
Please @potiuk, let me know if something wrong here! :muscle:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782261702
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+import psutil
+
+os.system("sh _in_container_script_init.sh")
Review comment:
Will these common functions ever get rewritten in python? or they are no longer necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1010362731
@potiuk I corrected your observations, I will appreciate your feedback when you have free time :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782537077
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
Review comment:
```suggestion
resources= {
'Memory': [mem_available, MAX_MEMORY]
'Cpus' : ....
```
This would be far simpler and shorter (no need to use `setdefault` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782262027
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+import psutil
+
+os.system("sh _in_container_script_init.sh")
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("Checking resources.")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+
+ resources = resoure_check()
+
+ resources.setdefault("memory", []).append(4)
+ resources.setdefault("cpus", []).append(2)
+ resources.setdefault("disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ if (
+ resource == "memory"
+ and available[0] < 4
+ or resource == "cpus"
+ and available[0] < 2
+ or resource == "disk"
+ and available[0] < 20
+ ):
+ print(f"== {resource} ==")
+ print(f"WARNING!!!: Not enough {resource} available for Docker.")
+ print(f"At least {available[1]}GB of {resource} required. You have {available[0]}\n")
Review comment:
yeah, I am working in this now
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+
+import psutil
+
+os.system("sh _in_container_script_init.sh")
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("Checking resources.")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+
+ resources = resoure_check()
+
+ resources.setdefault("memory", []).append(4)
+ resources.setdefault("cpus", []).append(2)
+ resources.setdefault("disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ if (
+ resource == "memory"
+ and available[0] < 4
+ or resource == "cpus"
+ and available[0] < 2
+ or resource == "disk"
+ and available[0] < 20
+ ):
+ print(f"== {resource} ==")
+ print(f"WARNING!!!: Not enough {resource} available for Docker.")
+ print(f"At least {available[1]}GB of {resource} required. You have {available[0]}\n")
Review comment:
yeah, I am working on this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782442354
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -93,8 +95,16 @@ Breeze, is not globally accessible until your PATH is updated. Add <USER FOLDER>
pipx ensurepath
```
+<<<<<<< HEAD
Review comment:
Oh yeah. Just `pre-commit install` and it will run always when you commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011406936
@potiuk this is how this script **run_resource_check.py** is showing the information in the job:
However, there is an error that I do not understand yet, related to:
The BREEZE variable is not set. Defaulting to a blank string.
ERROR: for airflow "Container" 00543b681b2e "is unhealthy.
Encountered errors while bringing up the project. "
In the action: https://github.com/apache/airflow/runs/4793789902?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1014932713
@potiuk I fixed. Now "This branch is 3 commits ahead of apache:main."
Thank you for your patience Jarek.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016473439
I got it @potiuk I assumed that the first one in the array will always be the mounted point: Like in my local machine. I didn't identify it when it ran as a workflow, maybe because there were other issues. (I will check the jobs right now)
This was the behavior on my machine:
```
>>> print(partitions)
[sdiskpart(device='/dev/sda3', mountpoint='/', fstype='ext4', opts='rw,r
```
I should have taken this directly :` partition_usage = psutil.disk_usage('/')`
I will test it again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786384657
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
Yes, this is definitely more understandable.
@potiuk some points I checked here:
**Point 01**: gsutil returns the size of resources in bytes of memory(available) and disk(free), and the function get_size takes bytes and returns Gb.
` 1 Gigabytes = 1024*1024*1024 = 1073741824 bytes`. Here is where I am taking just the integer value:
`value_gb = bytes // factor` if this is 4.605457306 I will take 4 because rounding to 5 is not convenient, it would be to complete the missing memory
**Point 02**: One of the things I realized in the previous code **run_resource_check.sh** is that in "human_readable_memory" we was storing the total memory in our system instead of Total Memory Available. Now I am using: `mem_available = get_size(svmem.available)`
**This is to my own understanding** :)
* MemFree != MemAvailable. We are using MemAvailable which is the case.
- MemFree: The amount of physical RAM, in kilobytes, left unused by the system.
- MemAvailable: An estimate of how much memory is available for starting new applications, without swapping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016437106
Sorry to revert rather than fix -- but everything was broken on main as a result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r780737492
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -32,6 +32,8 @@
alt="Airflow Breeze - Development and Test Environment for Apache Airflow">
</div>
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
Review comment:
wow, why I could be having conflicts from my last PR that already was merged (Breeze for windows documentation)? 🙄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #20763:
URL: https://github.com/apache/airflow/pull/20763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016434392
Hey @edithturn - looks like the "self-hosted" runners (with a different filesystem - in memory) had some problems so you will need to re-create the PR and we should test it there as well :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016473439
I got it @potiuk I assumed that the first one in the array will always be the mounted point: Like in my local machine. I didn't identify it when it ran as a workflow, maybe because there were other issues. (I will check the jobs right now)
```
>>> print(partitions)
[sdiskpart(device='/dev/sda3', mountpoint='/', fstype='ext4', opts='rw,r
```
I should have taken this directly :` partition_usage = psutil.disk_usage('/')`
I will test it again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1014649702
I think there is still something wrong:
When you look at your branch (by clicking on the branch name at the top of your PR): https://github.com/edithturn/airflow/tree/verify-enough-resources-for-breeze :
| This branch is 19 commits ahead, 30 commits behind apache:main.
What you really want is that your branch is "19 commits ahead" and it should not be behind `main`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1014658866
`git fetch apache && git reset --hard apache/main` should reset your local main to the "apache" main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782532063
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('Cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('Disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ resources.setdefault("Memory", []).append(4)
+ resources.setdefault("Cpus", []).append(2)
+ resources.setdefault("Disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
+ resource == "Memory"
+ and available[0] < 4
+ or resource == "Cpus"
+ and available[0] < 2
+ or resource == "Disk"
+ and available[0] < 20
+ ):
+ console.print(f"[yellow]WARNING!!!: Not enough {resource} available for Docker.")
+ print(f"At least {available[1]}{check} of {resource} required. You have {available[0]}{check}\n")
+ warning_resources = True
+ else:
+ console.print(f" * {resource} available {available[0]}{check}. [green]OK.\n")
Review comment:
NIT. It will look nicer without separate line:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782535098
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('Cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('Disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ resources.setdefault("Memory", []).append(4)
+ resources.setdefault("Cpus", []).append(2)
+ resources.setdefault("Disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
+ resource == "Memory"
+ and available[0] < 4
Review comment:
The numbers (4,2,20) are repeated here and above - they should be extracted as constants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011417214
> The BREEZE variable is not set. Defaulting to a blank string.
> ERROR: for airflow "Container" 00543b681b2e "is unhealthy.
No no , that's fine. The tests failed randomly :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011476143
I rebased it
, I am not sure if this is good, because I still having BREEZE.md and the rebase --continue didn't ask me to squash.
I need to master rebase, I am just following these instructions: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#how-to-rebase-pr
Please Jarek, let me know if something wrong here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011476143
I rebased it :fearful:
I am not sure if this is good, because I still have BREEZE.md and the rebase --continue didn't ask me to squash.
I need to master rebase, I am just following these instructions: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#how-to-rebase-pr
Please Jarek, let me know if something wrong here! :muscle:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016392774
Fantastifc one @edithturn ! The "available" choice is good idea! I just tested it and ii workes really nicely so I am merging it now.
70 less lines of BASH ! Yay :tada: :tada: :tada: :tada: !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1014654822
@potiuk my main, right now have the changes of my branch "check-free-space-python" to check if build-image was working :) .
Should I reset now, right? To work normally and fix this branch.
> git push -f edithturn main -> to reset it back to the "real" main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786384657
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
Yes, this is definitely more understandable.
@potiuk some points I checked here:
**Point 01**: gsutil returns the size of resources in bytes of memory(available) and disk(free), and the function get_size takes bytes and returns Gb.
` 1 Gigabytes = 1024*1024*1024 = 1073741824 bytes`. Here is where I am taking just the integer value:
`value_gb = bytes // factor` if this is 4.605457306 I will take 4 because rounding to 5 is not convenient, it would be to complete the missing memory, which is wrong.
**Point 02**: One of the things I realized in the previous code **run_resource_check.sh** is that in "human_readable_memory" we were storing the total memory in our system instead of Total Memory Available. Now I am using: `mem_available = get_size(svmem.available)`
**This is to my own understanding** :)
MemFree != MemAvailable. We are using MemAvailable which is the case.
- MemFree: The amount of physical RAM, in kilobytes, left unused by the system.
- MemAvailable: An estimate of how much memory is available for starting new applications, without swapping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016448859
It was not at all obvious, really I think the reason is this:
https://psutil.readthedocs.io/en/latest/#disks
> disk_partitions(all=False)
> If all parameter is False it tries to distinguish and return physical devices only (e.g. hard disks, cd-rom drives, USB keys) and ignore all others (e.g. pseudo, memory, duplicate, inaccessible filesystems)
Which precisely what happened.
I think you should list all partitions, and check the size of the partition that has "/" as mountpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016445860
Thank you so much @ashb and @potiuk, I will check the "filesystem - in memory" in our runners, and test based on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016444539
Yeah. Good call! It was for committers only but that's "enough"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786340660
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
It's for readability. It's much clearer to see that it's 1024 ** 3 (you could also write it this way but multiplication is likely faster than power of).
Simple question: Do you know by heart that 1073741824 is (as it should be) 1024 ^ 3 when you look at it? Can you verify it when code reviewing without using calculator? How about 1024*1024*1024.
For me - I can immediately see that the second one is 1024 ** 3 as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn edited a comment on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1011476143
I rebased it :fearful: I am not sure if this is good, because I still have BREEZE.md and the rebase --continue didn't ask me to squash.
I need to master rebase, I am just following these instructions: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#how-to-rebase-pr
Please Jarek, let me know if something wrong here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782540630
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
Review comment:
Actually it would be better to use a dataClass instead of array:
```
@dataclass
class Resource:
current: float
max: float
```
and then
```
resources: Dict[str, Resource] = {
'Memory': Resource(cuurent=mem_available, maxMAX_MEMORY),
'Cpus': ...
}
```
Then you could refer to it via:
```
available.current, available.max
```
rather than
```
available[0], available[1]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r782540630
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
Review comment:
Actually it would be better to use a dataClass instead of array:
```
@dataclass
class Resource:
current: float
max: float
```
and then
```
resources: Dict[str, Resource] = {
'Memory': Resource(current=mem_available, max=MAX_MEMORY),
'Cpus': ...
}
```
Then you could refer to it via:
```
available.current, available.max
```
rather than
```
available[0], available[1]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1014571088
Let me know about this if I need to rebase it again :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786065583
##########
File path: dev/breeze/doc/BREEZE.md
##########
@@ -0,0 +1,108 @@
+<!--
Review comment:
I think this one should be removed completely ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1014649702
I think there is still something wrong:
When you look at your branch (by clicking on the branch name at the top of your PR): https://github.com/edithturn/airflow/tree/verify-enough-resources-for-breeze :
| This branch is 19 commits ahead, 30 commits behind apache:main.
What you really want is that your branch is "19 commits ahead"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016391183
The PROD image fail is because of the problem with GPG key we fixed in `main` already. Ready to merge :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016485076
I will also push it as well when you finish to test "committer" workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016604747
You can see link in the PR of Ash https://github.com/apache/airflow/pull/20948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016400280
Woho! Thank youuuuuu!!! 🎊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#issuecomment-1016503868
> Hey @edithturn - looks like the "self-hosted" runners (with a different filesystem - in memory) had some problems so you will need to re-create the PR and we should test it there as well :(
@jarek where did you get these logs? could you point o me, thanks in advance.
I didn't find it in the workflow after it was merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786340660
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
It's for readability. It's much clearer to see that it's 1024 ** 3 (you could also write it this way but multiplication is likely faster than power of).
Simple question: Do you know by heart that 1073741824 is (as it should be) 1024 ^ 3 when you look at it? Can you verify it when code reviewing without using calculator? How about `1024*1024*1024` ?
For me - I can immediately see that the second one is 1024 ** 3 as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786071307
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ MINIMUM_ALLOWED_MEMORY = 4
+ MINIMUM_ALLOWED_CPUS = 2
+ MINIMUM_ALLOWED_DISK = 20
+ print("\nChecking resources.\n")
+
+ # Memory current available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+
+ # Cpus current available
+ cpus_available = psutil.cpu_count(logical=True)
+
+ # Disk current available
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+
+ resources: Dict[str, Resource] = {
+ 'Memory': Resource(current=mem_available, minimumAllowed=MINIMUM_ALLOWED_MEMORY),
+ 'Cpus': Resource(current=cpus_available, minimumAllowed=MINIMUM_ALLOWED_CPUS),
+ 'Disk': Resource(current=disk_available, minimumAllowed=MINIMUM_ALLOWED_DISK),
+ }
+ return resources
+
+
+def resoure_validate():
+
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ for resource, capacity in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
Review comment:
I totally agree! Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786334446
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
I was wondering why don't write directly: 1073741824 instead of 1024*1024*1024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r786090502
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from dataclasses import dataclass
+from typing import Dict
+
+import psutil
+from rich.console import Console
+
+
+@dataclass
+class Resource:
+ current: int
+ minimumAllowed: int
+
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
Review comment:
This will be much more accurate and easier to verify:
```suggestion
factor = 1024*1024*1024
```
Also the get_size is in `bytes` not kilobytes ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r783312075
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
Review comment:
I changed for dataclass @potiuk, I tried to give the variables an appropriate name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] edithturn commented on a change in pull request #20763: [WIP] Verify enough resources for breeze
Posted by GitBox <gi...@apache.org>.
edithturn commented on a change in pull request #20763:
URL: https://github.com/apache/airflow/pull/20763#discussion_r783163958
##########
File path: scripts/in_container/run_resource_check.py
##########
@@ -0,0 +1,102 @@
+#!/usr/bin/env python
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import psutil
+from rich.console import Console
+
+console = Console(force_terminal=True, color_system="standard", width=180)
+
+
+def get_size(kilobytes):
+ """
+ Convert kilobytes into gigabytes
+ 1 Gigabytes = 1,048,576 Kb
+ """
+ factor = 1000000000
+ value_gb = kilobytes // factor
+ return value_gb
+
+
+def resoure_check():
+ """
+ Use gsutil to get resources in kylobytes: memory, cpus and disk
+ """
+ resources = {}
+ print("\nChecking resources.\n")
+
+ # Memory available
+ svmem = psutil.virtual_memory()
+ mem_available = get_size(svmem.total)
+ resources.setdefault('Memory', []).append(mem_available)
+
+ # Cpus available
+ cpus_available = psutil.cpu_count(logical=True)
+ resources.setdefault('Cpus', []).append(cpus_available)
+
+ # Disk: Get all disk partitions /
+ partitions = psutil.disk_partitions()
+ partition_usage = psutil.disk_usage(partitions[0].mountpoint)
+ disk_available = get_size(partition_usage.free)
+ resources.setdefault('Disk', []).append(disk_available)
+
+ return resources
+
+
+def resoure_validate():
+ resources = resoure_check()
+ warning_resources = False
+ check = "OK"
+
+ resources.setdefault("Memory", []).append(4)
+ resources.setdefault("Cpus", []).append(2)
+ resources.setdefault("Disk", []).append(20)
+
+ for resource, available in resources.items():
+
+ check = '' if resource == "Cpus" else 'GB'
+
+ if (
+ resource == "Memory"
+ and available[0] < 4
+ or resource == "Cpus"
+ and available[0] < 2
+ or resource == "Disk"
+ and available[0] < 20
+ ):
+ console.print(f"[yellow]WARNING!!!: Not enough {resource} available for Docker.")
+ print(f"At least {available[1]}{check} of {resource} required. You have {available[0]}{check}\n")
+ warning_resources = True
+ else:
+ console.print(f" * {resource} available {available[0]}{check}. [green]OK.\n")
Review comment:
Done :), definitely looks great without an end line at the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org