You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/11/20 18:13:52 UTC
[PR] Improve ownership fixing for Breeze [airflow]
potiuk opened a new pull request, #35759:
URL: https://github.com/apache/airflow/pull/35759
When we run airflow package preparation in docker containers and asset compilation, we use docker containers - similarly as breeze CI image, those constainers use root user in container. This means that the files created by those containers are owned by root on Linux - which means that we should change ownership of these files after they were generated.
Used the opportunity to rewrtie "fix_ownership" part to Python and remove bash scripts used for it so far.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399605295
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
```suggestion
follow_symlinks = True
if file.is_symlink() and file.lstat().st_uid == root_uid:
follow_symlinks = False
os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=follow_symlinks)
count_files += 1
if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
print(f"Changed ownership of {file}")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399613884
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
Nope. It's good we want to change ownership for both - symlink and file it points at. The first if changes the ownership for the symlink (when it is one). The next if changes the ownership for either the file (if it is the file) or for the file symlink points at (if it's a symlink) by default `stat()` and `chown` follow symlinks so they only change the ownership of target file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399638492
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
Sorry...I just assumed it was a if-else block. My bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35759:
URL: https://github.com/apache/airflow/pull/35759#issuecomment-1819578579
World is a bit better place with every 150 or so less lines of bash script :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399605295
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
```suggestion
# Also change ownership of symlink itself (by default it will only change
# ownership of the file it points to)
follow_symlinks = True
if file.is_symlink() and file.lstat().st_uid == root_uid:
follow_symlinks = False
os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=follow_symlinks)
count_files += 1
if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
print(f"Changed ownership of {file}")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399638492
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
Sorry...I just overlooked and assumed it was a if-else block and it's not. My bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399605295
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
```suggestion
follow_symlinks = True
if file.is_symlink() and file.lstat().st_uid == root_uid:
follow_symlinks = False
os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=follow_symlinks)
count_files += 1
if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
print(f"Changed ownership of {file}")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399621198
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
I updated comments to explain it a bit better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35759:
URL: https://github.com/apache/airflow/pull/35759#issuecomment-1819635962
I also removed "fix ownership" step from CI as I moved the ownership fixing as "always happening" after anything is run in docker on linux - so we do nothave to run separate steps for those..
Captured the change in a new ADR record - as I noticed earlier decision was about using TRAP in bash - but TRAPs were removed from our bash scripts some time ago as they behaved inconsistetntly/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399606213
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
minor suggestion. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #35759:
URL: https://github.com/apache/airflow/pull/35759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399643942
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
Well. It was misleading - that's why it's good you noticed it - now I think the comment is more accurate - because it is quite surprising behviour (it was certainly quite surprising for me when I tested it today and notice a bunch of symlinks that continue to have root ownership even if I thought I changed them all :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve ownership fixing for Breeze [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #35759:
URL: https://github.com/apache/airflow/pull/35759#discussion_r1399657023
##########
scripts/in_container/run_fix_ownership.py:
##########
@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import pwd
+import sys
+from pathlib import Path
+
+HOST_OS = os.environ.get("HOST_OS", "")
+DOCKER_IS_ROOTLESS = os.environ.get("DOCKER_IS_ROOTLESS", "false") == "true"
+
+
+def change_ownership_of_files(path: Path) -> None:
+ host_user_id = os.environ.get("HOST_USER_ID", "")
+ host_group_id = os.environ.get("HOST_GROUP_ID", "")
+ if host_user_id == "" or host_group_id == "":
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"are not set: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ if not host_user_id.isnumeric() or not host_group_id.isnumeric():
+ print(
+ f"ERROR: HOST_USER_ID or HOST_GROUP_ID environment variables "
+ f"should be numeric: {host_user_id}:{host_group_id}"
+ )
+ sys.exit(1)
+ count_files = 0
+ root_uid = pwd.getpwnam("root").pw_uid
+ print("Attempting to see if there are files that need to be changed to host user ownership")
+ for file in path.rglob("*"):
+ try:
+ if file.is_symlink() and file.lstat().st_uid == root_uid:
+ # Also change ownership of symlink itself (by default it will only change
+ # ownership of the file it points to)
+ os.chown(file, int(host_user_id), int(host_group_id), follow_symlinks=False)
+ count_files += 1
+ if file.stat().st_uid == root_uid:
+ os.chown(file, int(host_user_id), int(host_group_id))
+ count_files += 1
+ if os.environ.get("VERBOSE_COMMANDS", "false") == "true":
+ print(f"Changed ownership of {file}")
Review Comment:
Ya, it is surprising now that it's pointed out. I wasn't aware of it either. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org