You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/17 19:53:57 UTC
[GitHub] [airflow] potiuk opened a new pull request #18335: Added more automation on latest version
potiuk opened a new pull request #18335:
URL: https://github.com/apache/airflow/pull/18335
Some of the versions stored in docs have to be stored as
verbatim, rather than as |version| placeholder - they have to refer
to latest released airflow version.
This PR automates that.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711366514
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Small update. I changed it to use "latest released" version for tests rather than fixed - this way we will not have to maintain it - right after releasing a new version to PyPI the `docekr builds` will start using the new version - but they should just continue succeding in people's PR. if they will start failing, that means that we should fix it (so that should solve the problem @mik-laj mentioned).
This way we get an early warning system - if any of our "example extension Dockerfile" for any reason will stop working with latest released airflow, we will know it right after the new version has been released.
This is also nice as it will flag the cases in case we pushed a new release to PyPI but forgot to release the image. There is a slight race condition - if someone makes PR between PyPI and image release, the Dockerfile test might fail - but that is really rare and should not be a problem if the release manager has fast build-push cycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711335628
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
This will fail even before we release as for example when releasing `2.1.4`, this will use `2.1.3` instead as 2.1.4 won't be available until I push it.
I think this should match version in `setup.py` and just maintain a list of all the files that needs to be updated when the version is updated in setup.py. And if it contains "dev" in the version, it can then use `https://pypi.org/pypi/apache-airflow/json`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711366514
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Small update. I changed it to use "latest released" version for tests rather than fixed - this way we will not have to maintain it - right after releasing a new version to PyPI the `docekr builds` will start using the new version - but they should just continue building in people's PR. if they will start failing, that means that we should fix it (so that should solve the problem @mik-laj mentioned).
This way we get an early warning system - if any of our "example extension Dockerfile" for any reason will stop working with latest released airflow, we will know it right after the new version has been released.
This is also nice as it will flag the cases in case we pushed a new release to PyPI but forgot to release the image. There is a slight race - if someone makes PR between PyPI and image release, the Dockerfile test might fail - but that is really rare and should not be a problem if the release manager has fast build-push cycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711326501
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
Review comment:
Thanks for Python and not another Bash script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711341553
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Well. I deliberately used "latest-released" version. Those dockerfiles actually **SHOULD** use 2.1.3 before we release 2.1.4 - because those docker-files are actually used during tests and the dockerfiles are run through `docker build -f ..../Dockerfile`.
And when we release 2.1.4 - they should start using 2.1.4.
If we start using setup.py - that's the moment the tests will start to fail actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711343486
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
We will only change to `2.1.4` when we will release `2.1.4` inside the feature branch; hence my comment about checking for "dev"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711344705
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
it is not just Dockerfile but also constraints, hence I will still suggest using my suggestion :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711344007
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Check above comment:
>And if it contains "dev" in the version, it can then use https://pypi.org/pypi/apache-airflow/json
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711366514
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Small update. I changed it to use "latest released" version for tests rather than fixed - this way we will not have to maintain it - right after releasing a new version to PyPI the `docekr builds` will start using the new version - but they should just continue succeding in people's PR. if they will start failing, that means that we should fix it (so that should solve the problem @mik-laj mentioned).
This way we get an early warning system - if any of our "example extension Dockerfile" for any reason will stop working with latest released airflow, we will know it right after the new version has been released.
This is also nice as it will flag the cases in case we pushed a new release to PyPI but forgot to release the image. There is a slight race condition - if someone makes PR between PyPI and image release, the Dockerfile test might fail - but that is really rare and should not be a problem if the release manager has fast build-push cycle (and we can always ignore those failed tests knowing that they were caused by not-yet-pushed image)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711366514
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Small update. I changed it to use "latest released" version for tests rather than fixed - this way we will not have to maintain it - right after releasing a new version to PyPI the `docekr builds` will start using the new version - but they should just continue succeding in people's PR. if they will start failing, that means that we should fix it (so that should solve the problem @mik-laj mentioned).
This way we get an early warning system - if any of our "example extension Dockerfile" for any reason will stop working with latest released airflow, we will know it right after the new version has been released.
This is also nice as it will flag the cases in case we pushed a new release to PyPI but forgot to release the image. There is a slight race - if someone makes PR between PyPI and image release, the Dockerfile test might fail - but that is really rare and should not be a problem if the release manager has fast build-push cycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711354278
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
> That's right. Any idea how we can improve it ?
We can add a new step in the release process. We already have a similar one: https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md#update-airflowconfig_templatesconfigyml-file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #18335:
URL: https://github.com/apache/airflow/pull/18335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711358226
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Should be fixed now
* in the Dockerfiles we use the setup.py version
* but when we run tests, we dyanmically replace it with some released version (I fixed it to 2.1.3 - but we can irregularly update it too)
This way we test if the example dockerfiles are working fine (docker build succeds) even if the "source" version of airflow is not released yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711341553
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Well. I deliberately used "latest-released" version. Those dockerfiles actually **SHOULD** use 2.1.3 before we release 2.1.4 - because those docker-files are actually used during tests and the dockerfiles are run through `docker build -f ..../Dockerfile`.
If we start using setup.py - that's the moment the tests will start to fail actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711328354
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
Review comment:
It;s easier in python and I can do both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711343486
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
We will only change to `2.1.4` when we will release `2.1.4` inside the feature branch (at this point, this docs are not published); hence my comment about checking for "dev"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711343858
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
We could also use "latest" (but then the dockerfile example is not best - because users should use a specfiic version).
It does not **really** matter if we use current or latest in this case - both should work fine because the tests are just testing if the "Dockerfiles" are valid and build fine, not whether particular version of Airflow works.
Actually the best way is to add "setup.py" version when the image is committed, but use "latest released" when we run tests. Which I might well do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#issuecomment-922235107
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#issuecomment-922244795
🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711369495
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Yep https://github.com/apache/airflow/pull/18335/checks?check_run_id=3636921660 - the tests work nicely building image based on "2.1.3" even if we have "2.2.0.dev0" in the Dockerfiles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711366514
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
Small update. I changed it to use "latest released" version for tests rather than fixed - this way we will not have to maintain it - right after releasing a new version to PyPI the `docekr builds` will start using the new version - but they should just continue succeding in people's PR (because the docker builds should work fine for newly released versions). if they will start failing, that means that we should fix it (so that should solve the problem @mik-laj mentioned).
This way we get an early warning system - if any of our "example extension Dockerfile" for any reason will stop working with latest released airflow, we will know it right after the new version has been released.
This is also nice as it will flag the cases in case we pushed a new release to PyPI but forgot to release the image. There is a slight race condition - if someone makes PR between PyPI and image release, the Dockerfile test might fail - but that is really rare and should not be a problem if the release manager has fast build-push cycle (and we can always ignore those failed tests knowing that they were caused by not-yet-pushed image)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711325684
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
This is dangerous as it means that after a new version is released, all PRs have to do a rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711328685
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
That's right. Any idea how we can improve it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18335: Added more automation on latest version
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18335:
URL: https://github.com/apache/airflow/pull/18335#discussion_r711328685
##########
File path: scripts/ci/pre_commit/pre_commit_update_versions.py
##########
@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import glob
+import json
+import os
+import re
+from os.path import abspath, dirname, join
+from re import Pattern
+
+import httpx
+
+AIRFLOW_SOURCES_DIR = abspath(join(dirname(__file__), os.pardir, os.pardir, os.pardir))
+
+
+def update_version(pattern: Pattern, version: str, file_path: str):
+ print(f"Replacing {pattern} to {version} in {file_path}")
+ with open(file_path, "r+") as f:
+ file_contents = f.read()
+ lines = file_contents.splitlines(keepends=True)
+ for i in range(0, len(lines)):
+ lines[i] = re.sub(pattern, fr'\g<1>{version}\g<2>', lines[i])
+ file_contents = "".join(lines)
+ f.seek(0)
+ f.truncate()
+ f.write(file_contents)
+
+
+REPLACEMENTS = {
+ r'(FROM apache/airflow:).*($)': "docs/docker-stack/docker-examples/extending/*/Dockerfile",
+ r'(apache/airflow:)[^-]*(\-)': "docs/docker-stack/entrypoint.rst",
+ r'(/constraints-)[^-]*(/constraints)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+ r'(AIRFLOW_VERSION=")[^"]*(" \\)': "docs/docker-stack/docker-examples/"
+ "restricted/restricted_environments.sh",
+}
+
+if __name__ == '__main__':
+ airflow_pypi_info = json.loads(httpx.get("https://pypi.org/pypi/apache-airflow/json").text)
Review comment:
That's right any idea how we can improve it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org