You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/25 03:13:07 UTC
[GitHub] [airflow] mik-laj opened a new pull request #19819: Rewrite image building tests to Python
mik-laj opened a new pull request #19819:
URL: https://github.com/apache/airflow/pull/19819
Follow up: https://github.com/apache/airflow/pull/19737
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#issuecomment-980024213
@uranusjr - does it look good for you too ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19819:
URL: https://github.com/apache/airflow/pull/19819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r757438642
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
- "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
+ "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}"
Review comment:
I also updated other versions - If you want you can remove my fixup and I can make a separate PR out of that - BTW I just run all the "customize" image scripts and they are working fine .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r757375621
##########
File path: docker_tests/test_examples_of_prod_image_building.py
##########
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import glob
+import os
+import re
+from functools import lru_cache
+from pathlib import Path
+
+import pytest
+import requests
+
+from docker_tests.command_utils import run_command
+from docker_tests.constants import SOURCE_ROOT
+
+DOCKER_EXAMPLES_DIR = SOURCE_ROOT / "docs" / "docker-stack" / "docker-examples"
+
+
+@lru_cache(maxsize=None)
+def get_latest_airflow_version_released():
+ response = requests.get('https://pypi.org/pypi/apache-airflow/json')
+ response.raise_for_status()
+ return response.json()['info']['version']
+
+
+@pytest.mark.skipif(
+ os.environ.get('CI') == "true",
+ reason="Skipping the script builds on CI! They take very long time to build.",
+)
+@pytest.mark.parametrize("script_file", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/*.sh", recursive=True))
+def test_shell_script_example(script_file):
+ run_command(["bash", script_file])
+
+
+@pytest.mark.parametrize("dockerfile", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/Dockerfile", recursive=True))
+def test_dockerfile_example(dockerfile):
+ rel_dockerfile_path = Path(dockerfile).relative_to(DOCKER_EXAMPLES_DIR)
+ image_name = str(rel_dockerfile_path).lower().replace("/", "-")
+ content = Path(dockerfile).read_text()
+ new_content = re.sub(
+ r'FROM apache/airflow:.*', fr'FROM apache/airflow:{get_latest_airflow_version_released()}', content
+ )
+ try:
+ run_command(
+ ["docker", "build", ".", "--tag", image_name, '-f', '-'],
+ cwd=str(Path(dockerfile).parent),
+ input=new_content.encode(),
+ )
Review comment:
Agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r757614774
##########
File path: docker_tests/test_examples_of_prod_image_building.py
##########
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import glob
+import os
+import re
+from functools import lru_cache
+from pathlib import Path
+
+import pytest
+import requests
+
+from docker_tests.command_utils import run_command
+from docker_tests.constants import SOURCE_ROOT
+
+DOCKER_EXAMPLES_DIR = SOURCE_ROOT / "docs" / "docker-stack" / "docker-examples"
+
+
+@lru_cache(maxsize=None)
+def get_latest_airflow_version_released():
+ response = requests.get('https://pypi.org/pypi/apache-airflow/json')
Review comment:
Just a note the JSON API is not stable and may change. There’s not really a good alternative though (unless you want to parse HTML), so this is fine for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756919682
##########
File path: docker_tests/test_examples_of_prod_image_building.py
##########
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import glob
+import os
+import re
+from functools import lru_cache
+from pathlib import Path
+
+import pytest
+import requests
+
+from docker_tests.command_utils import run_command
+from docker_tests.constants import SOURCE_ROOT
+
+DOCKER_EXAMPLES_DIR = SOURCE_ROOT / "docs" / "docker-stack" / "docker-examples"
+
+
+@lru_cache(maxsize=None)
+def get_latest_airflow_version_released():
+ response = requests.get('https://pypi.org/pypi/apache-airflow/json')
+ response.raise_for_status()
+ return response.json()['info']['version']
+
+
+@pytest.mark.skipif(
+ os.environ.get('CI') == "true",
+ reason="Skipping the script builds on CI! They take very long time to build.",
+)
+@pytest.mark.parametrize("script_file", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/*.sh", recursive=True))
+def test_shell_script_example(script_file):
+ run_command(["bash", script_file])
+
+
+@pytest.mark.parametrize("dockerfile", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/Dockerfile", recursive=True))
+def test_dockerfile_example(dockerfile):
+ rel_dockerfile_path = Path(dockerfile).relative_to(DOCKER_EXAMPLES_DIR)
+ image_name = str(rel_dockerfile_path).lower().replace("/", "-")
+ content = Path(dockerfile).read_text()
+ new_content = re.sub(
+ r'FROM apache/airflow:.*', fr'FROM apache/airflow:{get_latest_airflow_version_released()}', content
+ )
+ try:
+ run_command(
+ ["docker", "build", ".", "--tag", image_name, '-f', '-'],
+ cwd=str(Path(dockerfile).parent),
+ input=new_content.encode(),
+ )
Review comment:
I beleive there is no "remove intermediate images when failed" option. There is `--rm` which removes intermediate images if you intend to only care about the final image, but that does not let you reuse the cache when you rebuild the image again. We do not "care" too much for it in this case so adding `--rm` might be a good idea though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756576698
##########
File path: docker_tests/test_examples_of_prod_image_building.py
##########
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import glob
+import os
+import re
+from functools import lru_cache
+from pathlib import Path
+
+import pytest
+import requests
+
+from docker_tests.command_utils import run_command
+from docker_tests.constants import SOURCE_ROOT
+
+DOCKER_EXAMPLES_DIR = SOURCE_ROOT / "docs" / "docker-stack" / "docker-examples"
+
+
+@lru_cache(maxsize=None)
+def get_latest_airflow_version_released():
+ response = requests.get('https://pypi.org/pypi/apache-airflow/json')
+ response.raise_for_status()
+ return response.json()['info']['version']
+
+
+@pytest.mark.skipif(
+ os.environ.get('CI') == "true",
+ reason="Skipping the script builds on CI! They take very long time to build.",
+)
+@pytest.mark.parametrize("script_file", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/*.sh", recursive=True))
+def test_shell_script_example(script_file):
+ run_command(["bash", script_file])
+
+
+@pytest.mark.parametrize("dockerfile", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/Dockerfile", recursive=True))
+def test_dockerfile_example(dockerfile):
+ rel_dockerfile_path = Path(dockerfile).relative_to(DOCKER_EXAMPLES_DIR)
+ image_name = str(rel_dockerfile_path).lower().replace("/", "-")
+ content = Path(dockerfile).read_text()
+ new_content = re.sub(
+ r'FROM apache/airflow:.*', fr'FROM apache/airflow:{get_latest_airflow_version_released()}', content
+ )
+ try:
+ run_command(
+ ["docker", "build", ".", "--tag", image_name, '-f', '-'],
+ cwd=str(Path(dockerfile).parent),
+ input=new_content.encode(),
+ )
Review comment:
I think this leaves dangling intermediate images if the command fails. Is this intended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756577183
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
- "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
+ "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}"
Review comment:
Was this just a typo? The version is `2.3.0.dev0` everywhere else in this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r757170177
##########
File path: docker_tests/test_examples_of_prod_image_building.py
##########
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import glob
+import os
+import re
+from functools import lru_cache
+from pathlib import Path
+
+import pytest
+import requests
+
+from docker_tests.command_utils import run_command
+from docker_tests.constants import SOURCE_ROOT
+
+DOCKER_EXAMPLES_DIR = SOURCE_ROOT / "docs" / "docker-stack" / "docker-examples"
+
+
+@lru_cache(maxsize=None)
+def get_latest_airflow_version_released():
+ response = requests.get('https://pypi.org/pypi/apache-airflow/json')
+ response.raise_for_status()
+ return response.json()['info']['version']
+
+
+@pytest.mark.skipif(
+ os.environ.get('CI') == "true",
+ reason="Skipping the script builds on CI! They take very long time to build.",
+)
+@pytest.mark.parametrize("script_file", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/*.sh", recursive=True))
+def test_shell_script_example(script_file):
+ run_command(["bash", script_file])
+
+
+@pytest.mark.parametrize("dockerfile", glob.glob(f"{DOCKER_EXAMPLES_DIR}/**/Dockerfile", recursive=True))
+def test_dockerfile_example(dockerfile):
+ rel_dockerfile_path = Path(dockerfile).relative_to(DOCKER_EXAMPLES_DIR)
+ image_name = str(rel_dockerfile_path).lower().replace("/", "-")
+ content = Path(dockerfile).read_text()
+ new_content = re.sub(
+ r'FROM apache/airflow:.*', fr'FROM apache/airflow:{get_latest_airflow_version_released()}', content
+ )
+ try:
+ run_command(
+ ["docker", "build", ".", "--tag", image_name, '-f', '-'],
+ cwd=str(Path(dockerfile).parent),
+ input=new_content.encode(),
+ )
Review comment:
```
--rm Remove intermediate containers after a successful build (default true)
```
By default, this option is set, so we don't need to add it. In addition, it is worth adding that these side effects do not negatively affect the tests, so we do not have to worry about them. It is only a cache and docker makes sure that no corrupted data is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756924024
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
- "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
+ "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}"
Review comment:
Should be 2.2. in both cases. It was originally 2.0.2 in both cases, but has been replaced by one of bulk version replacements apparently. (this example was showing also the PIP version downgrade needed for pre 2.1.0 version - but we should definitely drop it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#issuecomment-979250098
BTW. It would be great if you actually run those examples for "build customization" @mik-laj. They are not run on CI as those "image customisation" take a lot of time (only extending ones are run in CI). I run those occasionally to see if they work though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r757169435
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
- "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
+ "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}"
Review comment:
This field is updated to version according to `setup.py`. https://github.com/apache/airflow/blob/aff647715baca7da26edf635abd42c7e911f5313/scripts/ci/pre_commit/pre_commit_update_versions.py#L24-L29
I can change it, but it will cause a lot of unrelated changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r757475397
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
- "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
+ "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}"
Review comment:
@potiuk LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756923132
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
Review comment:
Yeah. Lets' remove it It used to be needed at the beginning when I wanted to show example of using previous PIP version but that need is long gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#issuecomment-979248559
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r757437714
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
- "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
+ "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}"
Review comment:
I just removed the file from auto-fix (see the fixup i added - if you are ok we can merge it) . The cases with customization can be easily updated manually (and they do not even have to be because they are about using specific version, so it does not really matter which version is used).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756924024
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
- "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
+ "apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==${AIRFLOW_VERSION}"
Review comment:
Should be 2.2. in both cases. It was originally 2.0.2 in both cases, but has been replaced by one of bulk version replacements apparently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756923358
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
Review comment:
```suggestion
export AIRFLOW_VERSION="2.2.2"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19819: Rewrite image building tests to Python
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19819:
URL: https://github.com/apache/airflow/pull/19819#discussion_r756577340
##########
File path: docs/docker-stack/docker-examples/restricted/restricted_environments.sh
##########
@@ -22,28 +22,29 @@ AIRFLOW_SOURCES="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../" && pwd)"
cd "${AIRFLOW_SOURCES}"
# [START download]
+export AIRFLOW_VERSION="2.3.0.dev0"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-3.7.txt" \
- https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0.dev0/constraints-3.7.txt
+ "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.7.txt"
# For Airflow pre 2.1 you need to use PIP 20.2.4 to install/download Airflow packages.
pip install pip==20.2.4
Review comment:
Is this still needed? Or maybe we need to parse `AIRFLOW_VERSION` to decide what version of pip to use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org