You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/27 16:14:50 UTC
[GitHub] [airflow] dimberman opened a new pull request #9038: Use production image for k8s tests
dimberman opened a new pull request #9038:
URL: https://github.com/apache/airflow/pull/9038
The CI image has become too large to load into KinD,
it also only really makes sense to use the production image for
integration tests
---
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Target Github ISSUE in description if exists
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on a change in pull request #9038: Use production image for k8s tests
Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #9038:
URL: https://github.com/apache/airflow/pull/9038#discussion_r431297732
##########
File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh
##########
@@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/
echo
echo "Building image from ${AIRFLOW_CI_IMAGE} with latest sources"
echo
-start_output_heartbeat "Rebuilding Kubernetes image" 3
-docker build \
- --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
- --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
- --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
- --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
- --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
- --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
- --build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH}" \
- --build-arg AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="${AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD}" \
- --build-arg UPGRADE_TO_LATEST_REQUIREMENTS="${UPGRADE_TO_LATEST_REQUIREMENTS}" \
- --build-arg HOME="${HOME}" \
- --cache-from "${AIRFLOW_CI_IMAGE}" \
- --tag="${AIRFLOW_CI_IMAGE}" \
- --target="main" \
- -f Dockerfile.ci . >> "${OUTPUT_LOG}"
-echo
+#export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}"
Review comment:
@potiuk can you please point me to how I can initialize these env variables. I can't find it and there are a lot of bash scripts to navigate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #9038: Use production image for k8s tests
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #9038:
URL: https://github.com/apache/airflow/pull/9038#issuecomment-635115335
Still the same, old error `Couldn't resolve host 'github.com'` :<
But it seems the persistent mode work. Would we be ok with having only one k8s test for a while?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman merged pull request #9038: Use production image for k8s tests
Posted by GitBox <gi...@apache.org>.
dimberman merged pull request #9038:
URL: https://github.com/apache/airflow/pull/9038
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #9038: Use production image for k8s tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #9038:
URL: https://github.com/apache/airflow/pull/9038#discussion_r431367442
##########
File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh
##########
@@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/
echo
echo "Building image from ${AIRFLOW_CI_IMAGE} with latest sources"
echo
-start_output_heartbeat "Rebuilding Kubernetes image" 3
-docker build \
- --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
- --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
- --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
- --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
- --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
- --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
- --build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH}" \
- --build-arg AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="${AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD}" \
- --build-arg UPGRADE_TO_LATEST_REQUIREMENTS="${UPGRADE_TO_LATEST_REQUIREMENTS}" \
- --build-arg HOME="${HOME}" \
- --cache-from "${AIRFLOW_CI_IMAGE}" \
- --tag="${AIRFLOW_CI_IMAGE}" \
- --target="main" \
- -f Dockerfile.ci . >> "${OUTPUT_LOG}"
-echo
+#export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}"
Review comment:
Those variablse are all set in the HOST and you need to pass them to the container. But I think this is not really needed . I just rebased #8265 which moves setting up kubernetes kind cluster and deploying production image to host rather than to CI image and I am going to finish it up this week, so I think it's better to do it this way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on a change in pull request #9038: Use production image for k8s tests
Posted by GitBox <gi...@apache.org>.
dimberman commented on a change in pull request #9038:
URL: https://github.com/apache/airflow/pull/9038#discussion_r431398123
##########
File path: scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh
##########
@@ -38,46 +38,25 @@ cp /entrypoint.sh scripts/docker/
echo
echo "Building image from ${AIRFLOW_CI_IMAGE} with latest sources"
echo
-start_output_heartbeat "Rebuilding Kubernetes image" 3
-docker build \
- --build-arg PYTHON_BASE_IMAGE="${PYTHON_BASE_IMAGE}" \
- --build-arg PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION}" \
- --build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
- --build-arg AIRFLOW_EXTRAS="${AIRFLOW_EXTRAS}" \
- --build-arg ADDITIONAL_AIRFLOW_EXTRAS="${ADDITIONAL_AIRFLOW_EXTRAS}" \
- --build-arg ADDITIONAL_PYTHON_DEPS="${ADDITIONAL_PYTHON_DEPS}" \
- --build-arg AIRFLOW_BRANCH="${AIRFLOW_BRANCH}" \
- --build-arg AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD="${AIRFLOW_CONTAINER_CI_OPTIMISED_BUILD}" \
- --build-arg UPGRADE_TO_LATEST_REQUIREMENTS="${UPGRADE_TO_LATEST_REQUIREMENTS}" \
- --build-arg HOME="${HOME}" \
- --cache-from "${AIRFLOW_CI_IMAGE}" \
- --tag="${AIRFLOW_CI_IMAGE}" \
- --target="main" \
- -f Dockerfile.ci . >> "${OUTPUT_LOG}"
-echo
+#export AIRFLOW_PROD_BASE_TAG="${DEFAULT_BRANCH}-python${PYTHON_MAJOR_MINOR_VERSION}"
Review comment:
@potiuk I agree your way is better, but until this is merged the k8s won't pass because the image is so massive so we should figure out a hack for now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org