You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/09 10:25:40 UTC
[GitHub] [airflow] feluelle opened a new pull request #8228: Expose Airflow
Webserver Port in Production Docker Image
feluelle opened a new pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228
The production docker image currently does not expose a port which is needed for Airflow's webserver to be accessible from the host. This PR exposes a port and the breeze script will pass the port configured in breeze env.
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407943507
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
+ - ../../../.dockerignore:/opt/airflow/.dockerignore:cached
+ - ../../../.flake8:/opt/airflow/.flake8:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
+ - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+ - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+ - ../../../LICENSE:/opt/airflow/LICENSE:cached
+ - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+ - ../../../NOTICE:/opt/airflow/NOTICE:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
+ - ../../../common:/opt/airflow/common:cached
+ - ../../../dags:/opt/airflow/dags:cached
+ - ../../../dev:/opt/airflow/dev:cached
+ - ../../../docs:/opt/airflow/docs:cached
+ - ../../../files:/files:cached
+ - ../../../dist:/dist:cached
+ - ../../../hooks:/opt/airflow/hooks:cached
+ - ../../../logs:/root/airflow/logs:cached
+ - ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pytest.ini:/opt/airflow/pytest.ini:cached
+ - ../../../requirements:/opt/airflow/requirements:cached
+ - ../../../scripts:/opt/airflow/scripts:cached
+ - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+ - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
+ - ../../../setup.py:/opt/airflow/setup.py:cached
+ - ../../../tests:/opt/airflow/tests:cached
+ - ../../../tmp:/opt/airflow/tmp:cached
+ # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle edited a comment on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613389048
> we should not map the: "airflow, common, dags, requirements, scripts" in production image. They might be totally misleading and can have side effects if /opt/airflow is on the PYTHONPATH
Yes, without it - it works. `/version` and `/home` 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613449528
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=h1) Report
> Merging [#8228](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8cae07ea1873a90516120d9ffbd28e7fdd2f78a4&el=desc) will **decrease** coverage by `0.45%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8228 +/- ##
==========================================
- Coverage 88.42% 87.96% -0.46%
==========================================
Files 937 937
Lines 45234 45234
==========================================
- Hits 39996 39790 -206
- Misses 5238 5444 +206
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
| [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `54.28% <0.00%> (-31.43%)` | :arrow_down: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `78.87% <0.00%> (-15.50%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=footer). Last update [8cae07e...116279a](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8228: Expose
Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613449528
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=h1) Report
> Merging [#8228](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/eee4ebaeeb1991480ee178ddb600bc69b2a88764&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8228 +/- ##
==========================================
- Coverage 88.43% 88.42% -0.01%
==========================================
Files 940 940
Lines 45353 45355 +2
==========================================
- Hits 40108 40106 -2
- Misses 5245 5249 +4
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.52% <0.00%> (-1.12%)` | :arrow_down: |
| [...oviders/google/cloud/utils/credentials\_provider.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL3V0aWxzL2NyZWRlbnRpYWxzX3Byb3ZpZGVyLnB5) | `94.11% <0.00%> (+0.14%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=footer). Last update [eee4eba...b4ecb42](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406370913
##########
File path: scripts/ci/docker-compose/backend-mysql.yml
##########
@@ -25,6 +25,8 @@ services:
- AIRFLOW__CELERY__RESULT_BACKEND=db+mysql://root@mysql/airflow?charset=utf8mb4
depends_on:
- mysql
+ ports:
Review comment:
Currently we are using it local.yml for CI build. But it's not used in PROD image. I think we should have another "local-prod.yml" and expose PORT there. then you'd need to update `prepare_command_files()` function in `breeze` and modify COMPOSE_PROD_FILE to add this local-prod.yml to this variable inside this if:.
```
if [[ "${MOUNT_LOCAL_SOURCES}" != "false" ]]; then
COMPOSE_CI_FILE=${COMPOSE_CI_FILE}:${LOCAL_DOCKER_COMPOSE_FILE}
fi
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8228: Expose Airflow Webserver
Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613325507
Looks Great!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r408062746
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle edited a comment on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613384798
I also had an issue with volumes being mounted `GET`ting on `/home`:
```
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
return f(self, *args, **kwargs)
File "/opt/airflow/airflow/www/views.py", line 355, in index
status_count_paused=status_count_paused)
File "/opt/airflow/airflow/www/views.py", line 190, in render_template
**kwargs
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 281, in render_template
template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
File "/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
ctx.app,
File "/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "/opt/airflow/airflow/www/templates/airflow/dags.html", line 20, in top-level template code
{% extends base_template %}
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 1005, in render
return concat(self.root_render_func(self.new_context(vars)))
File "/opt/airflow/airflow/www/templates/airflow/dags.html", line 17, in root
under the License.
File "/opt/airflow/airflow/www/templates/airflow/master.html", line 16, in root
specific language governing permissions and limitations
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 17, in root
{% include 'appbuilder/flash.html' %}
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 32, in root
<link href="{{url_for('appbuilder.static',filename='select2/select2.css')}}" rel="stylesheet">
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 37, in block_body
File "/opt/airflow/airflow/www/templates/airflow/dags.html", line 106, in block_content
</td>
TypeError: can only concatenate str (not "NoneType") to str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613399645
Should I add docs regarding that in this PR? Just a note about what you said not using both at the same time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-611454225
@potiuk Is it correct that we should map the port like I did in every backend compose file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407041384
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
+ - ../../../.dockerignore:/opt/airflow/.dockerignore:cached
+ - ../../../.flake8:/opt/airflow/.flake8:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
+ - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+ - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+ - ../../../LICENSE:/opt/airflow/LICENSE:cached
+ - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+ - ../../../NOTICE:/opt/airflow/NOTICE:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
+ - ../../../common:/opt/airflow/common:cached
+ - ../../../dags:/opt/airflow/dags:cached
+ - ../../../dev:/opt/airflow/dev:cached
+ - ../../../docs:/opt/airflow/docs:cached
+ - ../../../files:/files:cached
+ - ../../../dist:/dist:cached
+ - ../../../hooks:/opt/airflow/hooks:cached
+ - ../../../logs:/root/airflow/logs:cached
+ - ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pytest.ini:/opt/airflow/pytest.ini:cached
+ - ../../../requirements:/opt/airflow/requirements:cached
+ - ../../../scripts:/opt/airflow/scripts:cached
+ - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+ - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
+ - ../../../setup.py:/opt/airflow/setup.py:cached
+ - ../../../tests:/opt/airflow/tests:cached
+ - ../../../tmp:/opt/airflow/tmp:cached
+ # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
Review comment:
the linting, ci and testing stuff..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406365699
##########
File path: scripts/ci/_utils.sh
##########
@@ -1428,6 +1429,7 @@ function prepare_prod_build() {
"--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
# TODO We should uncomment it just before we merge this change to master/v1_10_test
#"--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/v1-10-test/entrypoint.sh"
+ "--build-arg" "AIRFLOW_WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT}"
Review comment:
Same here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8228: Expose Airflow Webserver
Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613972370
Go! Thanks @feluelle !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613398714
Yes that is true. It is not only that - both are using the same database if for example both (by default) have backend mysql or postgres.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r408061941
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
Review comment:
As discussed in #8229 -> we should not map the: "airflow, common, dags, requirements, scripts" in production image. They might be totally misleading and can have side effects if /opt/airflow is on the PYTHONPATH
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406113149
##########
File path: scripts/ci/_utils.sh
##########
@@ -1419,6 +1419,7 @@ function prepare_prod_build() {
"--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
# TODO We should uncomment it just before we merge this change to master/v1_10_test
Review comment:
Also what about this `TODO`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407043149
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
+ - ../../../.dockerignore:/opt/airflow/.dockerignore:cached
+ - ../../../.flake8:/opt/airflow/.flake8:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
+ - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+ - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+ - ../../../LICENSE:/opt/airflow/LICENSE:cached
+ - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+ - ../../../NOTICE:/opt/airflow/NOTICE:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
+ - ../../../common:/opt/airflow/common:cached
+ - ../../../dags:/opt/airflow/dags:cached
+ - ../../../dev:/opt/airflow/dev:cached
+ - ../../../docs:/opt/airflow/docs:cached
+ - ../../../files:/files:cached
+ - ../../../dist:/dist:cached
+ - ../../../hooks:/opt/airflow/hooks:cached
+ - ../../../logs:/root/airflow/logs:cached
+ - ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pytest.ini:/opt/airflow/pytest.ini:cached
+ - ../../../requirements:/opt/airflow/requirements:cached
+ - ../../../scripts:/opt/airflow/scripts:cached
+ - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+ - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
+ - ../../../setup.py:/opt/airflow/setup.py:cached
+ - ../../../tests:/opt/airflow/tests:cached
+ - ../../../tmp:/opt/airflow/tmp:cached
+ # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
Review comment:
Well some of those might be useful as they give a lot of nice things when it comes to development. They keep history of commands, you can define your own aliases etc + you can for example build backport packages outside (in dist) and install them inside the container. Some of the stuff is used for pylint/mypy/etc so indeed it is not relevant:
I reviewed the list and I think this - shorter - list might make sense to stay:
```
- ../../../.bash_aliases:/root/.bash_aliases:cached
- ../../../.bash_history:/root/.bash_history:cached
- ../../../.github:/opt/airflow/.github:cached
- ../../../.inputrc:/root/.inputrc:cached
- ../../../.kube:/root/.kube:cached
- ../../../airflow:/opt/airflow/airflow:cached
- ../../../common:/opt/airflow/common:cached
- ../../../dags:/opt/airflow/dags:cached
- ../../../files:/files:cached
- ../../../dist:/dist:cached
- ../../../requirements:/opt/airflow/requirements:cached
- ../../../scripts:/opt/airflow/scripts:cached
- ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
- ../../../setup.cfg:/opt/airflow/setup.cfg:cached
- ../../../setup.py:/opt/airflow/setup.py:cached
- ../../../tests:/opt/airflow/tests:cached
- ../../../tmp:/opt/airflow/tmp:cached
```
And remember we can skip it via `--skip-mounting-local-sources` flag to breeze as well - to have a vanilla production container without any mounts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8228: Expose Airflow Webserver
Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613532777
Perfect! Thanks @feluelle ! Let's wait for Travis and I am super happy to merge it :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8228: Expose
Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613449528
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=h1) Report
> Merging [#8228](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8cae07ea1873a90516120d9ffbd28e7fdd2f78a4&el=desc) will **decrease** coverage by `0.45%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8228 +/- ##
==========================================
- Coverage 88.42% 87.96% -0.46%
==========================================
Files 937 937
Lines 45234 45234
==========================================
- Hits 39996 39790 -206
- Misses 5238 5444 +206
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
| [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `54.28% <0.00%> (-31.43%)` | :arrow_down: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `78.87% <0.00%> (-15.50%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=footer). Last update [8cae07e...116279a](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406365402
##########
File path: Dockerfile
##########
@@ -328,5 +328,10 @@ WORKDIR ${AIRFLOW_HOME}
ENV AIRFLOW__CORE__LOAD_EXAMPLES="false"
+ARG AIRFLOW_WEBSERVER_HOST_PORT=8080
Review comment:
I do no think we need to have it configurable. Simply 8080 should be good enough?. That would make it much simpler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407065268
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
+ - ../../../.dockerignore:/opt/airflow/.dockerignore:cached
+ - ../../../.flake8:/opt/airflow/.flake8:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
+ - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+ - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+ - ../../../LICENSE:/opt/airflow/LICENSE:cached
+ - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+ - ../../../NOTICE:/opt/airflow/NOTICE:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
+ - ../../../common:/opt/airflow/common:cached
+ - ../../../dags:/opt/airflow/dags:cached
+ - ../../../dev:/opt/airflow/dev:cached
+ - ../../../docs:/opt/airflow/docs:cached
+ - ../../../files:/files:cached
+ - ../../../dist:/dist:cached
+ - ../../../hooks:/opt/airflow/hooks:cached
+ - ../../../logs:/root/airflow/logs:cached
+ - ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pytest.ini:/opt/airflow/pytest.ini:cached
+ - ../../../requirements:/opt/airflow/requirements:cached
+ - ../../../scripts:/opt/airflow/scripts:cached
+ - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+ - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
+ - ../../../setup.py:/opt/airflow/setup.py:cached
+ - ../../../tests:/opt/airflow/tests:cached
+ - ../../../tmp:/opt/airflow/tmp:cached
+ # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
Review comment:
or maybe we keep it for now.. because these get auto-generated and I am not really sure what to change in the _utils for it to work. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613389048
> we should not map the: "airflow, common, dags, requirements, scripts" in production image. They might be totally misleading and can have side effects if /opt/airflow is on the PYTHONPATH
Yes, withouth it - it works. `/version` and `/home` 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613390895
But the `skip` mounting flag is misleading when we also map ports, isn't it? You won't think that when you do not mount you will also disable port mapping in prod image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8228: Expose
Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613449528
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=h1) Report
> Merging [#8228](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8cae07ea1873a90516120d9ffbd28e7fdd2f78a4&el=desc) will **decrease** coverage by `0.45%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8228 +/- ##
==========================================
- Coverage 88.42% 87.96% -0.46%
==========================================
Files 937 937
Lines 45234 45234
==========================================
- Hits 39996 39790 -206
- Misses 5238 5444 +206
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
| [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `54.28% <0.00%> (-31.43%)` | :arrow_down: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `78.87% <0.00%> (-15.50%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=footer). Last update [8cae07e...116279a](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407041243
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
+ - ../../../.dockerignore:/opt/airflow/.dockerignore:cached
+ - ../../../.flake8:/opt/airflow/.flake8:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
+ - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+ - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+ - ../../../LICENSE:/opt/airflow/LICENSE:cached
+ - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+ - ../../../NOTICE:/opt/airflow/NOTICE:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
+ - ../../../common:/opt/airflow/common:cached
+ - ../../../dags:/opt/airflow/dags:cached
+ - ../../../dev:/opt/airflow/dev:cached
+ - ../../../docs:/opt/airflow/docs:cached
+ - ../../../files:/files:cached
+ - ../../../dist:/dist:cached
+ - ../../../hooks:/opt/airflow/hooks:cached
+ - ../../../logs:/root/airflow/logs:cached
+ - ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pytest.ini:/opt/airflow/pytest.ini:cached
+ - ../../../requirements:/opt/airflow/requirements:cached
+ - ../../../scripts:/opt/airflow/scripts:cached
+ - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+ - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
+ - ../../../setup.py:/opt/airflow/setup.py:cached
+ - ../../../tests:/opt/airflow/tests:cached
+ - ../../../tmp:/opt/airflow/tmp:cached
+ # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
Review comment:
We don't need any of those volumes, do we?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle edited a comment on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613394829
But then you won't be able to test prod image locally while having airflow ci running, because both are using the same port `WEBSERVER_HOST_PORT`
EDIT:
```
Error response from daemon: driver failed programming external connectivity on endpoint docker-compose_airflow_run_c577066264a5 (c0f164193ce289b5af71856fca35aef987f96e7d34f2109ec9426d4ed480c052): Bind for 0.0.0.0:28080 failed: port is already allocated
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle edited a comment on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613398714
Yes that is true. It is not only that - both are using the same database if for example both (by default) have backend mysql or postgres. (which is configured in the compose file :D)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613394829
But then you won't be able to test prod image locally while having airflow ci running, because both are using the same port `WEBSERVER_HOST_PORT`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406113403
##########
File path: scripts/ci/_utils.sh
##########
@@ -1428,6 +1429,7 @@ function prepare_prod_build() {
"--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
# TODO We should uncomment it just before we merge this change to master/v1_10_test
Review comment:
same here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8228: Expose Airflow Webserver
Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613396962
I think there are more reasons why you cannot run both prod/ci in parallel - they use the same docker-compose configuration and they are both named "airflow" so I would not worry about that. I think you should always stop the CI Breeze before starting Prod one. We might add some better diagnosis/explanation of the case rather than "port listening" error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8228: Expose Airflow Webserver
Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613400027
> Should I add docs regarding that in this PR? Just a note about what you said not using both at the same time?
Absolutely :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8228: Expose
Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613449528
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=h1) Report
> Merging [#8228](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8cae07ea1873a90516120d9ffbd28e7fdd2f78a4&el=desc) will **decrease** coverage by `0.45%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8228 +/- ##
==========================================
- Coverage 88.42% 87.96% -0.46%
==========================================
Files 937 937
Lines 45234 45234
==========================================
- Hits 39996 39790 -206
- Misses 5238 5444 +206
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
| [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0.00%> (-38.47%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `50.67% <0.00%> (-37.84%)` | :arrow_down: |
| [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `54.28% <0.00%> (-31.43%)` | :arrow_down: |
| [airflow/providers/postgres/hooks/postgres.py](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvaG9va3MvcG9zdGdyZXMucHk=) | `78.87% <0.00%> (-15.50%)` | :arrow_down: |
| ... and [5 more](https://codecov.io/gh/apache/airflow/pull/8228/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=footer). Last update [8cae07e...116279a](https://codecov.io/gh/apache/airflow/pull/8228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613478395
@potiuk can you check again, please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8228: Expose Airflow Webserver
Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613391557
Indeed. I think we should move ports to "base.yaml". I can't think of any reason why it would be harm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613384798
I also had an issue with volumes being mounted on `/home`:
```
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/home/airflow/.local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
return f(self, *args, **kwargs)
File "/opt/airflow/airflow/www/views.py", line 355, in index
status_count_paused=status_count_paused)
File "/opt/airflow/airflow/www/views.py", line 190, in render_template
**kwargs
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/baseviews.py", line 281, in render_template
template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
File "/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
ctx.app,
File "/home/airflow/.local/lib/python3.7/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "/opt/airflow/airflow/www/templates/airflow/dags.html", line 20, in top-level template code
{% extends base_template %}
File "/home/airflow/.local/lib/python3.7/site-packages/jinja2/environment.py", line 1005, in render
return concat(self.root_render_func(self.new_context(vars)))
File "/opt/airflow/airflow/www/templates/airflow/dags.html", line 17, in root
under the License.
File "/opt/airflow/airflow/www/templates/airflow/master.html", line 16, in root
specific language governing permissions and limitations
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 17, in root
{% include 'appbuilder/flash.html' %}
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 32, in root
<link href="{{url_for('appbuilder.static',filename='select2/select2.css')}}" rel="stylesheet">
File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 37, in block_body
File "/opt/airflow/airflow/www/templates/airflow/dags.html", line 106, in block_content
</td>
TypeError: can only concatenate str (not "NoneType") to str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406365616
##########
File path: scripts/ci/_utils.sh
##########
@@ -1419,6 +1419,7 @@ function prepare_prod_build() {
"--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
# TODO We should uncomment it just before we merge this change to master/v1_10_test
#"--build-arg" "ENTRYPOINT_FILE=https://raw.githubusercontent.com/apache/airflow/v1-10-test/entrypoint.sh"
+ "--build-arg" "AIRFLOW_WEBSERVER_HOST_PORT=${WEBSERVER_HOST_PORT}"
Review comment:
If we expose 8080 static, we won't need those.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406362501
##########
File path: scripts/ci/_utils.sh
##########
@@ -1419,6 +1419,7 @@ function prepare_prod_build() {
"--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
# TODO We should uncomment it just before we merge this change to master/v1_10_test
Review comment:
This is already corrected in #8234. Also we should adapt it slightly after we release 1.10.10 (for releases < 1.10.10 it should be pointing to 1.10.10 and for >= 1.10.10 it should point to ${AIRFLOW_VERSION}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407064781
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
+ - ../../../.dockerignore:/opt/airflow/.dockerignore:cached
+ - ../../../.flake8:/opt/airflow/.flake8:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
+ - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+ - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+ - ../../../LICENSE:/opt/airflow/LICENSE:cached
+ - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+ - ../../../NOTICE:/opt/airflow/NOTICE:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
+ - ../../../common:/opt/airflow/common:cached
+ - ../../../dags:/opt/airflow/dags:cached
+ - ../../../dev:/opt/airflow/dev:cached
+ - ../../../docs:/opt/airflow/docs:cached
+ - ../../../files:/files:cached
+ - ../../../dist:/dist:cached
+ - ../../../hooks:/opt/airflow/hooks:cached
+ - ../../../logs:/root/airflow/logs:cached
+ - ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pytest.ini:/opt/airflow/pytest.ini:cached
+ - ../../../requirements:/opt/airflow/requirements:cached
+ - ../../../scripts:/opt/airflow/scripts:cached
+ - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+ - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
+ - ../../../setup.py:/opt/airflow/setup.py:cached
+ - ../../../tests:/opt/airflow/tests:cached
+ - ../../../tmp:/opt/airflow/tmp:cached
+ # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
Review comment:
looks good. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613971784
Are we good to go @potiuk ? :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407081546
##########
File path: scripts/ci/docker-compose/local-prod.yml
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+---
+version: "2.2"
+services:
+ airflow:
+ # We need to mount files an directories individually because some files
+ # such apache_airflow.egg-info should not be mounted from host
+ # we only mount those files that it makes sense to edit while developing
+ # or those that might be useful to see in the host as output of the
+ # tests (such as logs)
+ volumes:
+ # START automatically generated volumes from LOCAL_MOUNTS in _utils.sh
+ - ../../../.bash_aliases:/root/.bash_aliases:cached
+ - ../../../.bash_history:/root/.bash_history:cached
+ - ../../../.coveragerc:/opt/airflow/.coveragerc:cached
+ - ../../../.dockerignore:/opt/airflow/.dockerignore:cached
+ - ../../../.flake8:/opt/airflow/.flake8:cached
+ - ../../../.github:/opt/airflow/.github:cached
+ - ../../../.inputrc:/root/.inputrc:cached
+ - ../../../.kube:/root/.kube:cached
+ - ../../../.rat-excludes:/opt/airflow/.rat-excludes:cached
+ - ../../../CHANGELOG.txt:/opt/airflow/CHANGELOG.txt:cached
+ - ../../../Dockerfile.ci:/opt/airflow/Dockerfile.ci:cached
+ - ../../../LICENSE:/opt/airflow/LICENSE:cached
+ - ../../../MANIFEST.in:/opt/airflow/MANIFEST.in:cached
+ - ../../../NOTICE:/opt/airflow/NOTICE:cached
+ - ../../../airflow:/opt/airflow/airflow:cached
+ - ../../../common:/opt/airflow/common:cached
+ - ../../../dags:/opt/airflow/dags:cached
+ - ../../../dev:/opt/airflow/dev:cached
+ - ../../../docs:/opt/airflow/docs:cached
+ - ../../../files:/files:cached
+ - ../../../dist:/dist:cached
+ - ../../../hooks:/opt/airflow/hooks:cached
+ - ../../../logs:/root/airflow/logs:cached
+ - ../../../pylintrc:/opt/airflow/pylintrc:cached
+ - ../../../pytest.ini:/opt/airflow/pytest.ini:cached
+ - ../../../requirements:/opt/airflow/requirements:cached
+ - ../../../scripts:/opt/airflow/scripts:cached
+ - ../../../scripts/ci/in_container/entrypoint_ci.sh:/entrypoint_ci.sh:cached
+ - ../../../setup.cfg:/opt/airflow/setup.cfg:cached
+ - ../../../setup.py:/opt/airflow/setup.py:cached
+ - ../../../tests:/opt/airflow/tests:cached
+ - ../../../tmp:/opt/airflow/tmp:cached
+ # END automatically generated volumes from LOCAL_MOUNTS in _utils.sh
Review comment:
I think we do not need to get them auto-generated in the prod version. The main reason they are auto-generated is that I wanted to keep the the same in docker-compose and in the scripts which are running pre-commit checks. Pre-commit checks do not use docker compose but they are running docker run commands directly. This only makes sense for CI image. For prod image it's ok to keep separate list I think - most of the pre-commit/testrelated files might be misleading/are unnecessary in the production image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r406362562
##########
File path: scripts/ci/_utils.sh
##########
@@ -1428,6 +1429,7 @@ function prepare_prod_build() {
"--build-arg" "CONSTRAINT_REQUIREMENTS=https://raw.githubusercontent.com/apache/airflow/v1-10-test/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
# TODO We should uncomment it just before we merge this change to master/v1_10_test
Review comment:
same here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #8228:
Expose Airflow Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#discussion_r407041120
##########
File path: scripts/ci/docker-compose/backend-mysql.yml
##########
@@ -25,6 +25,8 @@ services:
- AIRFLOW__CELERY__RESULT_BACKEND=db+mysql://root@mysql/airflow?charset=utf8mb4
depends_on:
- mysql
+ ports:
Review comment:
Like this https://github.com/apache/airflow/pull/8228/files ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #8228: Expose Airflow
Webserver Port in Production Docker Image
Posted by GitBox <gi...@apache.org>.
feluelle commented on issue #8228: Expose Airflow Webserver Port in Production Docker Image
URL: https://github.com/apache/airflow/pull/8228#issuecomment-613290232
Ready for re-review @potiuk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services