You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/09 12:25:04 UTC
[GitHub] [airflow] Taragolis opened a new pull request, #28257: Dynamically forward ports from trino integration service to host
Taragolis opened a new pull request, #28257:
URL: https://github.com/apache/airflow/pull/28257
Time to time we got annoying error in CI
```
ERROR: for trino Cannot start service trino: driver failed programming external connectivity on endpoint airflow-test-integration_trino_1 (SOMEDIGEST): Error starting userland proxy: listen tcp4 0.0.0.0:38080: bind: address already in use
```
Looks like we do not actually use this forwarded ports in our integrations test that mean we might also remove forwarding.
However better just forward from to first available port instead.
**How to validate and check locally**
_Terminal Session 1_
```console
❯ docker run -p 38080:8080 nginx
```
_Terminal Session 2_
```console
❯ breeze stop
❯ breeze --python 3.7 --backend postgres shell --integration trino
...
Checking backend and integrations.
PostgreSQL: OK.
Kerberos: OK.
Trino (HTTP): OK.
Trino (HTTPS): OK.
Trino (API): OK.
...
root@krb5-machine-example-com:/opt/airflow# pytest --integration trino tests/integration
...
============================================================ 9 passed, 36 skipped, 8 warnings in 11.83s ============================================================
```
_Terminal Session 3_
```console
❯ docker port docker-compose_trino_1
7778/tcp -> 0.0.0.0:37780
8080/tcp -> 0.0.0.0:38086
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #28257: Dynamically forward ports from trino integration service to host
Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #28257:
URL: https://github.com/apache/airflow/pull/28257#issuecomment-1345282264
Aaaaand... it doesn't help https://github.com/apache/airflow/actions/runs/3664583959/jobs/6195181628 😠 😭
```console
Creating airflow-test-integration_pinot_1 ...
Creating airflow-test-integration_mongo_1 ...
Creating airflow-test-integration_rabbitmq_1 ...
Creating airflow-test-integration_trino_1 ...
Creating airflow-test-integration_redis_1 ...
Creating airflow-test-integration_kdc-server-example-com_1 ...
Creating airflow-test-integration_grafana_1 ...
Creating airflow-test-integration_postgres_1 ...
Creating airflow-test-integration_statsd-exporter_1 ...
Creating airflow-test-integration_cassandra_1 ...
Host is already in use by another container
Creating airflow-test-integration_trino_1 ... error
ERROR: for airflow-test-integration_trino_1 Cannot start service trino: driver failed programming external connectivity on endpoint airflow-test-integration_trino_1 (644f482dac0e33d0838bacc0ffb6753f76d5e289a517bf2601379f57b8647e2d): Error starting userland proxy: listen tcp4 0.0.0.0:37778: bind: address already in use
Creating airflow-test-integration_mongo_1 ... done
Creating airflow-test-integration_pinot_1 ... done
Creating airflow-test-integration_kdc-server-example-com_1 ... done
Creating airflow-test-integration_rabbitmq_1 ... done
Creating airflow-test-integration_redis_1 ... done
Creating airflow-test-integration_grafana_1 ... done
Creating airflow-test-integration_cassandra_1 ... done
Creating airflow-test-integration_postgres_1 ... done
Creating airflow-test-integration_statsd-exporter_1 ... done
ERROR: for trino Cannot start service trino: driver failed programming external connectivity on endpoint airflow-test-integration_trino_1 (644f482dac0e33d0838bacc0ffb6753f76d5e289a517bf2601379f57b8647e2d): Error starting userland proxy: listen tcp4 0.0.0.0:37778: bind: address already in use
Encountered errors while bringing up the project.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28257: Dynamically forward ports from trino integration service to host
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28257:
URL: https://github.com/apache/airflow/pull/28257#issuecomment-1344313594
> Let say not well documented in short form (which almost all use) but has description in [long form](https://docs.docker.com/compose/compose-file/#long-syntax-2). That is why it not easy to find roll_eyes
>
> > `target`: the container port
> > `published`: the publicly exposed port. **Can be set as a range using syntax start-end, then actual port SHOULD be assigned within this range based on available ports.**
RIGHT :exploding_head:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #28257: Dynamically forward ports from trino integration service to host
Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #28257:
URL: https://github.com/apache/airflow/pull/28257#issuecomment-1344273128
Let say not well documented in short form (which almost all use) but has description in [long form](https://docs.docker.com/compose/compose-file/#long-syntax-2). That is why it not easy to find 🙄
> `target`: the container port
`published`: the publicly exposed port. **Can be set as a range using syntax start-end, then actual port SHOULD be assigned within this range based on available ports.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28257: Dynamically forward ports from trino integration service to host
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28257:
URL: https://github.com/apache/airflow/pull/28257#issuecomment-1344314385
Cool! Thanks For taking care of that one, I hope it will stop plaguing us
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28257: Dynamically forward ports from trino integration service to host
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #28257:
URL: https://github.com/apache/airflow/pull/28257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28257: Dynamically forward ports from trino integration service to host
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28257:
URL: https://github.com/apache/airflow/pull/28257#issuecomment-1344267157
> Oh, it actually not really easy to find this info (even with Google): https://docs.docker.com/compose/compose-file/#ports
Hmm. Actually the docs you pointed out, actually say that you CANNOT do what you did:
> Host IP, if not set, MUST bind to all network interfaces. Port can be either a single value or a range. Host and container MUST use equivalent ranges.
And all examples point to `a-b: c-d` - where all ports from a-b range map 1-1 to ports from c-d range.
Undocumented feature ? :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #28257: Dynamically forward ports from trino integration service to host
Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #28257:
URL: https://github.com/apache/airflow/pull/28257#issuecomment-1344258009
Oh, it actually not really easy to find this info (even with Google): https://docs.docker.com/compose/compose-file/#ports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org