You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Nicholas Hodgkinson <ni...@collectivehealth.com> on 2017/08/15 21:58:41 UTC
Celery with SQS?
I'm wondering if anyone has had any experience setting up Airflow on AWS
using SQS as the Celery backend. I've got an instance credential set but
I'm getting this error on the worker:
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1
handlers were checked. ['HmacAuthV4Handler'] Check your credentials
Wondering if anyone has any insight.
Thanks,
-Nik
nik.hodgkinson@collectivehealth.com
--
See why we are one of WSJ's Top Tech Companies to Watch
<https://www.wsj.com/articles/collective-healths-goal-medical-benefits-that-work-1497492300?mg=prod/accounts-wsj>
*This message may contain confidential, proprietary, or protected
information. If you are not the intended recipient, you may not review,
copy, or distribute this message. If you received this message in error,
please notify the sender by reply email and delete this message.*
Re: Celery with SQS?
Posted by Nicholas Hodgkinson <ni...@collectivehealth.com>.
Here's the full log if it's helpful for anyone in helping:
-------------- celery@e6f3c1e29507 v4.0.2 (latentcall)
---- **** -----
--- * *** * -- Linux-3.13.0-125-generic-x86_64-with-Ubuntu-14.04-trusty
2017-08-16 01:53:32
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:
airflow.executors.celery_executor:0x7fad429d8978
- ** ---------- .> transport: sqs://localhost//
- ** ---------- .> results:
postgresql://airflow:**@pg-database:5432/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this
worker)
--- ***** -----
-------------- [queues]
.> queue-1 exchange=queue-1(direct) key=queue-1
[2017-08-16 01:54:36,481] {__init__.py:57} INFO - Using executor
CeleryExecutor
[2017-08-16 01:54:36,571] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/Grammar.txt
[2017-08-16 01:54:36,593] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/PatternGrammar.txt
[2017-08-16 01:54:37,393] {__init__.py:57} INFO - Using executor
CeleryExecutor
[2017-08-16 01:54:37,480] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/Grammar.txt
[2017-08-16 01:54:37,502] {driver.py:120} INFO - Generating grammar tables
from /usr/lib/python3.4/lib2to3/PatternGrammar.txt
Starting flask
[2017-08-16 01:54:37,744] {_internal.py:87} INFO - * Running on
http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2017-08-16 01:54:38,582: ERROR/MainProcess] Caught exception reading
instance data
Traceback (most recent call last):
File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 921, in create_channel
return self._avail_channels.pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/usr/lib/python3.4/http/client.py", line 1125, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1163, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1121, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 951, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 886, in send
self.connect()
File "/usr/lib/python3.4/http/client.py", line 863, in connect
self.timeout, self.source_address)
File "/usr/lib/python3.4/socket.py", line 512, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 503, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/boto/utils.py", line 210, in
retry_url
r = opener.open(req, timeout=timeout)
File "/usr/lib/python3.4/urllib/request.py", line 463, in open
response = self._open(req, data)
File "/usr/lib/python3.4/urllib/request.py", line 481, in _open
'_open', req)
File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
result = func(*args)
File "/usr/lib/python3.4/urllib/request.py", line 1210, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/lib/python3.4/urllib/request.py", line 1184, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error timed out>
[2017-08-16 01:54:38,585: ERROR/MainProcess] Unable to read instance data,
giving up
[2017-08-16 01:54:38,585: CRITICAL/MainProcess] Unrecoverable error:
NoAuthHandlerFound("No handler was ready to authenticate. 1 handlers were
checked. ['HmacAuthV4Handler'] Check your credentials",)
Traceback (most recent call last):
File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 921, in create_channel
return self._avail_channels.pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/celery/worker/worker.py",
line 203, in start
self.blueprint.start(self)
File "/usr/local/lib/python3.4/dist-packages/celery/bootsteps.py", line
119, in start
step.start(parent)
File "/usr/local/lib/python3.4/dist-packages/celery/bootsteps.py", line
370, in start
return self.obj.start()
File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 318, in start
blueprint.start(self)
File "/usr/local/lib/python3.4/dist-packages/celery/bootsteps.py", line
119, in start
step.start(parent)
File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/connection.py",
line 21, in start
c.connection = c.connect()
File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 405, in connect
conn = self.connection_for_read(heartbeat=self.amqheartbeat)
File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 412, in connection_for_read
self.app.connection_for_read(heartbeat=heartbeat))
File
"/usr/local/lib/python3.4/dist-packages/celery/worker/consumer/consumer.py",
line 436, in ensure_connected
callback=maybe_shutdown,
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
405, in ensure_connection
callback)
File "/usr/local/lib/python3.4/dist-packages/kombu/utils/functional.py",
line 333, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
261, in connect
return self.connection
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
802, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line
757, in _establish_connection
conn = self.transport.establish_connection()
File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 941, in establish_connection
self._avail_channels.append(self.create_channel(self))
File
"/usr/local/lib/python3.4/dist-packages/kombu/transport/virtual/base.py",
line 923, in create_channel
channel = self.Channel(connection)
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 102, in __init__
self._update_queue_cache(self.queue_name_prefix)
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 108, in _update_queue_cache
queues = self.sqs.get_all_queues(prefix=queue_name_prefix)
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 405, in sqs
self._sqs = self._aws_connect_to(SQSConnection, regions())
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/SQS.py",
line 400, in _aws_connect_to
port=port)
File "/usr/local/lib/python3.4/dist-packages/boto/sqs/connection.py",
line 62, in __init__
profile_name=profile_name)
File "/usr/local/lib/python3.4/dist-packages/boto/connection.py", line
1100, in __init__
provider=provider)
File "/usr/local/lib/python3.4/dist-packages/boto/connection.py", line
569, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/local/lib/python3.4/dist-packages/boto/auth.py", line 1021, in
get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1
handlers were checked. ['HmacAuthV4Handler'] Check your credentials
-N
nik.hodgkinson@collectivehealth.com
On Tue, Aug 15, 2017 at 2:58 PM, Nicholas Hodgkinson <
nik.hodgkinson@collectivehealth.com> wrote:
> I'm wondering if anyone has had any experience setting up Airflow on AWS
> using SQS as the Celery backend. I've got an instance credential set but
> I'm getting this error on the worker:
>
> boto.exception.NoAuthHandlerFound: No handler was ready to authenticate.
> 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials
>
> Wondering if anyone has any insight.
>
> Thanks,
> -Nik
> nik.hodgkinson@collectivehealth.com
>
>
--
See why we are one of WSJ's Top Tech Companies to Watch
<https://www.wsj.com/articles/collective-healths-goal-medical-benefits-that-work-1497492300?mg=prod/accounts-wsj>
*This message may contain confidential, proprietary, or protected
information. If you are not the intended recipient, you may not review,
copy, or distribute this message. If you received this message in error,
please notify the sender by reply email and delete this message.*