You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jason Chen <ch...@gmail.com> on 2016/09/01 05:49:44 UTC
Airflow webserver responses slow intermittently
Hi airflow team,
I am using airflow v1.7.1.3 (gunicorn 19.3.0)
It seems airflow webserver responses slow intermittently.
It sometimes takes time to just go to the home page of airflow DAG UI.
I tried to start webserver in debug mode (`airflow webserver -d`) and can
see the client/UI requests. not saw unusual messages in debug mode though.
Any suggestions ?
Thanks.
-Jason
Note scheduler is running, just webserver is slow. We only have one DAG.
BTW, in the webserver log, we see following messages sometimes (not having
such messages when webserver responses slow though).
[2016-09-01 04:54:43 +0000] [14158] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/gunicorn/workers/sync.py",
line 129, in handle
req = six.next(parser)
File "/usr/local/lib/python2.7/site-packages/gunicorn/http/parser.py",
line 41, in __next__
self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
line 153, in __init__
super(Request, self).__init__(cfg, unreader)
File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
line 53, in __init__
unused = self.parse(self.unreader)
File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
line 165, in parse
self.get_data(unreader, buf, stop=True)
File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
line 156, in get_data
data = unreader.read()
File "/usr/local/lib/python2.7/site-packages/gunicorn/http/unreader.py",
line 38, in read
d = self.chunk()
File "/usr/local/lib/python2.7/site-packages/gunicorn/http/unreader.py",
line 65, in chunk
return self.sock.recv(self.mxchunk)
error: [Errno 4] Interrupted system call
Re: Airflow webserver responses slow intermittently
Posted by Jason Chen <ch...@gmail.com>.
Hi Koen,
Thanks for your feedback.
Understood.
Yeah, I have tried several things and I do plan to upgrade our MySQL to
see if it helps.
Thanks again!
Jason
On Fri, Sep 2, 2016 at 11:52 AM, Koen Mevissen <km...@travix.com> wrote:
> Hey Jason,
>
> We encountered similar behavior on our GCE instance which used a 1st
> generation Cloud SQL as back-end. I looked into this today and replaced the
> 1st generation with a 2nd generation instance and that seems to have
> resolved it. In our case, most likely it was related to long running /
> unused TCP connections getting closed which seem to have affected the web
> session. This is just a bit of a wild guess though, haven't debugged this.
>
> HTH,
>
> Koen
>
>
>
> On Thu, Sep 1, 2016 at 7:49 AM, Jason Chen <ch...@gmail.com>
> wrote:
>
> > Hi airflow team,
> > I am using airflow v1.7.1.3 (gunicorn 19.3.0)
> >
> > It seems airflow webserver responses slow intermittently.
> > It sometimes takes time to just go to the home page of airflow DAG UI.
> >
> > I tried to start webserver in debug mode (`airflow webserver -d`) and can
> > see the client/UI requests. not saw unusual messages in debug mode
> though.
> >
> > Any suggestions ?
> >
> > Thanks.
> >
> > -Jason
> >
> > Note scheduler is running, just webserver is slow. We only have one DAG.
> >
> > BTW, in the webserver log, we see following messages sometimes (not
> having
> > such messages when webserver responses slow though).
> >
> >
> > [2016-09-01 04:54:43 +0000] [14158] [ERROR] Socket error processing
> > request.
> > Traceback (most recent call last):
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/
> workers/sync.py",
> > line 129, in handle
> > req = six.next(parser)
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/http/parser.py",
> > line 41, in __next__
> > self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/http/
> message.py",
> > line 153, in __init__
> > super(Request, self).__init__(cfg, unreader)
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/http/
> message.py",
> > line 53, in __init__
> > unused = self.parse(self.unreader)
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/http/
> message.py",
> > line 165, in parse
> > self.get_data(unreader, buf, stop=True)
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/http/
> message.py",
> > line 156, in get_data
> > data = unreader.read()
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/http/
> unreader.py",
> > line 38, in read
> > d = self.chunk()
> > File "/usr/local/lib/python2.7/site-packages/gunicorn/http/
> unreader.py",
> > line 65, in chunk
> > return self.sock.recv(self.mxchunk)
> > error: [Errno 4] Interrupted system call
> >
>
>
>
> --
> Kind regards,
> Met vriendelijke groet,
>
> *Koen Mevissen*
> Senior BI Developer
>
>
> *Travix Nederland B.V.*
> Piet Heinkade 55
> 1019 GM Amsterdam
> The Netherlands
>
> T. +31 (0)20 203 3241
> M. +31 (0)6 12345678
> E: KMevissen@travix.com
> www.travix.com
>
> *Brands: * CheapTickets | Vliegwinkel | Vayama | BudgetAir |
> Flugladen
>
Re: Airflow webserver responses slow intermittently
Posted by Koen Mevissen <km...@travix.com>.
Hey Jason,
We encountered similar behavior on our GCE instance which used a 1st
generation Cloud SQL as back-end. I looked into this today and replaced the
1st generation with a 2nd generation instance and that seems to have
resolved it. In our case, most likely it was related to long running /
unused TCP connections getting closed which seem to have affected the web
session. This is just a bit of a wild guess though, haven't debugged this.
HTH,
Koen
On Thu, Sep 1, 2016 at 7:49 AM, Jason Chen <ch...@gmail.com>
wrote:
> Hi airflow team,
> I am using airflow v1.7.1.3 (gunicorn 19.3.0)
>
> It seems airflow webserver responses slow intermittently.
> It sometimes takes time to just go to the home page of airflow DAG UI.
>
> I tried to start webserver in debug mode (`airflow webserver -d`) and can
> see the client/UI requests. not saw unusual messages in debug mode though.
>
> Any suggestions ?
>
> Thanks.
>
> -Jason
>
> Note scheduler is running, just webserver is slow. We only have one DAG.
>
> BTW, in the webserver log, we see following messages sometimes (not having
> such messages when webserver responses slow though).
>
>
> [2016-09-01 04:54:43 +0000] [14158] [ERROR] Socket error processing
> request.
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/site-packages/gunicorn/workers/sync.py",
> line 129, in handle
> req = six.next(parser)
> File "/usr/local/lib/python2.7/site-packages/gunicorn/http/parser.py",
> line 41, in __next__
> self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
> File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
> line 153, in __init__
> super(Request, self).__init__(cfg, unreader)
> File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
> line 53, in __init__
> unused = self.parse(self.unreader)
> File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
> line 165, in parse
> self.get_data(unreader, buf, stop=True)
> File "/usr/local/lib/python2.7/site-packages/gunicorn/http/message.py",
> line 156, in get_data
> data = unreader.read()
> File "/usr/local/lib/python2.7/site-packages/gunicorn/http/unreader.py",
> line 38, in read
> d = self.chunk()
> File "/usr/local/lib/python2.7/site-packages/gunicorn/http/unreader.py",
> line 65, in chunk
> return self.sock.recv(self.mxchunk)
> error: [Errno 4] Interrupted system call
>
--
Kind regards,
Met vriendelijke groet,
*Koen Mevissen*
Senior BI Developer
*Travix Nederland B.V.*
Piet Heinkade 55
1019 GM Amsterdam
The Netherlands
T. +31 (0)20 203 3241
M. +31 (0)6 12345678
E: KMevissen@travix.com
www.travix.com
*Brands: * CheapTickets | Vliegwinkel | Vayama | BudgetAir |
Flugladen