You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Sai Phanindhra <ph...@gmail.com> on 2018/09/25 09:48:18 UTC
Tail log instead of serving all logs for long running tasks/huge log volume
Hi guys,
Now we serve whole log file irrespective of its size. Sometimes when i
log size if few Mb(more than some 5mb or so) we end up rendering whole 5mb
file. This sometimes lead to huge payloads moving across network and after
rendering user have to scroll down thousands and thousands of lines.
Instead if we just tail last few hundreds of lines we can address both of
the above mentioned issues.
Can we have real time logging instead of reloading page again and again
for logs.
--
Sai Phanindhra,
Ph: +91 9043258999
Re: Tail log instead of serving all logs for long running tasks/huge
log volume
Posted by Sai Phanindhra <ph...@gmail.com>.
I wanted to work on it. But confirming did someone tried it already and had
some practical difficulties/limitations on not going ahead with and all.
On Tue 25 Sep, 2018, 16:13 Sumit Maheshwari, <su...@gmail.com> wrote:
> Yeah, that would be the next step. Maybe you can build that on top of that
> PR and raise a new PR?
>
> On Tue, Sep 25, 2018 at 3:41 PM Sai Phanindhra <ph...@gmail.com>
> wrote:
>
> > Hi sumit,
> > Along with a refresh log button if we can tail last N lines it will be
> more
> > useful. We can add filter on view to show N lines but lets implement it
> on
> > worker end to fetch last N lines.
> >
> > On Tue 25 Sep, 2018, 15:37 Sumit Maheshwari, <su...@gmail.com>
> > wrote:
> >
> > > Tried to solve that problem (partially) with below PR
> > >
> > > https://github.com/apache/incubator-airflow/pull/1875
> > >
> > > On Tue, Sep 25, 2018 at 3:18 PM Sai Phanindhra <ph...@gmail.com>
> > > wrote:
> > >
> > > > Hi guys,
> > > > Now we serve whole log file irrespective of its size. Sometimes
> > when i
> > > > log size if few Mb(more than some 5mb or so) we end up rendering
> whole
> > > 5mb
> > > > file. This sometimes lead to huge payloads moving across network and
> > > after
> > > > rendering user have to scroll down thousands and thousands of lines.
> > > > Instead if we just tail last few hundreds of lines we can address
> both
> > of
> > > > the above mentioned issues.
> > > > Can we have real time logging instead of reloading page again and
> > again
> > > > for logs.
> > > >
> > > > --
> > > > Sai Phanindhra,
> > > > Ph: +91 9043258999
> > > >
> > >
> >
>
Re: Tail log instead of serving all logs for long running tasks/huge
log volume
Posted by Sumit Maheshwari <su...@gmail.com>.
Yeah, that would be the next step. Maybe you can build that on top of that
PR and raise a new PR?
On Tue, Sep 25, 2018 at 3:41 PM Sai Phanindhra <ph...@gmail.com> wrote:
> Hi sumit,
> Along with a refresh log button if we can tail last N lines it will be more
> useful. We can add filter on view to show N lines but lets implement it on
> worker end to fetch last N lines.
>
> On Tue 25 Sep, 2018, 15:37 Sumit Maheshwari, <su...@gmail.com>
> wrote:
>
> > Tried to solve that problem (partially) with below PR
> >
> > https://github.com/apache/incubator-airflow/pull/1875
> >
> > On Tue, Sep 25, 2018 at 3:18 PM Sai Phanindhra <ph...@gmail.com>
> > wrote:
> >
> > > Hi guys,
> > > Now we serve whole log file irrespective of its size. Sometimes
> when i
> > > log size if few Mb(more than some 5mb or so) we end up rendering whole
> > 5mb
> > > file. This sometimes lead to huge payloads moving across network and
> > after
> > > rendering user have to scroll down thousands and thousands of lines.
> > > Instead if we just tail last few hundreds of lines we can address both
> of
> > > the above mentioned issues.
> > > Can we have real time logging instead of reloading page again and
> again
> > > for logs.
> > >
> > > --
> > > Sai Phanindhra,
> > > Ph: +91 9043258999
> > >
> >
>
Re: Tail log instead of serving all logs for long running tasks/huge
log volume
Posted by Sai Phanindhra <ph...@gmail.com>.
Hi sumit,
Along with a refresh log button if we can tail last N lines it will be more
useful. We can add filter on view to show N lines but lets implement it on
worker end to fetch last N lines.
On Tue 25 Sep, 2018, 15:37 Sumit Maheshwari, <su...@gmail.com> wrote:
> Tried to solve that problem (partially) with below PR
>
> https://github.com/apache/incubator-airflow/pull/1875
>
> On Tue, Sep 25, 2018 at 3:18 PM Sai Phanindhra <ph...@gmail.com>
> wrote:
>
> > Hi guys,
> > Now we serve whole log file irrespective of its size. Sometimes when i
> > log size if few Mb(more than some 5mb or so) we end up rendering whole
> 5mb
> > file. This sometimes lead to huge payloads moving across network and
> after
> > rendering user have to scroll down thousands and thousands of lines.
> > Instead if we just tail last few hundreds of lines we can address both of
> > the above mentioned issues.
> > Can we have real time logging instead of reloading page again and again
> > for logs.
> >
> > --
> > Sai Phanindhra,
> > Ph: +91 9043258999
> >
>
Re: Tail log instead of serving all logs for long running tasks/huge
log volume
Posted by Sumit Maheshwari <su...@gmail.com>.
Tried to solve that problem (partially) with below PR
https://github.com/apache/incubator-airflow/pull/1875
On Tue, Sep 25, 2018 at 3:18 PM Sai Phanindhra <ph...@gmail.com> wrote:
> Hi guys,
> Now we serve whole log file irrespective of its size. Sometimes when i
> log size if few Mb(more than some 5mb or so) we end up rendering whole 5mb
> file. This sometimes lead to huge payloads moving across network and after
> rendering user have to scroll down thousands and thousands of lines.
> Instead if we just tail last few hundreds of lines we can address both of
> the above mentioned issues.
> Can we have real time logging instead of reloading page again and again
> for logs.
>
> --
> Sai Phanindhra,
> Ph: +91 9043258999
>