You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@livy.apache.org by Sadha Chilukoori <sa...@gmail.com> on 2023/08/09 14:49:11 UTC

API rate limit

Hi folks,

Hope you are all doing well.

My company is using Livy to submit spark jobs on AWS. On one project where
we are submitting the jobs to compact data at about one job per minute per
Jenkins job, we are getting 40x errors from Livy. There are about 20 such
Jenkins tasks running at the time. When we restart the Jenkins task, it
continues to submit jobs successfully.

I'm wondering if there is a rate limit in the default configuration of
Livy. Is this the case? Could you please point me to where this can be
updated.

Thank you,
Sadha

Re: API rate limit

Posted by Sadha Chilukoori <sa...@gmail.com>.

Thank you Marco. This looks promising, I'll change e the default value and
see if that resolves the issue.

-Sadha

On Fri, Aug 11, 2023, 1:03 PM Marco Gaido <ma...@gmail.com> wrote:

> Hi,
>
> please check https://issues.apache.org/jira/browse/LIVY-412.
>
> Thanks,
> Marco
>
> Il giorno mer 9 ago 2023 alle ore 16:51 Sadha Chilukoori <
> sage.quotient@gmail.com> ha scritto:
>
> > Hi folks,
> >
> > Hope you are all doing well.
> >
> > My company is using Livy to submit spark jobs on AWS. On one project
> where
> > we are submitting the jobs to compact data at about one job per minute
> per
> > Jenkins job, we are getting 40x errors from Livy. There are about 20 such
> > Jenkins tasks running at the time. When we restart the Jenkins task, it
> > continues to submit jobs successfully.
> >
> > I'm wondering if there is a rate limit in the default configuration of
> > Livy. Is this the case? Could you please point me to where this can be
> > updated.
> >
> > Thank you,
> > Sadha
> >
>

Re: API rate limit

Posted by Sadha Chilukoori <sa...@gmail.com>.

Thank you Marco. This looks promising, I'll change e the default value and
see if that resolves the issue.

-Sadha

On Fri, Aug 11, 2023, 1:03 PM Marco Gaido <ma...@gmail.com> wrote:

> Hi,
>
> please check https://issues.apache.org/jira/browse/LIVY-412.
>
> Thanks,
> Marco
>
> Il giorno mer 9 ago 2023 alle ore 16:51 Sadha Chilukoori <
> sage.quotient@gmail.com> ha scritto:
>
> > Hi folks,
> >
> > Hope you are all doing well.
> >
> > My company is using Livy to submit spark jobs on AWS. On one project
> where
> > we are submitting the jobs to compact data at about one job per minute
> per
> > Jenkins job, we are getting 40x errors from Livy. There are about 20 such
> > Jenkins tasks running at the time. When we restart the Jenkins task, it
> > continues to submit jobs successfully.
> >
> > I'm wondering if there is a rate limit in the default configuration of
> > Livy. Is this the case? Could you please point me to where this can be
> > updated.
> >
> > Thank you,
> > Sadha
> >
>

Re: API rate limit

Posted by Marco Gaido <ma...@gmail.com>.

Hi,

please check https://issues.apache.org/jira/browse/LIVY-412.

Thanks,
Marco

Il giorno mer 9 ago 2023 alle ore 16:51 Sadha Chilukoori <
sage.quotient@gmail.com> ha scritto:

> Hi folks,
>
> Hope you are all doing well.
>
> My company is using Livy to submit spark jobs on AWS. On one project where
> we are submitting the jobs to compact data at about one job per minute per
> Jenkins job, we are getting 40x errors from Livy. There are about 20 such
> Jenkins tasks running at the time. When we restart the Jenkins task, it
> continues to submit jobs successfully.
>
> I'm wondering if there is a rate limit in the default configuration of
> Livy. Is this the case? Could you please point me to where this can be
> updated.
>
> Thank you,
> Sadha
>

Re: API rate limit

Posted by Marco Gaido <ma...@gmail.com>.

Hi,

please check https://issues.apache.org/jira/browse/LIVY-412.

Thanks,
Marco

Il giorno mer 9 ago 2023 alle ore 16:51 Sadha Chilukoori <
sage.quotient@gmail.com> ha scritto:

> Hi folks,
>
> Hope you are all doing well.
>
> My company is using Livy to submit spark jobs on AWS. On one project where
> we are submitting the jobs to compact data at about one job per minute per
> Jenkins job, we are getting 40x errors from Livy. There are about 20 such
> Jenkins tasks running at the time. When we restart the Jenkins task, it
> continues to submit jobs successfully.
>
> I'm wondering if there is a rate limit in the default configuration of
> Livy. Is this the case? Could you please point me to where this can be
> updated.
>
> Thank you,
> Sadha
>