You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Muhammad Yaseen <ya...@gmail.com> on 2017/08/31 06:59:00 UTC
Flume HTTP Source Handler Request size limit
Hello,
I am using an http source for my Flume agent and would like to know if
there a size limit (explicit or implicit) on POST request body/data
(content length) for Flume's HTTP Source ?
I have a handler which is throwing* "MalformedJsonException: Unterminated
string at line 1 column 966657*" (number varies) . I know for a fact that
the JSON is correct, for some reason the Handler is dropping the POST body.
This leads to *"HTTPBadRequestException: Request has invalid JSON Syntax."*
The source receives a huge request with an array of JSON objects as events
(10000-30000 events per request body).
This problem occurs when the content length ( request.getContentLength() )
is large e.g. 2142675 but works fine when the length is smaller, e.g. 38320.
Has anyone experienced this ?
I am interested in the reason causing this problem.
Regards,
Muhammad Yaseen
Re: Flume HTTP Source Handler Request size limit
Posted by Muhammad Yaseen <ya...@gmail.com>.
Hi,
Thanks for the suggestion. I will take a look at Blob Handler.
On Thu, Aug 31, 2017 at 1:28 PM, Archana Ravindran <sr...@gmail.com>
wrote:
> You can try using blob handler in http source. Though blob is for binary,
> we were able to push relatively large event.
>
> For size limit, in our case we were not able to find default support in
> flume. So we customized the code with a small function for checking the
> size.
> You can try similar.
>
> Thanks,
> Archana.
>
>
> On Aug 31, 2017 12:29 PM, "Muhammad Yaseen" <ya...@gmail.com>
> wrote:
>
> Hello,
>
> I am using an http source for my Flume agent and would like to know if
> there a size limit (explicit or implicit) on POST request body/data
> (content length) for Flume's HTTP Source ?
>
> I have a handler which is throwing* "MalformedJsonException: Unterminated
> string at line 1 column 966657*" (number varies) . I know for a fact that
> the JSON is correct, for some reason the Handler is dropping the POST body.
> This leads to *"HTTPBadRequestException: Request has invalid JSON
> Syntax."*
>
> The source receives a huge request with an array of JSON objects as events
> (10000-30000 events per request body).
> This problem occurs when the content length ( request.getContentLength() )
> is large e.g. 2142675 but works fine when the length is smaller, e.g. 38320.
>
> Has anyone experienced this ?
>
> I am interested in the reason causing this problem.
>
> Regards,
> Muhammad Yaseen
>
>
On Thu, Aug 31, 2017 at 2:45 PM, Muhammad Yaseen <ya...@gmail.com>
wrote:
> Hi,
>
> Thanks for the suggestion. I will take a look at Blob Handler.
>
> On Thu, Aug 31, 2017 at 1:28 PM, Archana Ravindran <sr...@gmail.com>
> wrote:
>
>> You can try using blob handler in http source. Though blob is for binary,
>> we were able to push relatively large event.
>>
>> For size limit, in our case we were not able to find default support in
>> flume. So we customized the code with a small function for checking the
>> size.
>> You can try similar.
>>
>> Thanks,
>> Archana.
>>
>>
>> On Aug 31, 2017 12:29 PM, "Muhammad Yaseen" <ya...@gmail.com>
>> wrote:
>>
>> Hello,
>>
>> I am using an http source for my Flume agent and would like to know if
>> there a size limit (explicit or implicit) on POST request body/data
>> (content length) for Flume's HTTP Source ?
>>
>> I have a handler which is throwing* "MalformedJsonException:
>> Unterminated string at line 1 column 966657*" (number varies) . I know
>> for a fact that the JSON is correct, for some reason the Handler is
>> dropping the POST body. This leads to *"HTTPBadRequestException: Request
>> has invalid JSON Syntax."*
>>
>> The source receives a huge request with an array of JSON objects as
>> events (10000-30000 events per request body).
>> This problem occurs when the content length ( request.getContentLength()
>> ) is large e.g. 2142675 but works fine when the length is smaller,
>> e.g. 38320.
>>
>> Has anyone experienced this ?
>>
>> I am interested in the reason causing this problem.
>>
>> Regards,
>> Muhammad Yaseen
>>
>>
>>
>
Re: Flume HTTP Source Handler Request size limit
Posted by Muhammad Yaseen <ya...@gmail.com>.
Hi,
Thanks for the suggestion. I will take a look at Blob Handler.
On Thu, Aug 31, 2017 at 1:28 PM, Archana Ravindran <sr...@gmail.com>
wrote:
> You can try using blob handler in http source. Though blob is for binary,
> we were able to push relatively large event.
>
> For size limit, in our case we were not able to find default support in
> flume. So we customized the code with a small function for checking the
> size.
> You can try similar.
>
> Thanks,
> Archana.
>
>
> On Aug 31, 2017 12:29 PM, "Muhammad Yaseen" <ya...@gmail.com>
> wrote:
>
> Hello,
>
> I am using an http source for my Flume agent and would like to know if
> there a size limit (explicit or implicit) on POST request body/data
> (content length) for Flume's HTTP Source ?
>
> I have a handler which is throwing* "MalformedJsonException: Unterminated
> string at line 1 column 966657*" (number varies) . I know for a fact that
> the JSON is correct, for some reason the Handler is dropping the POST body.
> This leads to *"HTTPBadRequestException: Request has invalid JSON
> Syntax."*
>
> The source receives a huge request with an array of JSON objects as events
> (10000-30000 events per request body).
> This problem occurs when the content length ( request.getContentLength() )
> is large e.g. 2142675 but works fine when the length is smaller, e.g. 38320.
>
> Has anyone experienced this ?
>
> I am interested in the reason causing this problem.
>
> Regards,
> Muhammad Yaseen
>
>
>
Re: Flume HTTP Source Handler Request size limit
Posted by Archana Ravindran <sr...@gmail.com>.
You can try using blob handler in http source. Though blob is for binary,
we were able to push relatively large event.
For size limit, in our case we were not able to find default support in
flume. So we customized the code with a small function for checking the
size.
You can try similar.
Thanks,
Archana.
On Aug 31, 2017 12:29 PM, "Muhammad Yaseen" <ya...@gmail.com>
wrote:
Hello,
I am using an http source for my Flume agent and would like to know if
there a size limit (explicit or implicit) on POST request body/data
(content length) for Flume's HTTP Source ?
I have a handler which is throwing* "MalformedJsonException: Unterminated
string at line 1 column 966657*" (number varies) . I know for a fact that
the JSON is correct, for some reason the Handler is dropping the POST body.
This leads to *"HTTPBadRequestException: Request has invalid JSON Syntax."*
The source receives a huge request with an array of JSON objects as events
(10000-30000 events per request body).
This problem occurs when the content length ( request.getContentLength() )
is large e.g. 2142675 but works fine when the length is smaller, e.g. 38320.
Has anyone experienced this ?
I am interested in the reason causing this problem.
Regards,
Muhammad Yaseen