Re: Flume HTTP Source Handler Request size limit

[Muhammad Yaseen <ya...@gmail.com>]

Hi,

Thanks for the suggestion. I will take a look at Blob Handler.

On Thu, Aug 31, 2017 at 1:28 PM, Archana Ravindran <sr...@gmail.com>
 wrote:

> You can try using blob handler in http source. Though blob is for binary,
> we were able to push relatively large event.
>
> For size limit, in our case we were not able to find default support in
> flume. So we customized the code with a small function for checking the
> size.
> You can try similar.
>
> Thanks,
> Archana.
>
>
> On Aug 31, 2017 12:29 PM, "Muhammad Yaseen" <ya...@gmail.com>
> wrote:
>
> Hello,
>
> I am using an http source for my Flume agent and would like to know if
> there a size limit (explicit or implicit) on POST request body/data
> (content length) for Flume's HTTP Source ?
>
> I have a handler which is throwing* "MalformedJsonException: Unterminated
> string at line 1 column 966657*" (number varies) . I know for a fact that
> the JSON is correct, for some reason the Handler is dropping the POST body.
> This leads to *"HTTPBadRequestException: Request has invalid JSON
> Syntax."*
>
> The source receives a huge request with an array of JSON objects as events
> (10000-30000 events per request body).
> This problem occurs when the content length ( request.getContentLength() )
> is large e.g. 2142675 but works fine when the length is smaller, e.g. 38320.
>
> Has anyone experienced this ?
>
> I am interested in the reason causing this problem.
>
> Regards,
> Muhammad Yaseen
>
>
On Thu, Aug 31, 2017 at 2:45 PM, Muhammad Yaseen <ya...@gmail.com>
wrote:

> Hi,
>
> Thanks for the suggestion. I will take a look at Blob Handler.
>
> On Thu, Aug 31, 2017 at 1:28 PM, Archana Ravindran <sr...@gmail.com>
> wrote:
>
>> You can try using blob handler in http source. Though blob is for binary,
>> we were able to push relatively large event.
>>
>> For size limit, in our case we were not able to find default support in
>> flume. So we customized the code with a small function for checking the
>> size.
>> You can try similar.
>>
>> Thanks,
>> Archana.
>>
>>
>> On Aug 31, 2017 12:29 PM, "Muhammad Yaseen" <ya...@gmail.com>
>> wrote:
>>
>> Hello,
>>
>> I am using an http source for my Flume agent and would like to know if
>> there a size limit (explicit or implicit) on POST request body/data
>> (content length) for Flume's HTTP Source ?
>>
>> I have a handler which is throwing* "MalformedJsonException:
>> Unterminated string at line 1 column 966657*" (number varies) . I know
>> for a fact that the JSON is correct, for some reason the Handler is
>> dropping the POST body. This leads to *"HTTPBadRequestException: Request
>> has invalid JSON Syntax."*
>>
>> The source receives a huge request with an array of JSON objects as
>> events (10000-30000 events per request body).
>> This problem occurs when the content length ( request.getContentLength()
>> ) is large e.g. 2142675 but works fine when the length is smaller,
>> e.g. 38320.
>>
>> Has anyone experienced this ?
>>
>> I am interested in the reason causing this problem.
>>
>> Regards,
>> Muhammad Yaseen
>>
>>
>>
>