Flume JSON netcat source

[jean garutti <la...@yahoo.fr>]

Hi,I want flume to listen to JSON event send by netcat.But the event is not correct because each new line result in a new flume event.
How can i send json event to flume by tcp/udp and have flume handle correctly the json format ?
thanks

Re: Flume JSON netcat source

[Joey Echeverria <jo...@cloudera.com>]

The ideal would be if your app could strip the newlines before writing
to the logs. If that's not possible, you need a script that will tail
the logs and parse the JSON (or at least parse it enough to find the
beginning and ending curly braces) and strip out the new line
characters.

Using morphlines to go from JSON to Avro is a good idea.

-Joey

On Sun, Dec 14, 2014 at 9:13 AM, Jean <la...@yahoo.fr> wrote:
> Ok,  my app logs json event in local files and i also can pipe  to nc or logger so no avro.
>
> I'll check too if i can strip out new lines (with nc)
>
> Do you think tail files  with flume (with morphline to parse json) is a good way ?
>
>
>
>
>> Le 12 déc. 2014 à 19:05, Joey Echeverria <jo...@cloudera.com> a écrit :
>>
>> You'd either have to write a new source or use one of the RPC sources
>> and send the JSON in a container. For example, you could use the Avro
>> source to send just a string with the content of the  JSON. You'll
>> have to deserialize somewhere else in your flow.
>>
>> Alternatively, you could pre-process the JSON to strip out new lines
>> before sending them to the netcat source.
>>
>> -Joey
>>
>>> On Fri, Dec 12, 2014 at 12:01 AM, jean garutti <la...@yahoo.fr> wrote:
>>> Hi,
>>> I want flume to listen to JSON event send by netcat.
>>> But the event is not correct because each new line result in a new flume
>>> event.
>>>
>>> How can i send json event to flume by tcp/udp and have flume handle
>>> correctly the json format ?
>>>
>>> thanks
>>
>>
>>
>> --
>> Joey Echeverria



-- 
Joey Echeverria

Re: Flume JSON netcat source

[Joey Echeverria <jo...@cloudera.com>]

You'd either have to write a new source or use one of the RPC sources
and send the JSON in a container. For example, you could use the Avro
source to send just a string with the content of the  JSON. You'll
have to deserialize somewhere else in your flow.

Alternatively, you could pre-process the JSON to strip out new lines
before sending them to the netcat source.

-Joey

On Fri, Dec 12, 2014 at 12:01 AM, jean garutti <la...@yahoo.fr> wrote:
> Hi,
> I want flume to listen to JSON event send by netcat.
> But the event is not correct because each new line result in a new flume
> event.
>
> How can i send json event to flume by tcp/udp and have flume handle
> correctly the json format ?
>
> thanks



-- 
Joey Echeverria