You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by jean garutti <la...@yahoo.fr> on 2014/09/26 14:17:23 UTC
Better file sink
Hello,
For differents purposes, i want to store the events on local files and els/hdfs.
I checked the fileRoll sink but it's very limited in fucntionnality.
I can't specify a dynamic path, rolling, timeidle like the hdfs sink.
Do you think that it can be easy implemented ?
jean
Re: Better file sink
Posted by Jean <la...@yahoo.fr>.
Ok so perfect !!
It would be nice to have these details in the flume manual.
I don't think i'm the only one who need this.
Thanks all for your help, i have many other questions like auto retry between flume agent.
I'll post those to this list.
> Le 26 sept. 2014 à 20:41, Joey Echeverria <jo...@cloudera.com> a écrit :
>
> Those would be ignored. The sink uses the FileSystem interface and
> when you use file:// URL it will use the LocalFileSystem
> implementation so HDFS-specific configuration will just get safely
> ignored.
>
> -Joey
>
>> On Fri, Sep 26, 2014 at 11:28 AM, Jean <la...@yahoo.fr> wrote:
>> Sounds interesting. I'll test it out.
>> But what about the parameters like minblockreplicas and the others specific to hdfs ?
>>
>>> Le 26 sept. 2014 à 16:31, Joey Echeverria <jo...@cloudera.com> a écrit :
>>>
>>> The HDFS sink can also write to the local file system. Use a file://
>>> URL for the hdfs.path.
>>>
>>> -Joey
>>>
>>>> On Fri, Sep 26, 2014 at 6:14 AM, Sharninder <sh...@gmail.com> wrote:
>>>> Implementing a new sink isn't that difficult really so if your requirements
>>>> aren't met with the current file sink, by all means write your own. Take a
>>>> look at the existing code and the Dev guide.
>>>>
>>>> On 26-Sep-2014, at 5:47 pm, jean garutti <la...@yahoo.fr> wrote:
>>>>
>>>> Hello,
>>>> For differents purposes, i want to store the events on local files and
>>>> els/hdfs.
>>>> I checked the fileRoll sink but it's very limited in fucntionnality.
>>>> I can't specify a dynamic path, rolling, timeidle like the hdfs sink.
>>>>
>>>> Do you think that it can be easy implemented ?
>>>>
>>>> jean
>>>
>>>
>>>
>>> --
>>> Joey Echeverria
>
>
>
> --
> Joey Echeverria
Re: Better file sink
Posted by Joey Echeverria <jo...@cloudera.com>.
Those would be ignored. The sink uses the FileSystem interface and
when you use file:// URL it will use the LocalFileSystem
implementation so HDFS-specific configuration will just get safely
ignored.
-Joey
On Fri, Sep 26, 2014 at 11:28 AM, Jean <la...@yahoo.fr> wrote:
> Sounds interesting. I'll test it out.
> But what about the parameters like minblockreplicas and the others specific to hdfs ?
>
>> Le 26 sept. 2014 à 16:31, Joey Echeverria <jo...@cloudera.com> a écrit :
>>
>> The HDFS sink can also write to the local file system. Use a file://
>> URL for the hdfs.path.
>>
>> -Joey
>>
>>> On Fri, Sep 26, 2014 at 6:14 AM, Sharninder <sh...@gmail.com> wrote:
>>> Implementing a new sink isn't that difficult really so if your requirements
>>> aren't met with the current file sink, by all means write your own. Take a
>>> look at the existing code and the Dev guide.
>>>
>>> On 26-Sep-2014, at 5:47 pm, jean garutti <la...@yahoo.fr> wrote:
>>>
>>> Hello,
>>> For differents purposes, i want to store the events on local files and
>>> els/hdfs.
>>> I checked the fileRoll sink but it's very limited in fucntionnality.
>>> I can't specify a dynamic path, rolling, timeidle like the hdfs sink.
>>>
>>> Do you think that it can be easy implemented ?
>>>
>>> jean
>>
>>
>>
>> --
>> Joey Echeverria
--
Joey Echeverria
Re: Better file sink
Posted by Jean <la...@yahoo.fr>.
Sounds interesting. I'll test it out.
But what about the parameters like minblockreplicas and the others specific to hdfs ?
> Le 26 sept. 2014 à 16:31, Joey Echeverria <jo...@cloudera.com> a écrit :
>
> The HDFS sink can also write to the local file system. Use a file://
> URL for the hdfs.path.
>
> -Joey
>
>> On Fri, Sep 26, 2014 at 6:14 AM, Sharninder <sh...@gmail.com> wrote:
>> Implementing a new sink isn't that difficult really so if your requirements
>> aren't met with the current file sink, by all means write your own. Take a
>> look at the existing code and the Dev guide.
>>
>> On 26-Sep-2014, at 5:47 pm, jean garutti <la...@yahoo.fr> wrote:
>>
>> Hello,
>> For differents purposes, i want to store the events on local files and
>> els/hdfs.
>> I checked the fileRoll sink but it's very limited in fucntionnality.
>> I can't specify a dynamic path, rolling, timeidle like the hdfs sink.
>>
>> Do you think that it can be easy implemented ?
>>
>> jean
>
>
>
> --
> Joey Echeverria
Re: Better file sink
Posted by Joey Echeverria <jo...@cloudera.com>.
The HDFS sink can also write to the local file system. Use a file://
URL for the hdfs.path.
-Joey
On Fri, Sep 26, 2014 at 6:14 AM, Sharninder <sh...@gmail.com> wrote:
> Implementing a new sink isn't that difficult really so if your requirements
> aren't met with the current file sink, by all means write your own. Take a
> look at the existing code and the Dev guide.
>
> On 26-Sep-2014, at 5:47 pm, jean garutti <la...@yahoo.fr> wrote:
>
> Hello,
> For differents purposes, i want to store the events on local files and
> els/hdfs.
> I checked the fileRoll sink but it's very limited in fucntionnality.
> I can't specify a dynamic path, rolling, timeidle like the hdfs sink.
>
> Do you think that it can be easy implemented ?
>
> jean
--
Joey Echeverria
Re: Better file sink
Posted by Sharninder <sh...@gmail.com>.
Implementing a new sink isn't that difficult really so if your requirements aren't met with the current file sink, by all means write your own. Take a look at the existing code and the Dev guide.
> On 26-Sep-2014, at 5:47 pm, jean garutti <la...@yahoo.fr> wrote:
>
> Hello,
> For differents purposes, i want to store the events on local files and els/hdfs.
> I checked the fileRoll sink but it's very limited in fucntionnality.
> I can't specify a dynamic path, rolling, timeidle like the hdfs sink.
>
> Do you think that it can be easy implemented ?
>
> jean