You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Guillaume Polaert <gp...@cyres.fr> on 2012/05/21 15:03:57 UTC
RollingFile Sink problems
Hi everybody,
I've problems with RollingFile sink.
First, I'm seeking about the sink's properties. I've found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
Then, I've this error when I'm running the node :
...
SINKS: {sink=ComponentConfiguration[sink]
CONFIG: {directory=/tmp/log, type=file_roll}
RUNNER: ComponentConfiguration[runner]
CONFIG: {}
, log-sink1=ComponentConfiguration[log-sink1]
CONFIG: {type=logger, channel=ch1}
RUNNER: ComponentConfiguration[runner]
CONFIG: {}
}
...
2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:69)] Creating instance of sink sink typefile_roll
2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
java.lang.IllegalArgumentException: Directory may not be null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
...
Flume.conf
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414
# Define a logger sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = logger
agent1.sinks.sink.channel = ch1
agent1.sinks.sink.type = file_roll
agent1.sinks.sink.directory=/tmp/log
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = sink
What is wrong ? I don't understand the problem.
Guillaume
Re: RollingFile Sink problems
Posted by alo alt <wg...@googlemail.com>.
The .sink was in the main config available, but the identifier not.
@ Guillaume
agent1.sinks.file-sink-1.sink.channel = ch1
agent1.sinks.file-sink-1.sink.directory = /foo/bar
agent1.sinks.file-sink-1.sink.type = file_roll
works.
Not the "file-sink-1" identifier after .sinks.
I was reopened the Jira, but it is a doc gap, so I close again (sorry Mike). I'll file a doc jira about.
Thanks
- Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
On May 22, 2012, at 10:11 AM, Mike Percy wrote:
> Some of the flume NG config options are a little strange. This is one of them.
>
> You have to specify agent.sinks.file-sink.sink.directory = /some/dir
>
> It doesn't make a lot of sense, but it's historical.
>
> HTH,
> Mike
>
> On Monday, May 21, 2012 at 11:57 PM, Guillaume Polaert wrote:
>
>> Thank you for helping me out.
>>
>> Guillaume Polaert | Cyrès Conseil
>>
>>
>> -----Message d'origine-----
>> De : alo alt [mailto:wget.null@googlemail.com]
>> Envoyé : mardi 22 mai 2012 08:34
>> À : flume-user@incubator.apache.org
>> Objet : Re: RollingFile Sink problems
>>
>> created https://issues.apache.org/jira/browse/FLUME-1222
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>> On May 21, 2012, at 6:27 PM, alo alt wrote:
>>
>>> Looks like a config.parser issue in RollingFileSink.java or Configuration.java. Please open a Jira for.
>>> The error messages comes from the Preconditions check (Google Guava library) in RollingFileSink.java, indeed the variable for the directory is empty.
>>>
>>> - Alex
>>>
>>> --
>>> Alexander Alten-Lorenz
>>> http://mapredit.blogspot.com
>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>
>>> On May 21, 2012, at 3:50 PM, Guillaume Polaert wrote:
>>>
>>>> Thank you for the link.
>>>> I've created a empty directory /tmp/log (mkdir+chmod).
>>>>
>>>> Do I have to create other things ?
>>>>
>>>> -----Message d'origine-----
>>>> De : alo alt [mailto:wget.null@googlemail.com] Envoyé : lundi 21 mai
>>>> 2012 15:31 À : flume-user@incubator.apache.org Objet : Re:
>>>> RollingFile Sink problems
>>>>
>>>> Hi,
>>>>
>>>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/Flum
>>>> eUserGuide.html#h.frlqql6vu68
>>>>
>>>> Means, /tmp/log is empty or non existent, isn't it?
>>>>
>>>> - Alex
>>>>
>>>> --
>>>> Alexander Alten-Lorenz
>>>> http://mapredit.blogspot.com
>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>>>
>>>> On May 21, 2012, at 3:03 PM, Guillaume Polaert wrote:
>>>>
>>>>> Hi everybody,
>>>>>
>>>>> I've problems with RollingFile sink.
>>>>> First, I'm seeking about the sink's properties. I've found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
>>>>>
>>>>> Then, I've this error when I'm running the node :
>>>>>
>>>>> .
>>>>> SINKS: {sink=ComponentConfiguration[sink]
>>>>> CONFIG: {directory=/tmp/log, type=file_roll}
>>>>> RUNNER: ComponentConfiguration[runner]
>>>>> CONFIG: {}
>>>>>
>>>>>
>>>>> , log-sink1=ComponentConfiguration[log-sink1]
>>>>> CONFIG: {type=logger, channel=ch1}
>>>>> RUNNER: ComponentConfiguration[runner]
>>>>> CONFIG: {}
>>>>>
>>>>> }
>>>>> .
>>>>> 2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO -
>>>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.j
>>>>> av a:69)] Creating instance of sink sink typefile_roll
>>>>> 2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
>>>>> java.lang.IllegalArgumentException: Directory may not be null
>>>>> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>>>>> at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
>>>>> at
>>>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>>>> .
>>>>>
>>>>> Flume.conf
>>>>>
>>>>> # Define a memory channel called ch1 on agent1
>>>>> agent1.channels.ch1.type = memory
>>>>>
>>>>> # Define an Avro source called avro-source1 on agent1 and tell it #
>>>>> to bind to 0.0.0.0:41414. Connect it to channel ch1.
>>>>> agent1.sources.avro-source1.channels = ch1
>>>>> agent1.sources.avro-source1.type = avro
>>>>> agent1.sources.avro-source1.bind = 0.0.0.0
>>>>> agent1.sources.avro-source1.port = 41414
>>>>>
>>>>> # Define a logger sink that simply logs all events it receives # and
>>>>> connect it to the other end of the same channel.
>>>>> agent1.sinks.log-sink1.channel = ch1 agent1.sinks.log-sink1.type =
>>>>> logger agent1.sinks.sink.channel = ch1 agent1.sinks.sink.type =
>>>>> file_roll agent1.sinks.sink.directory=/tmp/log
>>>>>
>>>>> # Finally, now that we've defined all of our components, tell #
>>>>> agent1 which ones we want to activate.
>>>>> agent1.channels = ch1
>>>>> agent1.sources = avro-source1
>>>>> agent1.sinks = sink
>>>>>
>>>>> What is wrong ? I don't understand the problem.
>>>>>
>>>>> Guillaume
>
Re: RollingFile Sink problems
Posted by Mike Percy <mp...@cloudera.com>.
Some of the flume NG config options are a little strange. This is one of them.
You have to specify agent.sinks.file-sink.sink.directory = /some/dir
It doesn't make a lot of sense, but it's historical.
HTH,
Mike
On Monday, May 21, 2012 at 11:57 PM, Guillaume Polaert wrote:
> Thank you for helping me out.
>
> Guillaume Polaert | Cyrès Conseil
>
>
> -----Message d'origine-----
> De : alo alt [mailto:wget.null@googlemail.com]
> Envoyé : mardi 22 mai 2012 08:34
> À : flume-user@incubator.apache.org (mailto:flume-user@incubator.apache.org)
> Objet : Re: RollingFile Sink problems
>
> created https://issues.apache.org/jira/browse/FLUME-1222
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
> On May 21, 2012, at 6:27 PM, alo alt wrote:
>
> > Looks like a config.parser issue in RollingFileSink.java or Configuration.java. Please open a Jira for.
> > The error messages comes from the Preconditions check (Google Guava library) in RollingFileSink.java, indeed the variable for the directory is empty.
> >
> > - Alex
> >
> > --
> > Alexander Alten-Lorenz
> > http://mapredit.blogspot.com
> > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> >
> > On May 21, 2012, at 3:50 PM, Guillaume Polaert wrote:
> >
> > > Thank you for the link.
> > > I've created a empty directory /tmp/log (mkdir+chmod).
> > >
> > > Do I have to create other things ?
> > >
> > > -----Message d'origine-----
> > > De : alo alt [mailto:wget.null@googlemail.com] Envoyé : lundi 21 mai
> > > 2012 15:31 À : flume-user@incubator.apache.org (mailto:flume-user@incubator.apache.org) Objet : Re:
> > > RollingFile Sink problems
> > >
> > > Hi,
> > >
> > > http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/Flum
> > > eUserGuide.html#h.frlqql6vu68
> > >
> > > Means, /tmp/log is empty or non existent, isn't it?
> > >
> > > - Alex
> > >
> > > --
> > > Alexander Alten-Lorenz
> > > http://mapredit.blogspot.com
> > > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> > >
> > > On May 21, 2012, at 3:03 PM, Guillaume Polaert wrote:
> > >
> > > > Hi everybody,
> > > >
> > > > I've problems with RollingFile sink.
> > > > First, I'm seeking about the sink's properties. I've found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
> > > >
> > > > Then, I've this error when I'm running the node :
> > > >
> > > > .
> > > > SINKS: {sink=ComponentConfiguration[sink]
> > > > CONFIG: {directory=/tmp/log, type=file_roll}
> > > > RUNNER: ComponentConfiguration[runner]
> > > > CONFIG: {}
> > > >
> > > >
> > > > , log-sink1=ComponentConfiguration[log-sink1]
> > > > CONFIG: {type=logger, channel=ch1}
> > > > RUNNER: ComponentConfiguration[runner]
> > > > CONFIG: {}
> > > >
> > > > }
> > > > .
> > > > 2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO -
> > > > org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.j
> > > > av a:69)] Creating instance of sink sink typefile_roll
> > > > 2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
> > > > java.lang.IllegalArgumentException: Directory may not be null
> > > > at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> > > > at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
> > > > at
> > > > org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> > > > .
> > > >
> > > > Flume.conf
> > > >
> > > > # Define a memory channel called ch1 on agent1
> > > > agent1.channels.ch1.type = memory
> > > >
> > > > # Define an Avro source called avro-source1 on agent1 and tell it #
> > > > to bind to 0.0.0.0:41414. Connect it to channel ch1.
> > > > agent1.sources.avro-source1.channels = ch1
> > > > agent1.sources.avro-source1.type = avro
> > > > agent1.sources.avro-source1.bind = 0.0.0.0
> > > > agent1.sources.avro-source1.port = 41414
> > > >
> > > > # Define a logger sink that simply logs all events it receives # and
> > > > connect it to the other end of the same channel.
> > > > agent1.sinks.log-sink1.channel = ch1 agent1.sinks.log-sink1.type =
> > > > logger agent1.sinks.sink.channel = ch1 agent1.sinks.sink.type =
> > > > file_roll agent1.sinks.sink.directory=/tmp/log
> > > >
> > > > # Finally, now that we've defined all of our components, tell #
> > > > agent1 which ones we want to activate.
> > > > agent1.channels = ch1
> > > > agent1.sources = avro-source1
> > > > agent1.sinks = sink
> > > >
> > > > What is wrong ? I don't understand the problem.
> > > >
> > > > Guillaume
RE: RollingFile Sink problems
Posted by Guillaume Polaert <gp...@cyres.fr>.
Thank you for helping me out.
Guillaume Polaert | Cyrès Conseil
-----Message d'origine-----
De : alo alt [mailto:wget.null@googlemail.com]
Envoyé : mardi 22 mai 2012 08:34
À : flume-user@incubator.apache.org
Objet : Re: RollingFile Sink problems
created https://issues.apache.org/jira/browse/FLUME-1222
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
On May 21, 2012, at 6:27 PM, alo alt wrote:
> Looks like a config.parser issue in RollingFileSink.java or Configuration.java. Please open a Jira for.
> The error messages comes from the Preconditions check (Google Guava library) in RollingFileSink.java, indeed the variable for the directory is empty.
>
> - Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
> On May 21, 2012, at 3:50 PM, Guillaume Polaert wrote:
>
>> Thank you for the link.
>> I've created a empty directory /tmp/log (mkdir+chmod).
>>
>> Do I have to create other things ?
>>
>> -----Message d'origine-----
>> De : alo alt [mailto:wget.null@googlemail.com] Envoyé : lundi 21 mai
>> 2012 15:31 À : flume-user@incubator.apache.org Objet : Re:
>> RollingFile Sink problems
>>
>> Hi,
>>
>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/Flum
>> eUserGuide.html#h.frlqql6vu68
>>
>> Means, /tmp/log is empty or non existent, isn't it?
>>
>> - Alex
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>> On May 21, 2012, at 3:03 PM, Guillaume Polaert wrote:
>>
>>> Hi everybody,
>>>
>>> I've problems with RollingFile sink.
>>> First, I'm seeking about the sink's properties. I've found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
>>>
>>> Then, I've this error when I'm running the node :
>>>
>>> .
>>> SINKS: {sink=ComponentConfiguration[sink]
>>> CONFIG: {directory=/tmp/log, type=file_roll}
>>> RUNNER: ComponentConfiguration[runner]
>>> CONFIG: {}
>>>
>>>
>>> , log-sink1=ComponentConfiguration[log-sink1]
>>> CONFIG: {type=logger, channel=ch1}
>>> RUNNER: ComponentConfiguration[runner]
>>> CONFIG: {}
>>>
>>> }
>>> .
>>> 2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO -
>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.j
>>> av a:69)] Creating instance of sink sink typefile_roll
>>> 2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
>>> java.lang.IllegalArgumentException: Directory may not be null
>>> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>>> at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
>>> at
>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>> .
>>>
>>> Flume.conf
>>>
>>> # Define a memory channel called ch1 on agent1
>>> agent1.channels.ch1.type = memory
>>>
>>> # Define an Avro source called avro-source1 on agent1 and tell it #
>>> to bind to 0.0.0.0:41414. Connect it to channel ch1.
>>> agent1.sources.avro-source1.channels = ch1
>>> agent1.sources.avro-source1.type = avro
>>> agent1.sources.avro-source1.bind = 0.0.0.0
>>> agent1.sources.avro-source1.port = 41414
>>>
>>> # Define a logger sink that simply logs all events it receives # and
>>> connect it to the other end of the same channel.
>>> agent1.sinks.log-sink1.channel = ch1 agent1.sinks.log-sink1.type =
>>> logger agent1.sinks.sink.channel = ch1 agent1.sinks.sink.type =
>>> file_roll agent1.sinks.sink.directory=/tmp/log
>>>
>>> # Finally, now that we've defined all of our components, tell #
>>> agent1 which ones we want to activate.
>>> agent1.channels = ch1
>>> agent1.sources = avro-source1
>>> agent1.sinks = sink
>>>
>>> What is wrong ? I don't understand the problem.
>>>
>>> Guillaume
>>
>
Re: RollingFile Sink problems
Posted by alo alt <wg...@googlemail.com>.
created https://issues.apache.org/jira/browse/FLUME-1222
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
On May 21, 2012, at 6:27 PM, alo alt wrote:
> Looks like a config.parser issue in RollingFileSink.java or Configuration.java. Please open a Jira for.
> The error messages comes from the Preconditions check (Google Guava library) in RollingFileSink.java, indeed the variable for the directory is empty.
>
> - Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
> On May 21, 2012, at 3:50 PM, Guillaume Polaert wrote:
>
>> Thank you for the link.
>> I've created a empty directory /tmp/log (mkdir+chmod).
>>
>> Do I have to create other things ?
>>
>> -----Message d'origine-----
>> De : alo alt [mailto:wget.null@googlemail.com]
>> Envoyé : lundi 21 mai 2012 15:31
>> À : flume-user@incubator.apache.org
>> Objet : Re: RollingFile Sink problems
>>
>> Hi,
>>
>> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html#h.frlqql6vu68
>>
>> Means, /tmp/log is empty or non existent, isn't it?
>>
>> - Alex
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>> On May 21, 2012, at 3:03 PM, Guillaume Polaert wrote:
>>
>>> Hi everybody,
>>>
>>> I've problems with RollingFile sink.
>>> First, I'm seeking about the sink's properties. I've found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
>>>
>>> Then, I've this error when I'm running the node :
>>>
>>> .
>>> SINKS: {sink=ComponentConfiguration[sink]
>>> CONFIG: {directory=/tmp/log, type=file_roll}
>>> RUNNER: ComponentConfiguration[runner]
>>> CONFIG: {}
>>>
>>>
>>> , log-sink1=ComponentConfiguration[log-sink1]
>>> CONFIG: {type=logger, channel=ch1}
>>> RUNNER: ComponentConfiguration[runner]
>>> CONFIG: {}
>>>
>>> }
>>> .
>>> 2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO -
>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.jav
>>> a:69)] Creating instance of sink sink typefile_roll
>>> 2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
>>> java.lang.IllegalArgumentException: Directory may not be null
>>> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>>> at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
>>> at
>>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>>> .
>>>
>>> Flume.conf
>>>
>>> # Define a memory channel called ch1 on agent1
>>> agent1.channels.ch1.type = memory
>>>
>>> # Define an Avro source called avro-source1 on agent1 and tell it # to
>>> bind to 0.0.0.0:41414. Connect it to channel ch1.
>>> agent1.sources.avro-source1.channels = ch1
>>> agent1.sources.avro-source1.type = avro
>>> agent1.sources.avro-source1.bind = 0.0.0.0
>>> agent1.sources.avro-source1.port = 41414
>>>
>>> # Define a logger sink that simply logs all events it receives # and
>>> connect it to the other end of the same channel.
>>> agent1.sinks.log-sink1.channel = ch1
>>> agent1.sinks.log-sink1.type = logger
>>> agent1.sinks.sink.channel = ch1
>>> agent1.sinks.sink.type = file_roll
>>> agent1.sinks.sink.directory=/tmp/log
>>>
>>> # Finally, now that we've defined all of our components, tell # agent1
>>> which ones we want to activate.
>>> agent1.channels = ch1
>>> agent1.sources = avro-source1
>>> agent1.sinks = sink
>>>
>>> What is wrong ? I don't understand the problem.
>>>
>>> Guillaume
>>
>
Re: RollingFile Sink problems
Posted by alo alt <wg...@googlemail.com>.
Looks like a config.parser issue in RollingFileSink.java or Configuration.java. Please open a Jira for.
The error messages comes from the Preconditions check (Google Guava library) in RollingFileSink.java, indeed the variable for the directory is empty.
- Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
On May 21, 2012, at 3:50 PM, Guillaume Polaert wrote:
> Thank you for the link.
> I've created a empty directory /tmp/log (mkdir+chmod).
>
> Do I have to create other things ?
>
> -----Message d'origine-----
> De : alo alt [mailto:wget.null@googlemail.com]
> Envoyé : lundi 21 mai 2012 15:31
> À : flume-user@incubator.apache.org
> Objet : Re: RollingFile Sink problems
>
> Hi,
>
> http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html#h.frlqql6vu68
>
> Means, /tmp/log is empty or non existent, isn't it?
>
> - Alex
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
> On May 21, 2012, at 3:03 PM, Guillaume Polaert wrote:
>
>> Hi everybody,
>>
>> I've problems with RollingFile sink.
>> First, I'm seeking about the sink's properties. I've found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
>>
>> Then, I've this error when I'm running the node :
>>
>> .
>> SINKS: {sink=ComponentConfiguration[sink]
>> CONFIG: {directory=/tmp/log, type=file_roll}
>> RUNNER: ComponentConfiguration[runner]
>> CONFIG: {}
>>
>>
>> , log-sink1=ComponentConfiguration[log-sink1]
>> CONFIG: {type=logger, channel=ch1}
>> RUNNER: ComponentConfiguration[runner]
>> CONFIG: {}
>>
>> }
>> .
>> 2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO -
>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.jav
>> a:69)] Creating instance of sink sink typefile_roll
>> 2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
>> java.lang.IllegalArgumentException: Directory may not be null
>> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
>> at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
>> at
>> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>> .
>>
>> Flume.conf
>>
>> # Define a memory channel called ch1 on agent1
>> agent1.channels.ch1.type = memory
>>
>> # Define an Avro source called avro-source1 on agent1 and tell it # to
>> bind to 0.0.0.0:41414. Connect it to channel ch1.
>> agent1.sources.avro-source1.channels = ch1
>> agent1.sources.avro-source1.type = avro
>> agent1.sources.avro-source1.bind = 0.0.0.0
>> agent1.sources.avro-source1.port = 41414
>>
>> # Define a logger sink that simply logs all events it receives # and
>> connect it to the other end of the same channel.
>> agent1.sinks.log-sink1.channel = ch1
>> agent1.sinks.log-sink1.type = logger
>> agent1.sinks.sink.channel = ch1
>> agent1.sinks.sink.type = file_roll
>> agent1.sinks.sink.directory=/tmp/log
>>
>> # Finally, now that we've defined all of our components, tell # agent1
>> which ones we want to activate.
>> agent1.channels = ch1
>> agent1.sources = avro-source1
>> agent1.sinks = sink
>>
>> What is wrong ? I don't understand the problem.
>>
>> Guillaume
>
RE: RollingFile Sink problems
Posted by Guillaume Polaert <gp...@cyres.fr>.
Thank you for the link.
I've created a empty directory /tmp/log (mkdir+chmod).
Do I have to create other things ?
-----Message d'origine-----
De : alo alt [mailto:wget.null@googlemail.com]
Envoyé : lundi 21 mai 2012 15:31
À : flume-user@incubator.apache.org
Objet : Re: RollingFile Sink problems
Hi,
http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html#h.frlqql6vu68
Means, /tmp/log is empty or non existent, isn't it?
- Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
On May 21, 2012, at 3:03 PM, Guillaume Polaert wrote:
> Hi everybody,
>
> I've problems with RollingFile sink.
> First, I'm seeking about the sink's properties. I've found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
>
> Then, I've this error when I'm running the node :
>
> .
> SINKS: {sink=ComponentConfiguration[sink]
> CONFIG: {directory=/tmp/log, type=file_roll}
> RUNNER: ComponentConfiguration[runner]
> CONFIG: {}
>
>
> , log-sink1=ComponentConfiguration[log-sink1]
> CONFIG: {type=logger, channel=ch1}
> RUNNER: ComponentConfiguration[runner]
> CONFIG: {}
>
> }
> .
> 2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO -
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.jav
> a:69)] Creating instance of sink sink typefile_roll
> 2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
> java.lang.IllegalArgumentException: Directory may not be null
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
> at
> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> .
>
> Flume.conf
>
> # Define a memory channel called ch1 on agent1
> agent1.channels.ch1.type = memory
>
> # Define an Avro source called avro-source1 on agent1 and tell it # to
> bind to 0.0.0.0:41414. Connect it to channel ch1.
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
>
> # Define a logger sink that simply logs all events it receives # and
> connect it to the other end of the same channel.
> agent1.sinks.log-sink1.channel = ch1
> agent1.sinks.log-sink1.type = logger
> agent1.sinks.sink.channel = ch1
> agent1.sinks.sink.type = file_roll
> agent1.sinks.sink.directory=/tmp/log
>
> # Finally, now that we've defined all of our components, tell # agent1
> which ones we want to activate.
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = sink
>
> What is wrong ? I don't understand the problem.
>
> Guillaume
Re: RollingFile Sink problems
Posted by alo alt <wg...@googlemail.com>.
Hi,
http://archive.cloudera.com/cdh4/cdh/4/flume-ng-1.1.0-cdh4.0.0b2/FlumeUserGuide.html#h.frlqql6vu68
Means, /tmp/log is empty or non existent, isn't it?
- Alex
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
On May 21, 2012, at 3:03 PM, Guillaume Polaert wrote:
> Hi everybody,
>
> I’ve problems with RollingFile sink.
> First, I’m seeking about the sink’s properties. I’ve found two properties in RollingFileSink.java : rollInterval and directory. Are there anymore ?
>
> Then, I’ve this error when I’m running the node :
>
> …
> SINKS: {sink=ComponentConfiguration[sink]
> CONFIG: {directory=/tmp/log, type=file_roll}
> RUNNER: ComponentConfiguration[runner]
> CONFIG: {}
>
>
> , log-sink1=ComponentConfiguration[log-sink1]
> CONFIG: {type=logger, channel=ch1}
> RUNNER: ComponentConfiguration[runner]
> CONFIG: {}
>
> }
> …
> 2012-05-21 08:29:15,246 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:69)] Creating instance of sink sink typefile_roll
> 2012-05-21 08:29:15,253 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load configuration data. Exception follows.
> java.lang.IllegalArgumentException: Directory may not be null
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:74)
> at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> …
>
> Flume.conf
>
> # Define a memory channel called ch1 on agent1
> agent1.channels.ch1.type = memory
>
> # Define an Avro source called avro-source1 on agent1 and tell it
> # to bind to 0.0.0.0:41414. Connect it to channel ch1.
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
>
> # Define a logger sink that simply logs all events it receives
> # and connect it to the other end of the same channel.
> agent1.sinks.log-sink1.channel = ch1
> agent1.sinks.log-sink1.type = logger
> agent1.sinks.sink.channel = ch1
> agent1.sinks.sink.type = file_roll
> agent1.sinks.sink.directory=/tmp/log
>
> # Finally, now that we've defined all of our components, tell
> # agent1 which ones we want to activate.
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = sink
>
> What is wrong ? I don’t understand the problem.
>
> Guillaume