You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by prabhu k <pr...@gmail.com> on 2012/08/22 11:43:35 UTC
Unable to sink HDFS using tail source.
Hi Users,
I have followed the below link *
http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html
* for sample text file to HDFS sink using tail source.
and i have executed flume-ng like using below command. it seems got stuck.
attached flume.conf file and running script log.
*bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf*
Please suggest and help me on this issue.
flume.conf
===========
agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS
agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
agent1.sources.tail.channels = MemoryChannel-2
agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type =
org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserveExisting = true
agent1.sources.tail.interceptors.hostint.useIP = false
agent1.sinks.HDFS.channel = MemoryChannel-2
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}
agent1.sinks.HDFS.hdfs.file.Type = dataStream
agent1.sinks.HDFS.hdfs.writeFormat = text
agent1.channels.MemoryChannel-2.type = memory
script running output:
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng
agent -n agent1 -c /conf -f conf/flume.conf
Info: Including Hadoop libraries found via
(/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
classpath
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
classpath
+ exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
'/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
-Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
org.apache.flume.node.Application -n agent1 -f conf/flume.conf
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1
12/08/22 15:01:08 INFO nodemanager.DefaultLogicalNodeManager: Node manager
starting
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:conf/flume.conf
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
agent1
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration for agents: [agent1]
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
created channel MemoryChannel-2
12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink
HDFS typehdfs
Thanks,
Prabhu.
Re: How to upload compressed event to avro-source
Posted by Raymond Ng <ra...@gmail.com>.
I'm also interested in this as I need to stream a AVRO_EVENT file back to a
AvroSource after its been written out by the File Sink in AVRO_EVENT format
thanks
Ray
On Wed, Aug 22, 2012 at 4:20 PM, yue.zhang <yu...@chinacache.com> wrote:
> Hi all****
>
> ** **
>
> How to upload compressed event (e.g. gzip) to avro-source ? ****
>
> ** **
>
> Because command “flume-ng avro-client -F xxxx” only supports text file as
> input file.****
>
> ** **
>
> -heipark****
>
--
Rgds
Ray
How to upload compressed event to avro-source
Posted by "yue.zhang" <yu...@chinacache.com>.
Hi all
How to upload compressed event (e.g. gzip) to avro-source ?
Because command "flume-ng avro-client -F xxxx" only supports text file as input file.
-heipark
Re: Unable to sink HDFS using tail source.
Posted by prabhu k <pr...@gmail.com>.
Hi Ashutosh,
Please find the flume.log file as below.
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1
12/08/22 15:01:08 INFO nodemanager.
DefaultLogicalNodeManager: Node manager starting
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:conf/flume.conf
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
agent1
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration for agents: [agent1]
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
created channel MemoryChannel-2
12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink
HDFS typehdfs
Thanks,
Prabhu
On Thu, Aug 23, 2012 at 1:57 PM, ashutosh(오픈플랫폼개발팀)
<sh...@kt.com>wrote:
> Can you please post your log(flume.log)?
>
>
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
>
>
> *From:* prabhu k [mailto:prabhu.flume@gmail.com]
> *Sent:* Thursday, August 23, 2012 4:58 PM
>
> *To:* user@flume.apache.org
> *Subject:* Re: Unable to sink HDFS using tail source.
>
>
>
> Hi Ashutosh,
>
> Thanks for the response.
>
> I have verified the specified HDFS location.but flume is not writing the
> data into the HDFS location. by using bin/hadoop fs -ls /user command.
>
> can you please clarify the below line in my flume.conf file
>
> *agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}<http://10.10.12.100:54310/user/%25%7Bhost%7D>
> *
>
> Please let me know i would have missed anything?
>
> Thanks,
> Prabhu.
>
> On Thu, Aug 23, 2012 at 1:17 PM, ashutosh(오픈플랫폼개발팀) <
> sharma.ashutosh@kt.com> wrote:
>
> Hi Prabhu,
>
>
>
> Please check, is flume storing data in specified HDFS location? If flume
> is storing the data then, it’s doing its job. You need not to worry about
> the flume behavior.
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
>
>
> *From:* prabhu k [mailto:prabhu.flume@gmail.com]
> *Sent:* Thursday, August 23, 2012 4:12 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Unable to sink HDFS using tail source.
>
>
>
> can anyone help me on the below issue.early response i would appreciate.
>
> On Wed, Aug 22, 2012 at 3:13 PM, prabhu k <pr...@gmail.com> wrote:
>
> Hi Users,
>
> I have followed the below link *
> http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html
> * for sample text file to HDFS sink using tail source.
>
> and i have executed flume-ng like using below command. it seems got stuck.
> attached flume.conf file and running script log.
>
> *bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf*
>
> Please suggest and help me on this issue.
>
> flume.conf
> ===========
>
> agent1.sources = tail
> agent1.channels = MemoryChannel-2
> agent1.sinks = HDFS
> agent1.sources.tail.type = exec
> agent1.sources.tail.command = tail -F
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
> agent1.sources.tail.channels = MemoryChannel-2
> agent1.sources.tail.interceptors = hostint
> agent1.sources.tail.interceptors.hostint.type =
> org.apache.flume.interceptor.HostInterceptor$Builder
> agent1.sources.tail.interceptors.hostint.preserveExisting = true
> agent1.sources.tail.interceptors.hostint.useIP = false
> agent1.sinks.HDFS.channel = MemoryChannel-2
> agent1.sinks.HDFS.type = hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}<http://10.10.12.100:54310/user/%25%7Bhost%7D>
> agent1.sinks.HDFS.hdfs.file.Type = dataStream
> agent1.sinks.HDFS.hdfs.writeFormat = text
> agent1.channels.MemoryChannel-2.type = memory
>
>
>
> script running output:
>
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng
> agent -n agent1 -c /conf -f conf/flume.conf
> Info: Including Hadoop libraries found via
> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
>
> Info: Excluding
> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3..jar from
> classpath
> Info: Excluding
> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1...4.3.jar from
> classpath
> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
> '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/..../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8..jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1..jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0..3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0..3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0..3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1..8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
> org.apache.flume.node.Application -n agent1 -f conf/flume.conf
>
>
> 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 1
> 12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1
> 12/08/22 15:01:08 INFO nodemanager.DefaultLogicalNodeManager: Node manager
> starting
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider starting
> 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 10
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Reloading configuration file:conf/flume.conf
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
> agent1
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration for agents: [agent1]
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> created channel MemoryChannel-2
> 12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink
> HDFS typehdfs
>
> Thanks,
> Prabhu.
>
>
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에
> 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못
> 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright
> material. This email is intended for the use of the addressee only. If you
> receive this email by mistake, please either delete it without reproducing,
> distributing or retaining copies thereof or notify the sender immediately.
>
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에
> 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못
> 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright
> material. This email is intended for the use of the addressee only. If you
> receive this email by mistake, please either delete it without reproducing,
> distributing or retaining copies thereof or notify the sender immediately.
>
RE: Unable to sink HDFS using tail source.
Posted by "ashutosh (오픈플랫폼개발팀)" <sh...@kt.com>.
Can you please post your log(flume.log)?
Thanks & Regards,
Ashutosh Sharma
From: prabhu k [mailto:prabhu.flume@gmail.com]
Sent: Thursday, August 23, 2012 4:58 PM
To: user@flume.apache.org
Subject: Re: Unable to sink HDFS using tail source.
Hi Ashutosh,
Thanks for the response.
I have verified the specified HDFS location.but flume is not writing the data into the HDFS location. by using bin/hadoop fs -ls /user command.
can you please clarify the below line in my flume.conf file
agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}<http://10.10.12.100:54310/user/%25%7Bhost%7D>
Please let me know i would have missed anything?
Thanks,
Prabhu.
On Thu, Aug 23, 2012 at 1:17 PM, ashutosh(오픈플랫폼개발팀) <sh...@kt.com>> wrote:
Hi Prabhu,
Please check, is flume storing data in specified HDFS location? If flume is storing the data then, it’s doing its job. You need not to worry about the flume behavior.
Thanks & Regards,
Ashutosh Sharma
From: prabhu k [mailto:prabhu.flume@gmail.com<ma...@gmail.com>]
Sent: Thursday, August 23, 2012 4:12 PM
To: user@flume.apache.org<ma...@flume.apache.org>
Subject: Re: Unable to sink HDFS using tail source.
can anyone help me on the below issue.early response i would appreciate.
On Wed, Aug 22, 2012 at 3:13 PM, prabhu k <pr...@gmail.com>> wrote:
Hi Users,
I have followed the below link http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html for sample text file to HDFS sink using tail source.
and i have executed flume-ng like using below command. it seems got stuck. attached flume.conf file and running script log.
bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf
Please suggest and help me on this issue.
flume.conf
===========
agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS
agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
agent1.sources.tail.channels = MemoryChannel-2
agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type = org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserveExisting = true
agent1.sources.tail.interceptors.hostint.useIP = false
agent1.sinks.HDFS.channel = MemoryChannel-2
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}<http://10.10.12.100:54310/user/%25%7Bhost%7D>
agent1.sinks.HDFS.hdfs.file.Type = dataStream
agent1.sinks.HDFS.hdfs.writeFormat = text
agent1.channels.MemoryChannel-2.type = memory
script running output:
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf
Info: Including Hadoop libraries found via (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3..jar from classpath
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1..4.3.jar from classpath
+ exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0..3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0..3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 org.apache.flume.node.Application -n agent1 -f conf/flume.conf
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1
12/08/22 15:01:08 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:conf/flume.conf
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: agent1
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1]
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: Creating channels
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: created channel MemoryChannel-2
12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink HDFS typehdfs
Thanks,
Prabhu.
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
Re: Unable to sink HDFS using tail source.
Posted by prabhu k <pr...@gmail.com>.
Hi Ashutosh,
Thanks for the response.
I have verified the specified HDFS location.but flume is not writing the
data into the HDFS location. by using bin/hadoop fs -ls /user command.
can you please clarify the below line in my flume.conf file
*agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}*
Please let me know i would have missed anything?
Thanks,
Prabhu.
On Thu, Aug 23, 2012 at 1:17 PM, ashutosh(오픈플랫폼개발팀)
<sh...@kt.com>wrote:
> Hi Prabhu,
>
>
>
> Please check, is flume storing data in specified HDFS location? If flume
> is storing the data then, it’s doing its job. You need not to worry about
> the flume behavior.
>
> Thanks & Regards,
>
> Ashutosh Sharma
>
>
>
> *From:* prabhu k [mailto:prabhu.flume@gmail.com]
> *Sent:* Thursday, August 23, 2012 4:12 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Unable to sink HDFS using tail source.
>
>
>
> can anyone help me on the below issue.early response i would appreciate.
>
> On Wed, Aug 22, 2012 at 3:13 PM, prabhu k <pr...@gmail.com> wrote:
>
> Hi Users,
>
> I have followed the below link *
> http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html
> * for sample text file to HDFS sink using tail source.
>
> and i have executed flume-ng like using below command. it seems got stuck.
> attached flume.conf file and running script log.
>
> *bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf*
>
> Please suggest and help me on this issue.
>
> flume.conf
> ===========
>
> agent1.sources = tail
> agent1.channels = MemoryChannel-2
> agent1.sinks = HDFS
> agent1.sources.tail.type = exec
> agent1.sources.tail.command = tail -F
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
> agent1.sources.tail.channels = MemoryChannel-2
> agent1.sources.tail.interceptors = hostint
> agent1.sources.tail.interceptors.hostint.type =
> org.apache.flume.interceptor.HostInterceptor$Builder
> agent1.sources.tail.interceptors.hostint.preserveExisting = true
> agent1.sources.tail.interceptors.hostint.useIP = false
> agent1.sinks.HDFS.channel = MemoryChannel-2
> agent1.sinks.HDFS.type = hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}<http://10.10.12.100:54310/user/%25%7Bhost%7D>
> agent1.sinks.HDFS.hdfs.file.Type = dataStream
> agent1.sinks.HDFS.hdfs.writeFormat = text
> agent1.channels.MemoryChannel-2.type = memory
>
>
>
> script running output:
>
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng
> agent -n agent1 -c /conf -f conf/flume.conf
> Info: Including Hadoop libraries found via
> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
> Info: Excluding
> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3..jar from
> classpath
> Info: Excluding
> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1..4.3.jar from
> classpath
> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
> '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/.../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0..3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0..3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
> org.apache.flume.node.Application -n agent1 -f conf/flume.conf
>
> 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 1
> 12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1
> 12/08/22 15:01:08 INFO nodemanager.DefaultLogicalNodeManager: Node manager
> starting
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider starting
> 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 10
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Reloading configuration file:conf/flume.conf
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
> agent1
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration for agents: [agent1]
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> created channel MemoryChannel-2
> 12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink
> HDFS typehdfs
>
> Thanks,
> Prabhu.
>
>
>
>
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에
> 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못
> 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
> This E-mail may contain confidential information and/or copyright
> material. This email is intended for the use of the addressee only. If you
> receive this email by mistake, please either delete it without reproducing,
> distributing or retaining copies thereof or notify the sender immediately.
>
RE: Unable to sink HDFS using tail source.
Posted by "ashutosh (오픈플랫폼개발팀)" <sh...@kt.com>.
Hi Prabhu,
Please check, is flume storing data in specified HDFS location? If flume is storing the data then, it’s doing its job. You need not to worry about the flume behavior.
Thanks & Regards,
Ashutosh Sharma
From: prabhu k [mailto:prabhu.flume@gmail.com]
Sent: Thursday, August 23, 2012 4:12 PM
To: user@flume.apache.org
Subject: Re: Unable to sink HDFS using tail source.
can anyone help me on the below issue.early response i would appreciate.
On Wed, Aug 22, 2012 at 3:13 PM, prabhu k <pr...@gmail.com>> wrote:
Hi Users,
I have followed the below link http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html for sample text file to HDFS sink using tail source.
and i have executed flume-ng like using below command. it seems got stuck. attached flume.conf file and running script log.
bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf
Please suggest and help me on this issue.
flume.conf
===========
agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS
agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
agent1.sources.tail.channels = MemoryChannel-2
agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type = org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserveExisting = true
agent1.sources.tail.interceptors.hostint.useIP = false
agent1.sinks.HDFS.channel = MemoryChannel-2
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}<http://10.10.12.100:54310/user/%25%7Bhost%7D>
agent1.sinks.HDFS.hdfs.file.Type = dataStream
agent1.sinks.HDFS.hdfs.writeFormat = text
agent1.channels.MemoryChannel-2.type = memory
script running output:
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf
Info: Including Hadoop libraries found via (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath
+ exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32 org.apache.flume.node.Application -n agent1 -f conf/flume.conf
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1
12/08/22 15:01:08 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:conf/flume.conf
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: agent1
12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1]
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: Creating channels
12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider: created channel MemoryChannel-2
12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink HDFS typehdfs
Thanks,
Prabhu.
이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
Re: Unable to sink HDFS using tail source.
Posted by prabhu k <pr...@gmail.com>.
can anyone help me on the below issue.early response i would appreciate.
On Wed, Aug 22, 2012 at 3:13 PM, prabhu k <pr...@gmail.com> wrote:
> Hi Users,
>
> I have followed the below link *
> http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html
> * for sample text file to HDFS sink using tail source.
>
> and i have executed flume-ng like using below command. it seems got stuck.
> attached flume.conf file and running script log.
>
> *bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf*
>
> Please suggest and help me on this issue.
>
> flume.conf
> ===========
>
> agent1.sources = tail
> agent1.channels = MemoryChannel-2
> agent1.sinks = HDFS
> agent1.sources.tail.type = exec
> agent1.sources.tail.command = tail -F
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
> agent1.sources.tail.channels = MemoryChannel-2
> agent1.sources.tail.interceptors = hostint
> agent1.sources.tail.interceptors.hostint.type =
> org.apache.flume.interceptor.HostInterceptor$Builder
> agent1.sources.tail.interceptors.hostint.preserveExisting = true
> agent1.sources.tail.interceptors.hostint.useIP = false
> agent1.sinks.HDFS.channel = MemoryChannel-2
> agent1.sinks.HDFS.type = hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://10.10.12.100:54310/user/%{host}
> agent1.sinks.HDFS.hdfs.file.Type = dataStream
> agent1.sinks.HDFS.hdfs.writeFormat = text
> agent1.channels.MemoryChannel-2.type = memory
>
>
>
> script running output:
>
> /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT# bin/flume-ng
> agent -n agent1 -c /conf -f conf/flume.conf
> Info: Including Hadoop libraries found via
> (/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
> Info: Excluding
> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
> classpath
> Info: Excluding
> /usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
> classpath
> + exec /usr/lib/jvm/java-6-sun/bin/java -Xmx20m -cp
> '/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
> -Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
> org.apache.flume.node.Application -n agent1 -f conf/flume.conf
> 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 1
> 12/08/22 15:01:08 INFO node.FlumeNode: Flume node starting - agent1
> 12/08/22 15:01:08 INFO nodemanager.DefaultLogicalNodeManager: Node manager
> starting
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider starting
> 12/08/22 15:01:08 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 10
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Reloading configuration file:conf/flume.conf
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
> agent1
> 12/08/22 15:01:08 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration for agents: [agent1]
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 12/08/22 15:01:08 INFO properties.PropertiesFileConfigurationProvider:
> created channel MemoryChannel-2
> 12/08/22 15:01:08 INFO sink.DefaultSinkFactory: Creating instance of sink
> HDFS typehdfs
>
> Thanks,
> Prabhu.
>