You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "ashutosh (오픈플랫폼개발팀)" <sh...@kt.com> on 2012/06/29 09:01:36 UTC

HDFS event Sink

Hi All,

I am trying to push the data into hdfs through hdfs sink. I am facing following issue while running the agent:

2012-06-29 15:50:19,415 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
        at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
        at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
        at java.lang.Thread.run(Thread.java:662)
2012-06-29 15:50:24,003 (node-shutdownHook) [INFO - org.apache.flume.node.FlumeNode.stop(FlumeNode.java:67)] Flume node stopping - thdfs-agent
2012-06-29 15:50:24,003 (node-shutdownHook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:81)] Stopping lifecycle supervisor 9
2012-06-29 15:50:24,419 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
java.lang.IllegalStateException: begin() called when transaction is OPEN!
        at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
        at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
        at java.lang.Thread.run(Thread.java:662)
2012-06-29 15:50:24,420 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:161)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: begin() called when transaction is OPEN!
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:375)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
        at java.lang.Thread.run(Thread.java:662)

I found that this has been reported as part of https://issues.apache.org/jira/browse/FLUME-1075
and resolved as part of the flume 1.2.

Could you please help me to avoid this error as I am using the Flume 1.1 binary snapshot. My configuration file as below:
hdfs-agent.sources = tail-source
hdfs-agent.sinks = hdfs-cluster-sink
hdfs-agent.channels = mem-channel

hdfs-agent.sources.tail-source.type = exec
hdfs-agent.sources.tail-source.command = tail -f /home/hadoop/demo.txt
hdfs-agent.sources.tail-source.channels = mem-channel
hdfs-agent.sinks.hdfs-cluster-sink.type = hdfs

hdfs-agent.sinks.hdfs-cluster-sink.channel = mem-channel
hdfs-agent.sinks.hdfs-cluster-sink.hdfs.path = hdfs://hostname/logs/apache/rawng/
hdfs-agent.sinks.hdfs-cluster-sink.hdfs.filePrefix = demo
#hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollInterval = 60
hdfs-agent.sinks.hdfs-cluster-sink.hdfs.writeFormat = Text
#hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollSize = 1073741824
#hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollcount = 1000000

thdfs-agent.channels.mem-channel.type=memory

----------------------------------------
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------



이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.

RE: HDFS event Sink

Posted by "ashutosh (오픈플랫폼개발팀)" <sh...@kt.com>.
Hi Hari,

Thanks!! Yes, it works with 1.2.0. I hope to see the next release soon.

----------------------------------------
----------------------------------------
Thanks & Regards,
Ashutosh Sharma
----------------------------------------

From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Friday, June 29, 2012 4:34 PM
To: flume-user@incubator.apache.org
Subject: Re: HDFS event Sink

Hi Ashutosh,

Would you be able to use a binary of Flume 1.2.0? It is as yet unreleased, but you can use the latest snapshots from the Jenkins build, or you could checkout the source from svn/git and build it. Flume 1.2.0 will be released soon, though I cannot give you an accurate date. The release discussions have just begun on flume-dev.


Thanks
Hari

--
Hari Shreedharan


On Friday, June 29, 2012 at 12:01 AM, ashutosh(오픈플랫폼개발팀) wrote:

Hi All,



I am trying to push the data into hdfs through hdfs sink. I am facing following issue while running the agent:



2012-06-29 15:50:19,415 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms

java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first

        at com.google.common.base.Preconditions.checkState(Preconditions.java:172)

        at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)

        at java.lang.Thread.run(Thread.java:662)

2012-06-29 15:50:24,003 (node-shutdownHook) [INFO - org.apache.flume.node.FlumeNode.stop(FlumeNode.java:67)] Flume node stopping - thdfs-agent

2012-06-29 15:50:24,003 (node-shutdownHook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:81)] Stopping lifecycle supervisor 9

2012-06-29 15:50:24,419 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed

java.lang.IllegalStateException: begin() called when transaction is OPEN!

        at com.google.common.base.Preconditions.checkState(Preconditions.java:145)

        at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)

        at java.lang.Thread.run(Thread.java:662)

2012-06-29 15:50:24,420 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:161)] Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: begin() called when transaction is OPEN!

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:375)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)

        at java.lang.Thread.run(Thread.java:662)



I found that this has been reported as part of https://issues.apache.org/jira/browse/FLUME-1075

and resolved as part of the flume 1.2.



Could you please help me to avoid this error as I am using the Flume 1.1 binary snapshot. My configuration file as below:

hdfs-agent.sources = tail-source

hdfs-agent.sinks = hdfs-cluster-sink

hdfs-agent.channels = mem-channel



hdfs-agent.sources.tail-source.type = exec

hdfs-agent.sources.tail-source.command = tail -f /home/hadoop/demo.txt

hdfs-agent.sources.tail-source.channels = mem-channel

hdfs-agent.sinks.hdfs-cluster-sink.type = hdfs



hdfs-agent.sinks.hdfs-cluster-sink.channel = mem-channel

hdfs-agent.sinks.hdfs-cluster-sink.hdfs.path = hdfs://hostname/logs/apache/rawng/

hdfs-agent.sinks.hdfs-cluster-sink.hdfs.filePrefix = demo

#hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollInterval = 60

hdfs-agent.sinks.hdfs-cluster-sink.hdfs.writeFormat = Text

#hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollSize = 1073741824

#hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollcount = 1000000



thdfs-agent.channels.mem-channel.type=memory



----------------------------------------

----------------------------------------

Thanks & Regards,

Ashutosh Sharma

----------------------------------------




이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.



이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.
This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.

Re: HDFS event Sink

Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Ashutosh,  

Would you be able to use a binary of Flume 1.2.0? It is as yet unreleased, but you can use the latest snapshots from the Jenkins build, or you could checkout the source from svn/git and build it. Flume 1.2.0 will be released soon, though I cannot give you an accurate date. The release discussions have just begun on flume-dev.  


Thanks
Hari

--  
Hari Shreedharan


On Friday, June 29, 2012 at 12:01 AM, ashutosh(오픈플랫폼개발팀) wrote:

>  
> Hi All,
>  
>  
>   
>  
>  
> I am trying to push the data into hdfs through hdfs sink. I am facing following issue while running the agent:
>  
>  
>   
>  
>  
> 2012-06-29 15:50:19,415 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
>  
>  
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
>  
>  
>         at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
>  
>  
>         at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
>  
>  
>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
>  
>  
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
>  
>  
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
>  
>  
>         at java.lang.Thread.run(Thread.java:662)
>  
>  
> 2012-06-29 15:50:24,003 (node-shutdownHook) [INFO - org.apache.flume.node.FlumeNode.stop(FlumeNode.java:67)] Flume node stopping - thdfs-agent
>  
>  
> 2012-06-29 15:50:24,003 (node-shutdownHook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:81)] Stopping lifecycle supervisor 9
>  
>  
> 2012-06-29 15:50:24,419 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
>  
>  
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
>  
>  
>         at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
>  
>  
>         at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
>  
>  
>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
>  
>  
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
>  
>  
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
>  
>  
>         at java.lang.Thread.run(Thread.java:662)
>  
>  
> 2012-06-29 15:50:24,420 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:161)] Unable to deliver event. Exception follows.
>  
>  
> org.apache.flume.EventDeliveryException: begin() called when transaction is OPEN!
>  
>  
>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:375)
>  
>  
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
>  
>  
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
>  
>  
>         at java.lang.Thread.run(Thread.java:662)
>  
>  
>   
>  
>  
> I found that this has been reported as part of https://issues.apache.org/jira/browse/FLUME-1075
>  
>  
> and resolved as part of the flume 1.2.  
>  
>  
>   
>  
>  
> Could you please help me to avoid this error as I am using the Flume 1.1 binary snapshot. My configuration file as below:
>  
>  
> hdfs-agent.sources = tail-source
>  
>  
> hdfs-agent.sinks = hdfs-cluster-sink
>  
>  
> hdfs-agent.channels = mem-channel
>  
>  
>   
>  
>  
> hdfs-agent.sources.tail-source.type = exec
>  
>  
> hdfs-agent.sources.tail-source.command = tail -f /home/hadoop/demo.txt
>  
>  
> hdfs-agent.sources.tail-source.channels = mem-channel
>  
>  
> hdfs-agent.sinks.hdfs-cluster-sink.type = hdfs
>  
>  
>   
>  
>  
> hdfs-agent.sinks.hdfs-cluster-sink.channel = mem-channel
>  
>  
> hdfs-agent.sinks.hdfs-cluster-sink.hdfs.path = hdfs://hostname/logs/apache/rawng/
>  
>  
> hdfs-agent.sinks.hdfs-cluster-sink.hdfs.filePrefix = demo
>  
>  
> #hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollInterval = 60
>  
>  
> hdfs-agent.sinks.hdfs-cluster-sink.hdfs.writeFormat = Text
>  
>  
> #hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollSize = 1073741824
>  
>  
> #hdfs-agent.sinks.hdfs-cluster-sink.hdfs.rollcount = 1000000
>  
>  
>   
>  
>  
> thdfs-agent.channels.mem-channel.type=memory
>  
>  
>   
>  
>  
> ----------------------------------------
>  
>  
> ----------------------------------------
>  
>  
> Thanks & Regards,
>  
>  
> Ashutosh Sharma
>  
>  
> ----------------------------------------
>  
>  
>   
>  
>  
>   
>  
> 이 메일은 지정된 수취인만을 위해 작성되었으며, 중요한 정보나 저작권을 포함하고 있을 수 있습니다. 어떠한 권한 없이, 본 문서에 포함된 정보의 전부 또는 일부를 무단으로 제3자에게 공개, 배포, 복사 또는 사용하는 것을 엄격히 금지합니다. 만약, 본 메일이 잘못 전송된 경우, 발신인 또는 당사에 알려주시고, 본 메일을 즉시 삭제하여 주시기 바랍니다.  
> This E-mail may contain confidential information and/or copyright material. This email is intended for the use of the addressee only. If you receive this email by mistake, please either delete it without reproducing, distributing or retaining copies thereof or notify the sender immediately.
>  
>  
>  
>