You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by mardan Khan <ma...@gmail.com> on 2012/07/25 16:33:27 UTC
Broken Pip error
Hi,
I am getting the following warring and error message
Warring: Unexpected error reading responses on connection
Thread[IPC Client (421539177) connection to hadoopmr/134.83.35.24:9000 from
root,5,main]
Error: Broken Pipe
12/07/25 15:26:26 ERROR hdfs.HDFSEventSink: close on hdfs://
hadoopmr.brunel.ac.uk:9000/flume/FlumeData; called
org.apache.flume.sink.hdfs.HDFSEventSink$3@1d162212
java.io.IOException: Failed on local exception: java.io.IOException: Broken
pipe; Host Details : local host is: "java.net.UnknownHostException: brunel:
brunel"; destination host is: "hadoopmr.brunel.ac.uk":9000;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
at org.apache.hadoop.ipc.Client.call(Client.java:1165)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
at $Proxy9.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
at $Proxy9.getFileInfo(Unknown Source)
What is the reason of the error.
I am using hadoop version 0.20.0 and flume1.x
Thanks
Re: Broken Pip error
Posted by Mohammad Tariq <do...@gmail.com>.
It would be better to uninstall cloudera's distribution first..It
seems flume is trying to contact "hadoopmr.brunel.ac.uk" instead of
"134.83.35.24"..Also add "hadoop.tmp.dir" property in your
"core-site.xml" file and "dfs.data.dir" and "dfs.name.dir" properties
in "hdfs-site.xml" file, else you'll loose your data and metadata at
each restart, as the values of these properties default to "/tmp" dir.
Regards,
Mohammad Tariq
On Thu, Jul 26, 2012 at 12:19 AM, mardan Khan <ma...@gmail.com> wrote:
> Tariq
>
> Let to telling you that i am using hadoop 0.20.0 which i have downloaded
> from apache website. I did not configure the hadoop which is automatically
> downloaded with CHD4
> I think my hadoop is accessible.
>
> The configuration file as:
>
>
> Core-site.xml
>
> <configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://134.83.35.24:9000</value>
> </property>
> </configuration>
>
>
> hdfs-site.xml
>
>
> <configuration>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> </configuration>
>
>
>
> mapred-site.xml
>
> <configuration>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>134.83.35.24:9001</value>
> </property>
> </configuration>
>
>
>
>
> Flume configuration
>
> agent.sources = avro-AppSrv-source
> agent.sinks = hdfs-Cluster1-sink
> agent.channels = mem-channel-1
> # set channel for sources, sinks
> # properties of avro-AppSrv-source
> agent.sources.avro-AppSrv-source.type = SEQ
>
> agent.sources.avro-AppSrv-source.bind = localhost
> agent.sources.avro-AppSrv-source.port = 10000
>
> agent.sources.avro-AppSrv-source.channels = mem-channel-1
>
> # properties of mem-channel-1
> agent.channels.mem-channel-1.type = memory
> agent.channels.mem-channel-1.capacity = 1000
> agent.channels.mem-channel-1.transactionCapacity = 100
> # properties of hdfs-Cluster1-sink
> agent.sinks.hdfs-Cluster1-sink.type = hdfs
>
> agent.sinks.hdfs-Cluster1-sink.channel = mem-channel-1
> agent.sinks.hdfs-Cluster1-sink.hdfs.path = hdfs://134.83.35.24:9000/flume
> agent.sinks.hdfs-Cluster-sin.hdfs.rollInterval = 30
> agent.sinks.hdfs-Cluster-sin.hdfs.rollsize= 1024
> agent.sinks.hdfs-Cluster-sin.hdfs.batchSize= 1
> agent.sinks.hdfs-Cluster-sin.hdfs.fileType = DataStream
> agent.sinks.hdfs-Cluster-sin.hdfs.writeFormat = writable
>
>
>
> Please any suggestion
>
>
> Thanks
>
>
> On Wed, Jul 25, 2012 at 3:43 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> Hello mardan,
>>
>> It seems the host where your NameNode is running is not
>> reachable or your are trying catch some other host. Could you please
>> show us your conf file??
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Wed, Jul 25, 2012 at 8:03 PM, mardan Khan <ma...@gmail.com> wrote:
>> > Hi,=
>>
>> >
>> > I am getting the following warring and error message
>> >
>> >
>> >
>> > Warring: Unexpected error reading responses on connection
>> > Thread[IPC Client (421539177) connection to hadoopmr/134.83.35.24:9000
>> > from
>> > root,5,main]
>> >
>> >
>> > Error: Broken Pipe
>> >
>> >
>> > 12/07/25 15:26:26 ERROR hdfs.HDFSEventSink: close on
>> > hdfs://hadoopmr.brunel.ac.uk:9000/flume/FlumeData; called
>> > org.apache.flume.sink.hdfs.HDFSEventSink$3@1d162212
>> > java.io.IOException: Failed on local exception: java.io.IOException:
>> > Broken
>> > pipe; Host Details : local host is: "java.net.UnknownHostException:
>> > brunel:
>> > brunel"; destination host is: "hadoopmr.brunel.ac.uk":9000;
>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1165)
>> > at
>> >
>> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
>> > at $Proxy9.getFileInfo(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
>> > at
>> >
>> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
>> > at $Proxy9.getFileInfo(Unknown Source)
>> >
>> >
>> > What is the reason of the error.
>> >
>> >
>> > I am using hadoop version 0.20.0 and flume1.x
>> >
>> >
>> > Thanks
>
>
Re: Broken Pip error
Posted by mardan Khan <ma...@gmail.com>.
Tariq
Let to telling you that i am using hadoop 0.20.0 which i have downloaded
from apache website. I did not configure the hadoop which is automatically
downloaded with CHD4
I think my hadoop is accessible.
The configuration file as:
*Core-site.xml*
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://134.83.35.24:9000</value>
</property>
</configuration>
*
hdfs-site.xml*
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
*
mapred-site.xml*
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>134.83.35.24:9001</value>
</property>
</configuration>
*Flume configuration*
agent.sources = avro-AppSrv-source
agent.sinks = hdfs-Cluster1-sink
agent.channels = mem-channel-1
# set channel for sources, sinks
# properties of avro-AppSrv-source
agent.sources.avro-AppSrv-source.type = SEQ
agent.sources.avro-AppSrv-source.bind = localhost
agent.sources.avro-AppSrv-source.port = 10000
agent.sources.avro-AppSrv-source.channels = mem-channel-1
# properties of mem-channel-1
agent.channels.mem-channel-1.type = memory
agent.channels.mem-channel-1.capacity = 1000
agent.channels.mem-channel-1.transactionCapacity = 100
# properties of hdfs-Cluster1-sink
agent.sinks.hdfs-Cluster1-sink.type = hdfs
agent.sinks.hdfs-Cluster1-sink.channel = mem-channel-1
agent.sinks.hdfs-Cluster1-sink.hdfs.path = hdfs://134.83.35.24:9000/flume
agent.sinks.hdfs-Cluster-sin.hdfs.rollInterval = 30
agent.sinks.hdfs-Cluster-sin.hdfs.rollsize= 1024
agent.sinks.hdfs-Cluster-sin.hdfs.batchSize= 1
agent.sinks.hdfs-Cluster-sin.hdfs.fileType = DataStream
agent.sinks.hdfs-Cluster-sin.hdfs.writeFormat = writable
Please any suggestion
Thanks
On Wed, Jul 25, 2012 at 3:43 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello mardan,
>
> It seems the host where your NameNode is running is not
> reachable or your are trying catch some other host. Could you please
> show us your conf file??
>
> Regards,
> Mohammad Tariq
>
>
> On Wed, Jul 25, 2012 at 8:03 PM, mardan Khan <ma...@gmail.com> wrote:
> > Hi,=
> >
> > I am getting the following warring and error message
> >
> >
> >
> > Warring: Unexpected error reading responses on connection
> > Thread[IPC Client (421539177) connection to hadoopmr/134.83.35.24:9000from
> > root,5,main]
> >
> >
> > Error: Broken Pipe
> >
> >
> > 12/07/25 15:26:26 ERROR hdfs.HDFSEventSink: close on
> > hdfs://hadoopmr.brunel.ac.uk:9000/flume/FlumeData; called
> > org.apache.flume.sink.hdfs.HDFSEventSink$3@1d162212
> > java.io.IOException: Failed on local exception: java.io.IOException:
> Broken
> > pipe; Host Details : local host is: "java.net.UnknownHostException:
> brunel:
> > brunel"; destination host is: "hadoopmr.brunel.ac.uk":9000;
> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1165)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
> > at $Proxy9.getFileInfo(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
> > at $Proxy9.getFileInfo(Unknown Source)
> >
> >
> > What is the reason of the error.
> >
> >
> > I am using hadoop version 0.20.0 and flume1.x
> >
> >
> > Thanks
>
Re: Broken Pip error
Posted by Mohammad Tariq <do...@gmail.com>.
Hello mardan,
It seems the host where your NameNode is running is not
reachable or your are trying catch some other host. Could you please
show us your conf file??
Regards,
Mohammad Tariq
On Wed, Jul 25, 2012 at 8:03 PM, mardan Khan <ma...@gmail.com> wrote:
> Hi,
>
> I am getting the following warring and error message
>
>
>
> Warring: Unexpected error reading responses on connection
> Thread[IPC Client (421539177) connection to hadoopmr/134.83.35.24:9000 from
> root,5,main]
>
>
> Error: Broken Pipe
>
>
> 12/07/25 15:26:26 ERROR hdfs.HDFSEventSink: close on
> hdfs://hadoopmr.brunel.ac.uk:9000/flume/FlumeData; called
> org.apache.flume.sink.hdfs.HDFSEventSink$3@1d162212
> java.io.IOException: Failed on local exception: java.io.IOException: Broken
> pipe; Host Details : local host is: "java.net.UnknownHostException: brunel:
> brunel"; destination host is: "hadoopmr.brunel.ac.uk":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
> at org.apache.hadoop.ipc.Client.call(Client.java:1165)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
> at $Proxy9.getFileInfo(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
> at $Proxy9.getFileInfo(Unknown Source)
>
>
> What is the reason of the error.
>
>
> I am using hadoop version 0.20.0 and flume1.x
>
>
> Thanks