You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by vaibhav negi <ss...@gmail.com> on 2011/02/11 09:05:00 UTC

Hive Server - Transport error occurred during acceptance of message

Hi all,

I am loading data into hive tables by connecting to hiveserver through
thrift api using "load data local inpath ... " query .
Hive server is running as a background process for days . After some 4-5
days hiveserver throws this error :

11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Too many open files
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
        at
org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
        at
org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
        ... 9 more

I checked network connections using netstat and it show about 1000 active
network connections, most of them to hadoop datanode . Output is attached as
netstat.out

after i restarted hiveserver network connections falls to about 20 . output
is attached as netstat_after_HS_restart.out

Is this a problem with hive or hadoop configuration ?

Vaibhav Negi

Re: Hive Server - Transport error occurred during acceptance of message

Posted by Ajo Fod <aj...@gmail.com>.

You could try to save the information in a tab separated file and try to
load that into hive.

-Ajo

On Tue, Feb 15, 2011 at 1:24 AM, vaibhav negi <ss...@gmail.com>wrote:

> Hi ,
>
> I am using hive version 0.7 ( as suggested by lib/hive-hwi-0.7.0.war) . I
> checked out code from svn two weeks back . I use thrift api's to load data
> into hive programmatically .
>
> Vaibhav Negi
>
>
>
> On Sat, Feb 12, 2011 at 6:06 AM, Viral Bajaria <vi...@gmail.com>wrote:
>
>> We use hive through the thrift api too. I have seen the above error when
>> someone accidentally connects to the thrift server port through multiple
>> instances of our hive-wrapper code. We open a bunch of hive thrift ports to
>> achieve the automation of running multiple hive queries in parallel i.e.
>> continuous aggregations.
>>
>> you need to make sure you are not facing a similar issue because the hive
>> thrift server as far as i know is single threaded up until hive 0.5.0 (the
>> version that i use). Not too sure if that's been changed in 0.6.0 or higher.
>>
>> -Viral
>>
>> On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod <aj...@gmail.com> wrote:
>>
>>> Are you using hive 0.6? ... may be fixed in the latest version.
>>>
>>> Also I wonder why these thrift libraries are being used ... is this
>>> normal hive operation, or can you do something to avoid using thrift?
>>>
>>> -Ajo
>>>
>>>
>>> On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi <ss...@gmail.com>wrote:
>>>
>>>>
>>>> Hi all,
>>>>
>>>> I am loading data into hive tables by connecting to hiveserver through
>>>> thrift api using "load data local inpath ... " query .
>>>> Hive server is running as a background process for days . After some 4-5
>>>> days hiveserver throws this error :
>>>>
>>>> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error
>>>> occurred during acceptance of message.
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.SocketException: Too many open files
>>>>         at
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
>>>>         at
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>>>>         at
>>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>>>         at
>>>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
>>>>         at
>>>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>         at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>> Caused by: java.net.SocketException: Too many open files
>>>>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>         at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>         at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>         at
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
>>>>         ... 9 more
>>>>
>>>> I checked network connections using netstat and it show about 1000
>>>> active network connections, most of them to hadoop datanode . Output is
>>>> attached as netstat.out
>>>>
>>>> after i restarted hiveserver network connections falls to about 20 .
>>>> output is attached as netstat_after_HS_restart.out
>>>>
>>>> Is this a problem with hive or hadoop configuration ?
>>>>
>>>> Vaibhav Negi
>>>>
>>>
>>>
>>
>

Re: Hive Server - Transport error occurred during acceptance of message

Posted by vaibhav negi <ss...@gmail.com>.

Hi ,

I am using hive version 0.7 ( as suggested by lib/hive-hwi-0.7.0.war) . I
checked out code from svn two weeks back . I use thrift api's to load data
into hive programmatically .

Vaibhav Negi


On Sat, Feb 12, 2011 at 6:06 AM, Viral Bajaria <vi...@gmail.com>wrote:

> We use hive through the thrift api too. I have seen the above error when
> someone accidentally connects to the thrift server port through multiple
> instances of our hive-wrapper code. We open a bunch of hive thrift ports to
> achieve the automation of running multiple hive queries in parallel i.e.
> continuous aggregations.
>
> you need to make sure you are not facing a similar issue because the hive
> thrift server as far as i know is single threaded up until hive 0.5.0 (the
> version that i use). Not too sure if that's been changed in 0.6.0 or higher.
>
> -Viral
>
> On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod <aj...@gmail.com> wrote:
>
>> Are you using hive 0.6? ... may be fixed in the latest version.
>>
>> Also I wonder why these thrift libraries are being used ... is this normal
>> hive operation, or can you do something to avoid using thrift?
>>
>> -Ajo
>>
>>
>> On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi <ss...@gmail.com>wrote:
>>
>>>
>>> Hi all,
>>>
>>> I am loading data into hive tables by connecting to hiveserver through
>>> thrift api using "load data local inpath ... " query .
>>> Hive server is running as a background process for days . After some 4-5
>>> days hiveserver throws this error :
>>>
>>> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
>>> during acceptance of message.
>>> org.apache.thrift.transport.TTransportException:
>>> java.net.SocketException: Too many open files
>>>         at
>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
>>>         at
>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>>>         at
>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>>         at
>>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
>>>         at
>>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> Caused by: java.net.SocketException: Too many open files
>>>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>         at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>         at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>         at
>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
>>>         ... 9 more
>>>
>>> I checked network connections using netstat and it show about 1000 active
>>> network connections, most of them to hadoop datanode . Output is attached as
>>> netstat.out
>>>
>>> after i restarted hiveserver network connections falls to about 20 .
>>> output is attached as netstat_after_HS_restart.out
>>>
>>> Is this a problem with hive or hadoop configuration ?
>>>
>>> Vaibhav Negi
>>>
>>
>>
>

Re: Hive Server - Transport error occurred during acceptance of message

Posted by Viral Bajaria <vi...@gmail.com>.

We use hive through the thrift api too. I have seen the above error when
someone accidentally connects to the thrift server port through multiple
instances of our hive-wrapper code. We open a bunch of hive thrift ports to
achieve the automation of running multiple hive queries in parallel i.e.
continuous aggregations.

you need to make sure you are not facing a similar issue because the hive
thrift server as far as i know is single threaded up until hive 0.5.0 (the
version that i use). Not too sure if that's been changed in 0.6.0 or higher.

-Viral

On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod <aj...@gmail.com> wrote:

> Are you using hive 0.6? ... may be fixed in the latest version.
>
> Also I wonder why these thrift libraries are being used ... is this normal
> hive operation, or can you do something to avoid using thrift?
>
> -Ajo
>
>
> On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi <ss...@gmail.com>wrote:
>
>>
>> Hi all,
>>
>> I am loading data into hive tables by connecting to hiveserver through
>> thrift api using "load data local inpath ... " query .
>> Hive server is running as a background process for days . After some 4-5
>> days hiveserver throws this error :
>>
>> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
>> during acceptance of message.
>> org.apache.thrift.transport.TTransportException: java.net.SocketException:
>> Too many open files
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>>         at
>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>         at
>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
>>         at
>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.net.SocketException: Too many open files
>>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>         at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>         at java.net.ServerSocket.accept(ServerSocket.java:421)
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
>>         ... 9 more
>>
>> I checked network connections using netstat and it show about 1000 active
>> network connections, most of them to hadoop datanode . Output is attached as
>> netstat.out
>>
>> after i restarted hiveserver network connections falls to about 20 .
>> output is attached as netstat_after_HS_restart.out
>>
>> Is this a problem with hive or hadoop configuration ?
>>
>> Vaibhav Negi
>>
>
>

Re: Hive Server - Transport error occurred during acceptance of message

Posted by Ajo Fod <aj...@gmail.com>.

Are you using hive 0.6? ... may be fixed in the latest version.

Also I wonder why these thrift libraries are being used ... is this normal
hive operation, or can you do something to avoid using thrift?

-Ajo

On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi <ss...@gmail.com>wrote:

>
> Hi all,
>
> I am loading data into hive tables by connecting to hiveserver through
> thrift api using "load data local inpath ... " query .
> Hive server is running as a background process for days . After some 4-5
> days hiveserver throws this error :
>
> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
> during acceptance of message.
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Too many open files
>         at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
>         at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>         at
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>         at
> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
>         at
> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.net.SocketException: Too many open files
>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>         at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>         at java.net.ServerSocket.accept(ServerSocket.java:421)
>         at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
>         ... 9 more
>
> I checked network connections using netstat and it show about 1000 active
> network connections, most of them to hadoop datanode . Output is attached as
> netstat.out
>
> after i restarted hiveserver network connections falls to about 20 . output
> is attached as netstat_after_HS_restart.out
>
> Is this a problem with hive or hadoop configuration ?
>
> Vaibhav Negi
>