You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by John Omernik <jo...@omernik.com> on 2015/03/21 21:50:55 UTC
HAProxy for Hive Thrift Server on Mesos
I have a nice setup with a Hive thrift server running in a docker
container on Mesos. It works pretty well, but something, I believe in
how HAProxy works with the connection, is causing the thrift server
connection to die after a time. Basically, I can run a few queries,
but after 2 or 3, or specifically after a longer query, I get the
error below indicating End of File on the connection. Then no more
connections work until I reestablish the connection to the thrift
server. I've tried looking in logs. the thrift server std err logs
show no issues. I am guess I need to dig into haproxy logs, but I am
not seeing any issues in syslog so far. I'd love any pointers on how
to trouble shoot this. By the way, I have MySQl, hive metastore, and
a minecraft server all running on Mesos/Docker with no issues, not
sure why the thrift server is so sensitive.
:)
pyhs2 connection string:
hs2 = pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
Error
/usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in
read(self, sz)
118 if len(buff) == 0:
119 raise TTransportException(type=TTransportException.END_OF_FILE,
--> 120 message='TSocket read 0 bytes')
121 return buff
122
TTransportException: TSocket read 0 bytes
Re: HAProxy for Hive Thrift Server on Mesos
Posted by John Omernik <jo...@omernik.com>.
So after more testing, it does a appear to be the keep-alive with the
thrift server. I am currently running Hive 0.13.0 which doesn't
support a keep-alive, I am hoping when I am able to upgrade, that the
keep alive helps me through this issue. Any HAProxy experts out there
have ideas for dealing with an app that doesn't have a TCP keep alive?
On Wed, Mar 25, 2015 at 12:17 PM, Ben Whitehead
<be...@mesosphere.io> wrote:
> Awesome glad to hear it.
>
> On Tue, Mar 24, 2015 at 10:00 PM, xiaokun <xi...@gmail.com> wrote:
>>
>> After set the timeout value, the tcp request become stable.
>>
>> 2015-03-23 23:26 GMT+08:00 John Omernik <jo...@omernik.com>:
>>>
>>> So per my HA-Proxy logs, I am getting a condition where the client
>>> side time expired while waiting for the client to send or receive
>>> data, and this is occurring during the DATA phase. (the two letter
>>> HAProxy code is cD . To test, I expanded the timeout (on all things)
>>> to 100000.
>>>
>>> This was interesting in that it showed me the problem a bit clearer.
>>> Now, as long as I submit and results for a query in the 100 second
>>> range, my connection stays open and happy. If I let it sit for 100
>>> seconds, and then try a query, then it breaks again.
>>>
>>> It appears HA proxy gets irritable when things aren't being sent over
>>> a connection for a long period of time. For services like Minecraft,
>>> this is not an issue, when a client is connected, there is always data
>>> there, and no issues occur. For Thrift, there may be multiple long
>>> pauses as an analyst thinks through the next query etc. I wonder what
>>> would be the best way to handle that on a per container level. (not
>>> all containers need this sort of customization...)
>>>
>>>
>>>
>>> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xi...@gmail.com> wrote:
>>> > I have a similar error when launch jenkins in mesos cluster. And I use
>>> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from
>>> > time
>>> > to time, connection between jenkins master and jenkins slave will lost,
>>> > while will cause jobs fail.
>>> >
>>> > I will try to set timeout value for HAProxy
>>> >
>>> > 2015-03-23 4:18 GMT+08:00 John Omernik <jo...@omernik.com>:
>>> >>
>>> >> Yep all three are going through HAProxy and seem to be working well.
>>> >> I should do some testing to determine if I run a thrift server that
>>> >> connects to metastore in HAProxy to determine if that's the broken
>>> >> link, but I believe it's between the thrift server. I will check out
>>> >> the link you posted to get a better idea out of how it works. I am
>>> >> running this all on a MapR cluster which is Great for Docker/Mesos as
>>> >> I used the MapR FS, mounted locally on each node's NFS to put the data
>>> >> for MySQL, the logs for the metastore/thift server, and my world files
>>> >> for Minecraft. So far it's all really slick.
>>> >>
>>> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
>>> >> <be...@mesosphere.io> wrote:
>>> >> > It sounds like you may be running into the connection timeout that
>>> >> > is
>>> >> > configured for the HAProxy instance.
>>> >> >
>>> >> > You can see an instance of this in Marathon's
>>> >> > HAProxy-marathon-bridge[1].
>>> >> > You may need to raise the timeout values.
>>> >> >
>>> >> > Are you accessing MySql, Hive metastore and minecraft through
>>> >> > HAProxy as
>>> >> > well? I know that Minecraft running in docker on mesos works well,
>>> >> > though
>>> >> > I've always used a direct connection not a proxied one.
>>> >> >
>>> >> >
>>> >> > [1]
>>> >> >
>>> >> >
>>> >> > https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
>>> >> >
>>> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <jo...@omernik.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> I have a nice setup with a Hive thrift server running in a docker
>>> >> >> container on Mesos. It works pretty well, but something, I believe
>>> >> >> in
>>> >> >> how HAProxy works with the connection, is causing the thrift server
>>> >> >> connection to die after a time. Basically, I can run a few
>>> >> >> queries,
>>> >> >> but after 2 or 3, or specifically after a longer query, I get the
>>> >> >> error below indicating End of File on the connection. Then no more
>>> >> >> connections work until I reestablish the connection to the thrift
>>> >> >> server. I've tried looking in logs. the thrift server std err logs
>>> >> >> show no issues. I am guess I need to dig into haproxy logs, but I
>>> >> >> am
>>> >> >> not seeing any issues in syslog so far. I'd love any pointers on
>>> >> >> how
>>> >> >> to trouble shoot this. By the way, I have MySQl, hive metastore,
>>> >> >> and
>>> >> >> a minecraft server all running on Mesos/Docker with no issues, not
>>> >> >> sure why the thrift server is so sensitive.
>>> >> >>
>>> >> >> :)
>>> >> >>
>>> >> >>
>>> >> >> pyhs2 connection string:
>>> >> >>
>>> >> >> hs2 =
>>> >> >>
>>> >> >>
>>> >> >> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
>>> >> >>
>>> >> >>
>>> >> >> Error
>>> >> >>
>>> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc
>>> >> >> in
>>> >> >> read(self, sz)
>>> >> >> 118 if len(buff) == 0:
>>> >> >> 119 raise
>>> >> >> TTransportException(type=TTransportException.END_OF_FILE,
>>> >> >> --> 120 message='TSocket read 0
>>> >> >> bytes')
>>> >> >> 121 return buff
>>> >> >> 122
>>> >> >>
>>> >> >> TTransportException: TSocket read 0 bytes
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
>
Re: HAProxy for Hive Thrift Server on Mesos
Posted by Ben Whitehead <be...@mesosphere.io>.
Awesome glad to hear it.
On Tue, Mar 24, 2015 at 10:00 PM, xiaokun <xi...@gmail.com> wrote:
> After set the timeout value, the tcp request become stable.
>
> 2015-03-23 23:26 GMT+08:00 John Omernik <jo...@omernik.com>:
>
>> So per my HA-Proxy logs, I am getting a condition where the client
>> side time expired while waiting for the client to send or receive
>> data, and this is occurring during the DATA phase. (the two letter
>> HAProxy code is cD . To test, I expanded the timeout (on all things)
>> to 100000.
>>
>> This was interesting in that it showed me the problem a bit clearer.
>> Now, as long as I submit and results for a query in the 100 second
>> range, my connection stays open and happy. If I let it sit for 100
>> seconds, and then try a query, then it breaks again.
>>
>> It appears HA proxy gets irritable when things aren't being sent over
>> a connection for a long period of time. For services like Minecraft,
>> this is not an issue, when a client is connected, there is always data
>> there, and no issues occur. For Thrift, there may be multiple long
>> pauses as an analyst thinks through the next query etc. I wonder what
>> would be the best way to handle that on a per container level. (not
>> all containers need this sort of customization...)
>>
>>
>>
>> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xi...@gmail.com> wrote:
>> > I have a similar error when launch jenkins in mesos cluster. And I use
>> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from
>> time
>> > to time, connection between jenkins master and jenkins slave will lost,
>> > while will cause jobs fail.
>> >
>> > I will try to set timeout value for HAProxy
>> >
>> > 2015-03-23 4:18 GMT+08:00 John Omernik <jo...@omernik.com>:
>> >>
>> >> Yep all three are going through HAProxy and seem to be working well.
>> >> I should do some testing to determine if I run a thrift server that
>> >> connects to metastore in HAProxy to determine if that's the broken
>> >> link, but I believe it's between the thrift server. I will check out
>> >> the link you posted to get a better idea out of how it works. I am
>> >> running this all on a MapR cluster which is Great for Docker/Mesos as
>> >> I used the MapR FS, mounted locally on each node's NFS to put the data
>> >> for MySQL, the logs for the metastore/thift server, and my world files
>> >> for Minecraft. So far it's all really slick.
>> >>
>> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
>> >> <be...@mesosphere.io> wrote:
>> >> > It sounds like you may be running into the connection timeout that is
>> >> > configured for the HAProxy instance.
>> >> >
>> >> > You can see an instance of this in Marathon's
>> >> > HAProxy-marathon-bridge[1].
>> >> > You may need to raise the timeout values.
>> >> >
>> >> > Are you accessing MySql, Hive metastore and minecraft through
>> HAProxy as
>> >> > well? I know that Minecraft running in docker on mesos works well,
>> >> > though
>> >> > I've always used a direct connection not a proxied one.
>> >> >
>> >> >
>> >> > [1]
>> >> >
>> >> >
>> https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
>> >> >
>> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <jo...@omernik.com>
>> wrote:
>> >> >>
>> >> >> I have a nice setup with a Hive thrift server running in a docker
>> >> >> container on Mesos. It works pretty well, but something, I believe
>> in
>> >> >> how HAProxy works with the connection, is causing the thrift server
>> >> >> connection to die after a time. Basically, I can run a few
>> queries,
>> >> >> but after 2 or 3, or specifically after a longer query, I get the
>> >> >> error below indicating End of File on the connection. Then no more
>> >> >> connections work until I reestablish the connection to the thrift
>> >> >> server. I've tried looking in logs. the thrift server std err logs
>> >> >> show no issues. I am guess I need to dig into haproxy logs, but I am
>> >> >> not seeing any issues in syslog so far. I'd love any pointers on
>> how
>> >> >> to trouble shoot this. By the way, I have MySQl, hive metastore,
>> and
>> >> >> a minecraft server all running on Mesos/Docker with no issues, not
>> >> >> sure why the thrift server is so sensitive.
>> >> >>
>> >> >> :)
>> >> >>
>> >> >>
>> >> >> pyhs2 connection string:
>> >> >>
>> >> >> hs2 =
>> >> >>
>> >> >>
>> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
>> >> >>
>> >> >>
>> >> >> Error
>> >> >>
>> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc
>> in
>> >> >> read(self, sz)
>> >> >> 118 if len(buff) == 0:
>> >> >> 119 raise
>> >> >> TTransportException(type=TTransportException.END_OF_FILE,
>> >> >> --> 120 message='TSocket read 0
>> bytes')
>> >> >> 121 return buff
>> >> >> 122
>> >> >>
>> >> >> TTransportException: TSocket read 0 bytes
>> >> >
>> >> >
>> >
>> >
>>
>
>
Re: HAProxy for Hive Thrift Server on Mesos
Posted by xiaokun <xi...@gmail.com>.
After set the timeout value, the tcp request become stable.
2015-03-23 23:26 GMT+08:00 John Omernik <jo...@omernik.com>:
> So per my HA-Proxy logs, I am getting a condition where the client
> side time expired while waiting for the client to send or receive
> data, and this is occurring during the DATA phase. (the two letter
> HAProxy code is cD . To test, I expanded the timeout (on all things)
> to 100000.
>
> This was interesting in that it showed me the problem a bit clearer.
> Now, as long as I submit and results for a query in the 100 second
> range, my connection stays open and happy. If I let it sit for 100
> seconds, and then try a query, then it breaks again.
>
> It appears HA proxy gets irritable when things aren't being sent over
> a connection for a long period of time. For services like Minecraft,
> this is not an issue, when a client is connected, there is always data
> there, and no issues occur. For Thrift, there may be multiple long
> pauses as an analyst thinks through the next query etc. I wonder what
> would be the best way to handle that on a per container level. (not
> all containers need this sort of customization...)
>
>
>
> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xi...@gmail.com> wrote:
> > I have a similar error when launch jenkins in mesos cluster. And I use
> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from
> time
> > to time, connection between jenkins master and jenkins slave will lost,
> > while will cause jobs fail.
> >
> > I will try to set timeout value for HAProxy
> >
> > 2015-03-23 4:18 GMT+08:00 John Omernik <jo...@omernik.com>:
> >>
> >> Yep all three are going through HAProxy and seem to be working well.
> >> I should do some testing to determine if I run a thrift server that
> >> connects to metastore in HAProxy to determine if that's the broken
> >> link, but I believe it's between the thrift server. I will check out
> >> the link you posted to get a better idea out of how it works. I am
> >> running this all on a MapR cluster which is Great for Docker/Mesos as
> >> I used the MapR FS, mounted locally on each node's NFS to put the data
> >> for MySQL, the logs for the metastore/thift server, and my world files
> >> for Minecraft. So far it's all really slick.
> >>
> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
> >> <be...@mesosphere.io> wrote:
> >> > It sounds like you may be running into the connection timeout that is
> >> > configured for the HAProxy instance.
> >> >
> >> > You can see an instance of this in Marathon's
> >> > HAProxy-marathon-bridge[1].
> >> > You may need to raise the timeout values.
> >> >
> >> > Are you accessing MySql, Hive metastore and minecraft through HAProxy
> as
> >> > well? I know that Minecraft running in docker on mesos works well,
> >> > though
> >> > I've always used a direct connection not a proxied one.
> >> >
> >> >
> >> > [1]
> >> >
> >> >
> https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
> >> >
> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <jo...@omernik.com>
> wrote:
> >> >>
> >> >> I have a nice setup with a Hive thrift server running in a docker
> >> >> container on Mesos. It works pretty well, but something, I believe in
> >> >> how HAProxy works with the connection, is causing the thrift server
> >> >> connection to die after a time. Basically, I can run a few queries,
> >> >> but after 2 or 3, or specifically after a longer query, I get the
> >> >> error below indicating End of File on the connection. Then no more
> >> >> connections work until I reestablish the connection to the thrift
> >> >> server. I've tried looking in logs. the thrift server std err logs
> >> >> show no issues. I am guess I need to dig into haproxy logs, but I am
> >> >> not seeing any issues in syslog so far. I'd love any pointers on how
> >> >> to trouble shoot this. By the way, I have MySQl, hive metastore, and
> >> >> a minecraft server all running on Mesos/Docker with no issues, not
> >> >> sure why the thrift server is so sensitive.
> >> >>
> >> >> :)
> >> >>
> >> >>
> >> >> pyhs2 connection string:
> >> >>
> >> >> hs2 =
> >> >>
> >> >>
> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
> >> >>
> >> >>
> >> >> Error
> >> >>
> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc
> in
> >> >> read(self, sz)
> >> >> 118 if len(buff) == 0:
> >> >> 119 raise
> >> >> TTransportException(type=TTransportException.END_OF_FILE,
> >> >> --> 120 message='TSocket read 0
> bytes')
> >> >> 121 return buff
> >> >> 122
> >> >>
> >> >> TTransportException: TSocket read 0 bytes
> >> >
> >> >
> >
> >
>
Re: HAProxy for Hive Thrift Server on Mesos
Posted by John Omernik <jo...@omernik.com>.
So per my HA-Proxy logs, I am getting a condition where the client
side time expired while waiting for the client to send or receive
data, and this is occurring during the DATA phase. (the two letter
HAProxy code is cD . To test, I expanded the timeout (on all things)
to 100000.
This was interesting in that it showed me the problem a bit clearer.
Now, as long as I submit and results for a query in the 100 second
range, my connection stays open and happy. If I let it sit for 100
seconds, and then try a query, then it breaks again.
It appears HA proxy gets irritable when things aren't being sent over
a connection for a long period of time. For services like Minecraft,
this is not an issue, when a client is connected, there is always data
there, and no issues occur. For Thrift, there may be multiple long
pauses as an analyst thinks through the next query etc. I wonder what
would be the best way to handle that on a per container level. (not
all containers need this sort of customization...)
On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xi...@gmail.com> wrote:
> I have a similar error when launch jenkins in mesos cluster. And I use
> HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time
> to time, connection between jenkins master and jenkins slave will lost,
> while will cause jobs fail.
>
> I will try to set timeout value for HAProxy
>
> 2015-03-23 4:18 GMT+08:00 John Omernik <jo...@omernik.com>:
>>
>> Yep all three are going through HAProxy and seem to be working well.
>> I should do some testing to determine if I run a thrift server that
>> connects to metastore in HAProxy to determine if that's the broken
>> link, but I believe it's between the thrift server. I will check out
>> the link you posted to get a better idea out of how it works. I am
>> running this all on a MapR cluster which is Great for Docker/Mesos as
>> I used the MapR FS, mounted locally on each node's NFS to put the data
>> for MySQL, the logs for the metastore/thift server, and my world files
>> for Minecraft. So far it's all really slick.
>>
>> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
>> <be...@mesosphere.io> wrote:
>> > It sounds like you may be running into the connection timeout that is
>> > configured for the HAProxy instance.
>> >
>> > You can see an instance of this in Marathon's
>> > HAProxy-marathon-bridge[1].
>> > You may need to raise the timeout values.
>> >
>> > Are you accessing MySql, Hive metastore and minecraft through HAProxy as
>> > well? I know that Minecraft running in docker on mesos works well,
>> > though
>> > I've always used a direct connection not a proxied one.
>> >
>> >
>> > [1]
>> >
>> > https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
>> >
>> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <jo...@omernik.com> wrote:
>> >>
>> >> I have a nice setup with a Hive thrift server running in a docker
>> >> container on Mesos. It works pretty well, but something, I believe in
>> >> how HAProxy works with the connection, is causing the thrift server
>> >> connection to die after a time. Basically, I can run a few queries,
>> >> but after 2 or 3, or specifically after a longer query, I get the
>> >> error below indicating End of File on the connection. Then no more
>> >> connections work until I reestablish the connection to the thrift
>> >> server. I've tried looking in logs. the thrift server std err logs
>> >> show no issues. I am guess I need to dig into haproxy logs, but I am
>> >> not seeing any issues in syslog so far. I'd love any pointers on how
>> >> to trouble shoot this. By the way, I have MySQl, hive metastore, and
>> >> a minecraft server all running on Mesos/Docker with no issues, not
>> >> sure why the thrift server is so sensitive.
>> >>
>> >> :)
>> >>
>> >>
>> >> pyhs2 connection string:
>> >>
>> >> hs2 =
>> >>
>> >> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
>> >>
>> >>
>> >> Error
>> >>
>> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in
>> >> read(self, sz)
>> >> 118 if len(buff) == 0:
>> >> 119 raise
>> >> TTransportException(type=TTransportException.END_OF_FILE,
>> >> --> 120 message='TSocket read 0 bytes')
>> >> 121 return buff
>> >> 122
>> >>
>> >> TTransportException: TSocket read 0 bytes
>> >
>> >
>
>
Re: HAProxy for Hive Thrift Server on Mesos
Posted by xiaokun <xi...@gmail.com>.
I have a similar error when launch jenkins in mesos cluster. And I use
HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time
to time, connection between jenkins master and jenkins slave will lost,
while will cause jobs fail.
I will try to set timeout value for HAProxy
2015-03-23 4:18 GMT+08:00 John Omernik <jo...@omernik.com>:
> Yep all three are going through HAProxy and seem to be working well.
> I should do some testing to determine if I run a thrift server that
> connects to metastore in HAProxy to determine if that's the broken
> link, but I believe it's between the thrift server. I will check out
> the link you posted to get a better idea out of how it works. I am
> running this all on a MapR cluster which is Great for Docker/Mesos as
> I used the MapR FS, mounted locally on each node's NFS to put the data
> for MySQL, the logs for the metastore/thift server, and my world files
> for Minecraft. So far it's all really slick.
>
> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
> <be...@mesosphere.io> wrote:
> > It sounds like you may be running into the connection timeout that is
> > configured for the HAProxy instance.
> >
> > You can see an instance of this in Marathon's HAProxy-marathon-bridge[1].
> > You may need to raise the timeout values.
> >
> > Are you accessing MySql, Hive metastore and minecraft through HAProxy as
> > well? I know that Minecraft running in docker on mesos works well, though
> > I've always used a direct connection not a proxied one.
> >
> >
> > [1]
> >
> https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
> >
> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <jo...@omernik.com> wrote:
> >>
> >> I have a nice setup with a Hive thrift server running in a docker
> >> container on Mesos. It works pretty well, but something, I believe in
> >> how HAProxy works with the connection, is causing the thrift server
> >> connection to die after a time. Basically, I can run a few queries,
> >> but after 2 or 3, or specifically after a longer query, I get the
> >> error below indicating End of File on the connection. Then no more
> >> connections work until I reestablish the connection to the thrift
> >> server. I've tried looking in logs. the thrift server std err logs
> >> show no issues. I am guess I need to dig into haproxy logs, but I am
> >> not seeing any issues in syslog so far. I'd love any pointers on how
> >> to trouble shoot this. By the way, I have MySQl, hive metastore, and
> >> a minecraft server all running on Mesos/Docker with no issues, not
> >> sure why the thrift server is so sensitive.
> >>
> >> :)
> >>
> >>
> >> pyhs2 connection string:
> >>
> >> hs2 =
> >>
> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
> >>
> >>
> >> Error
> >>
> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in
> >> read(self, sz)
> >> 118 if len(buff) == 0:
> >> 119 raise
> >> TTransportException(type=TTransportException.END_OF_FILE,
> >> --> 120 message='TSocket read 0 bytes')
> >> 121 return buff
> >> 122
> >>
> >> TTransportException: TSocket read 0 bytes
> >
> >
>
Re: HAProxy for Hive Thrift Server on Mesos
Posted by John Omernik <jo...@omernik.com>.
Yep all three are going through HAProxy and seem to be working well.
I should do some testing to determine if I run a thrift server that
connects to metastore in HAProxy to determine if that's the broken
link, but I believe it's between the thrift server. I will check out
the link you posted to get a better idea out of how it works. I am
running this all on a MapR cluster which is Great for Docker/Mesos as
I used the MapR FS, mounted locally on each node's NFS to put the data
for MySQL, the logs for the metastore/thift server, and my world files
for Minecraft. So far it's all really slick.
On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead
<be...@mesosphere.io> wrote:
> It sounds like you may be running into the connection timeout that is
> configured for the HAProxy instance.
>
> You can see an instance of this in Marathon's HAProxy-marathon-bridge[1].
> You may need to raise the timeout values.
>
> Are you accessing MySql, Hive metastore and minecraft through HAProxy as
> well? I know that Minecraft running in docker on mesos works well, though
> I've always used a direct connection not a proxied one.
>
>
> [1]
> https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
>
> On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <jo...@omernik.com> wrote:
>>
>> I have a nice setup with a Hive thrift server running in a docker
>> container on Mesos. It works pretty well, but something, I believe in
>> how HAProxy works with the connection, is causing the thrift server
>> connection to die after a time. Basically, I can run a few queries,
>> but after 2 or 3, or specifically after a longer query, I get the
>> error below indicating End of File on the connection. Then no more
>> connections work until I reestablish the connection to the thrift
>> server. I've tried looking in logs. the thrift server std err logs
>> show no issues. I am guess I need to dig into haproxy logs, but I am
>> not seeing any issues in syslog so far. I'd love any pointers on how
>> to trouble shoot this. By the way, I have MySQl, hive metastore, and
>> a minecraft server all running on Mesos/Docker with no issues, not
>> sure why the thrift server is so sensitive.
>>
>> :)
>>
>>
>> pyhs2 connection string:
>>
>> hs2 =
>> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
>>
>>
>> Error
>>
>> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in
>> read(self, sz)
>> 118 if len(buff) == 0:
>> 119 raise
>> TTransportException(type=TTransportException.END_OF_FILE,
>> --> 120 message='TSocket read 0 bytes')
>> 121 return buff
>> 122
>>
>> TTransportException: TSocket read 0 bytes
>
>
Re: HAProxy for Hive Thrift Server on Mesos
Posted by Ben Whitehead <be...@mesosphere.io>.
It sounds like you may be running into the connection timeout that is
configured for the HAProxy instance.
You can see an instance of this in Marathon's HAProxy-marathon-bridge[1].
You may need to raise the timeout values.
Are you accessing MySql, Hive metastore and minecraft through HAProxy as
well? I know that Minecraft running in docker on mesos works well, though
I've always used a direct connection not a proxied one.
[1]
https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119
On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <jo...@omernik.com> wrote:
> I have a nice setup with a Hive thrift server running in a docker
> container on Mesos. It works pretty well, but something, I believe in
> how HAProxy works with the connection, is causing the thrift server
> connection to die after a time. Basically, I can run a few queries,
> but after 2 or 3, or specifically after a longer query, I get the
> error below indicating End of File on the connection. Then no more
> connections work until I reestablish the connection to the thrift
> server. I've tried looking in logs. the thrift server std err logs
> show no issues. I am guess I need to dig into haproxy logs, but I am
> not seeing any issues in syslog so far. I'd love any pointers on how
> to trouble shoot this. By the way, I have MySQl, hive metastore, and
> a minecraft server all running on Mesos/Docker with no issues, not
> sure why the thrift server is so sensitive.
>
> :)
>
>
> pyhs2 connection string:
>
> hs2 =
> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default')
>
>
> Error
>
> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in
> read(self, sz)
> 118 if len(buff) == 0:
> 119 raise
> TTransportException(type=TTransportException.END_OF_FILE,
> --> 120 message='TSocket read 0 bytes')
> 121 return buff
> 122
>
> TTransportException: TSocket read 0 bytes
>