You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Wojciech Langiewicz <wl...@gmail.com> on 2011/01/21 17:12:46 UTC

HBase performance evaluation master not running problem.

Hello
I have problem with running HBase performance tests from 
org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using version 
from CDH3,
Tests are ok when argument nclients is 1, but in case of greater number, 
after mappers reaching 100% I get this exception (I didn't test all 
tests, but all of tested by me failed, 'scan' and 'randomRead' fail for 
sure):

11/01/21 17:05:19 INFO mapred.JobClient: Task Id : 
attempt_201101211442_0005_m_000000_0, Status : FAILED
org.apache.hadoop.hbase.MasterNotRunningException
	at 
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
	at org.apache.hadoop.mapred.Child.main(Child.java:211)

Do you have any ideas how to solve this?

--
Wojciech Langiewicz

Re: HBase performance evaluation master not running problem.

Posted by Wojciech Langiewicz <wl...@gmail.com>.
Hi,
This is also not this, mappers fail even if I run 2, and I have changed 
that value to 60 and tested it.

--
Wojciech Langiewicz

On 22.01.2011 20:18, Stack wrote:
> Does it work if you have<  30 mappers running but if you go>  30, some
> start to fail?  Perhaps it this config:
> hbase.zookeeper.property.maxClientCnxns.  Its default is 30 max
> connections per host.
> St.Ack
>
>
>
> On Sat, Jan 22, 2011 at 10:42 AM, Wojciech Langiewicz
> <wl...@gmail.com>  wrote:
>> I have changed it to 120000, and there's no change.
>> What is worth noticing is that some mappers finish task without any error,
>> and during the same job, the same server can fail some tasks, and finishes
>> other.
>>
>>
>> On 22.01.2011 18:56, Wojciech Langiewicz wrote:
>>>
>>> Hi,
>>> sessionTimeout=60000,
>>> I didn't change it, so it's default.
>>> What value do you recommend?
>>>
>>> --
>>> Wojciech Langiewicz
>>>
>>> On 22.01.2011 16:49, Ted Yu wrote:
>>>>
>>>> What's the value of 'zookeeper.session.timeout' ?
>>>>
>>>> Maybe you can tune it higher.
>>>>
>>>> On Sat, Jan 22, 2011 at 3:13 AM, Wojciech Langiewicz
>>>> <wl...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>> I have re-run test with 2 or more mappers and looked into logs more
>>>>> closely: The mapreduce job has finished correctly, some map attempts are
>>>>> killed, but eventually all of mappers finish. And there's no rule which
>>>>> mappers on which servers fail (re-run tests multiple times).
>>>>>
>>>>> So I suspect that this is not classpath problem, maybe there are
>>>>> settings
>>>>> that limit number of connections, because before MasterNotRunning I
>>>>> get this
>>>>> in logs:
>>>>>
>>>>> 2011-01-22 12:06:43,411 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>>>> client connection, connectString=hd-master:2181 sessionTimeout=60000
>>>>> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@3b5e234c
>>>>> 2011-01-22 12:06:43,483 INFO org.apache.zookeeper.ClientCnxn: Opening
>>>>> socket connection to server hd-master/10.6.75.212:2181
>>>>> 2011-01-22 12:06:43,484 INFO org.apache.zookeeper.ClientCnxn: Socket
>>>>> connection established to hd-master/10.6.75.212:2181, initiating session
>>>>> 2011-01-22 12:06:43,488 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>>> read additional data from server sessionid 0x0, likely server has closed
>>>>> socket, closing socket connection and attempting reconnect
>>>>> 2011-01-22 12:06:43,605 INFO
>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
>>>>> getMaster
>>>>> attempt 0 of 10 failed; retrying after sleep of 1000
>>>>> java.io.IOException:
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
>>>>>
>>>>>
>>>>> at
>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>
>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
>>>>>
>>>>> ... 16 more
>>>>>
>>>>>
>>>>> On 22.01.2011 01:36, Stack wrote:
>>>>>
>>>>>> Then its odd that PE fails. Can you figure difference between the two
>>>>>> environments? Perhaps your MR jobs are fat jars that include the conf
>>>>>> and all dependencies whereas PE is dump and expects the dependencies
>>>>>> and conf on CLASSPATH?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
>>>>>> <wl...@gmail.com>  wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I have other mapreduce tasks that are running on this cluster and
>>>>>>> using
>>>>>>> HBase and they are working correctly. All my servers have the same
>>>>>>> configuration.
>>>>>>>
>>>>>>> --
>>>>>>> Wojciech Langiewicz
>>>>>>>
>>>>>>> On 21.01.2011 19:20, Stack wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> When clients are>  1, then PE tries to run a mapreduce job to host the
>>>>>>>> loading clients.
>>>>>>>>
>>>>>>>> Is it possible that the client out in MR task is trying to connect to
>>>>>>>> wrong location? Perhaps the HBase conf dir is not available to the
>>>>>>>> running task? Have you seen
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
>>>>>>>>
>>>>>>>> ?
>>>>>>>> Perhaps this will help?
>>>>>>>>
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>>>>>>>> <wl...@gmail.com>  wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>> I have problem with running HBase performance tests from
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using
>>>>>>>>> version
>>>>>>>>> from CDH3,
>>>>>>>>> Tests are ok when argument nclients is 1, but in case of greater
>>>>>>>>> number,
>>>>>>>>> after mappers reaching 100% I get this exception (I didn't test all
>>>>>>>>> tests,
>>>>>>>>> but all of tested by me failed, 'scan' and 'randomRead' fail for
>>>>>>>>> sure):
>>>>>>>>>
>>>>>>>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>>>>>>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>>>>>>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>>>>>
>>>>>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>>>>>
>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>>>>>>
>>>>>>>>> Do you have any ideas how to solve this?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Wojciech Langiewicz
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>>


Re: Building LZO jar

Posted by Todd Lipcon <to...@cloudera.com>.
GitHub Pull Requests happily accepted! :-D

-Todd

On Thu, Jan 27, 2011 at 12:40 AM, Friso van Vollenhoven <
fvanvollenhoven@xebia.com> wrote:

> Ah, I see. It's an undocumented feature...
>
> Thanks Lars.
>
>
> On 27 jan 2011, at 08:21, Lars George wrote:
>
> > Hi Friso,
> >
> > See
> https://github.com/toddlipcon/hadoop-lzo-packager/blob/master/run.sh#L127-133
> >
> > Todd's is the most current AFAIK.
> >
> > Lars
> >
> > On Wed, Jan 26, 2011 at 2:14 PM, Friso van Vollenhoven
> > <fv...@xebia.com> wrote:
> >> Hi Lars,
> >>
> >> The readme for this talks about the LZO from Google Code and using svn
> to get it. Does it also work with the more recent versions on Github?
> >>
> >> I usually build these things manually and move things around myself.
> Also because I always try to provide one hadoop+hbase binary + source dist
> internally that has all the native libs for Linux and OSX, so it also works
> for the local installs on dev machines...
> >>
> >>
> >> Friso
> >>
> >>
> >>
> >> On 26 jan 2011, at 12:38, Lars George wrote:
> >>
> >>> I agree with Friso, using Todd's LZO Packager this is really easy:
> >>> https://github.com/toddlipcon/hadoop-lzo-packager
> >>>
> >>> Lars
> >>>
> >>>
> >>> On Wed, Jan 26, 2011 at 10:41 AM, Friso van Vollenhoven
> >>> <fv...@xebia.com> wrote:
> >>>> Are you sure it is not a problem with the network on your side?
> Clicking the link and downloading that jar works fine for me.
> >>>>
> >>>> I don't think jar files are either 32 or 64 bit, but the native
> libraries that go with LZO are. I would recommend building them on your
> target platform/architecture and not use libs that are built elsewhere.
> >>>>
> >>>>
> >>>> Friso
> >>>>
> >>>>
> >>>>
> >>>> On 25 jan 2011, at 21:34, Peter Haidinyak wrote:
> >>>>
> >>>>> Hi, I am kind of in a pickle right now. I enabled LZO compression on
> a table without having the correct lzo jar. Now HBase doesn't startup
> because of the missing jar. I am trying to build the LZO jar but Ivy, for
> some reason, can't connect to
> >>>>>
> >>>>>
> http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
> >>>>>
> >>>>> to download the jar. I have tried running the build in offline mode
> but the Ivy gets stuck at..
> >>>>>
> >>>>> ivy-resolve-common:
> >>>>> [ivy:resolve] :: resolving dependencies ::
> com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
> >>>>> [ivy:resolve]   confs: [common]
> >>>>>
> >>>>> Would anyone have a pre-build 32bit LZO jar available?
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> -Pete
> >>>>
> >>>>
> >>
> >>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Building LZO jar

Posted by Friso van Vollenhoven <fv...@xebia.com>.
Ah, I see. It's an undocumented feature...

Thanks Lars.


On 27 jan 2011, at 08:21, Lars George wrote:

> Hi Friso,
> 
> See https://github.com/toddlipcon/hadoop-lzo-packager/blob/master/run.sh#L127-133
> 
> Todd's is the most current AFAIK.
> 
> Lars
> 
> On Wed, Jan 26, 2011 at 2:14 PM, Friso van Vollenhoven
> <fv...@xebia.com> wrote:
>> Hi Lars,
>> 
>> The readme for this talks about the LZO from Google Code and using svn to get it. Does it also work with the more recent versions on Github?
>> 
>> I usually build these things manually and move things around myself. Also because I always try to provide one hadoop+hbase binary + source dist internally that has all the native libs for Linux and OSX, so it also works for the local installs on dev machines...
>> 
>> 
>> Friso
>> 
>> 
>> 
>> On 26 jan 2011, at 12:38, Lars George wrote:
>> 
>>> I agree with Friso, using Todd's LZO Packager this is really easy:
>>> https://github.com/toddlipcon/hadoop-lzo-packager
>>> 
>>> Lars
>>> 
>>> 
>>> On Wed, Jan 26, 2011 at 10:41 AM, Friso van Vollenhoven
>>> <fv...@xebia.com> wrote:
>>>> Are you sure it is not a problem with the network on your side? Clicking the link and downloading that jar works fine for me.
>>>> 
>>>> I don't think jar files are either 32 or 64 bit, but the native libraries that go with LZO are. I would recommend building them on your target platform/architecture and not use libs that are built elsewhere.
>>>> 
>>>> 
>>>> Friso
>>>> 
>>>> 
>>>> 
>>>> On 25 jan 2011, at 21:34, Peter Haidinyak wrote:
>>>> 
>>>>> Hi, I am kind of in a pickle right now. I enabled LZO compression on a table without having the correct lzo jar. Now HBase doesn't startup because of the missing jar. I am trying to build the LZO jar but Ivy, for some reason, can't connect to
>>>>> 
>>>>> http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
>>>>> 
>>>>> to download the jar. I have tried running the build in offline mode but the Ivy gets stuck at..
>>>>> 
>>>>> ivy-resolve-common:
>>>>> [ivy:resolve] :: resolving dependencies :: com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
>>>>> [ivy:resolve]   confs: [common]
>>>>> 
>>>>> Would anyone have a pre-build 32bit LZO jar available?
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> -Pete
>>>> 
>>>> 
>> 
>> 


Re: Building LZO jar

Posted by Lars George <la...@gmail.com>.
Hi Friso,

See https://github.com/toddlipcon/hadoop-lzo-packager/blob/master/run.sh#L127-133

Todd's is the most current AFAIK.

Lars

On Wed, Jan 26, 2011 at 2:14 PM, Friso van Vollenhoven
<fv...@xebia.com> wrote:
> Hi Lars,
>
> The readme for this talks about the LZO from Google Code and using svn to get it. Does it also work with the more recent versions on Github?
>
> I usually build these things manually and move things around myself. Also because I always try to provide one hadoop+hbase binary + source dist internally that has all the native libs for Linux and OSX, so it also works for the local installs on dev machines...
>
>
> Friso
>
>
>
> On 26 jan 2011, at 12:38, Lars George wrote:
>
>> I agree with Friso, using Todd's LZO Packager this is really easy:
>> https://github.com/toddlipcon/hadoop-lzo-packager
>>
>> Lars
>>
>>
>> On Wed, Jan 26, 2011 at 10:41 AM, Friso van Vollenhoven
>> <fv...@xebia.com> wrote:
>>> Are you sure it is not a problem with the network on your side? Clicking the link and downloading that jar works fine for me.
>>>
>>> I don't think jar files are either 32 or 64 bit, but the native libraries that go with LZO are. I would recommend building them on your target platform/architecture and not use libs that are built elsewhere.
>>>
>>>
>>> Friso
>>>
>>>
>>>
>>> On 25 jan 2011, at 21:34, Peter Haidinyak wrote:
>>>
>>>> Hi, I am kind of in a pickle right now. I enabled LZO compression on a table without having the correct lzo jar. Now HBase doesn't startup because of the missing jar. I am trying to build the LZO jar but Ivy, for some reason, can't connect to
>>>>
>>>> http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
>>>>
>>>> to download the jar. I have tried running the build in offline mode but the Ivy gets stuck at..
>>>>
>>>> ivy-resolve-common:
>>>> [ivy:resolve] :: resolving dependencies :: com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
>>>> [ivy:resolve]   confs: [common]
>>>>
>>>> Would anyone have a pre-build 32bit LZO jar available?
>>>>
>>>> Thanks
>>>>
>>>> -Pete
>>>
>>>
>
>

Re: Building LZO jar

Posted by Friso van Vollenhoven <fv...@xebia.com>.
Hi Lars,

The readme for this talks about the LZO from Google Code and using svn to get it. Does it also work with the more recent versions on Github?

I usually build these things manually and move things around myself. Also because I always try to provide one hadoop+hbase binary + source dist internally that has all the native libs for Linux and OSX, so it also works for the local installs on dev machines...


Friso



On 26 jan 2011, at 12:38, Lars George wrote:

> I agree with Friso, using Todd's LZO Packager this is really easy:
> https://github.com/toddlipcon/hadoop-lzo-packager
> 
> Lars
> 
> 
> On Wed, Jan 26, 2011 at 10:41 AM, Friso van Vollenhoven
> <fv...@xebia.com> wrote:
>> Are you sure it is not a problem with the network on your side? Clicking the link and downloading that jar works fine for me.
>> 
>> I don't think jar files are either 32 or 64 bit, but the native libraries that go with LZO are. I would recommend building them on your target platform/architecture and not use libs that are built elsewhere.
>> 
>> 
>> Friso
>> 
>> 
>> 
>> On 25 jan 2011, at 21:34, Peter Haidinyak wrote:
>> 
>>> Hi, I am kind of in a pickle right now. I enabled LZO compression on a table without having the correct lzo jar. Now HBase doesn't startup because of the missing jar. I am trying to build the LZO jar but Ivy, for some reason, can't connect to
>>> 
>>> http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
>>> 
>>> to download the jar. I have tried running the build in offline mode but the Ivy gets stuck at..
>>> 
>>> ivy-resolve-common:
>>> [ivy:resolve] :: resolving dependencies :: com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
>>> [ivy:resolve]   confs: [common]
>>> 
>>> Would anyone have a pre-build 32bit LZO jar available?
>>> 
>>> Thanks
>>> 
>>> -Pete
>> 
>> 


Re: Building LZO jar

Posted by Lars George <la...@gmail.com>.
I agree with Friso, using Todd's LZO Packager this is really easy:
https://github.com/toddlipcon/hadoop-lzo-packager

Lars


On Wed, Jan 26, 2011 at 10:41 AM, Friso van Vollenhoven
<fv...@xebia.com> wrote:
> Are you sure it is not a problem with the network on your side? Clicking the link and downloading that jar works fine for me.
>
> I don't think jar files are either 32 or 64 bit, but the native libraries that go with LZO are. I would recommend building them on your target platform/architecture and not use libs that are built elsewhere.
>
>
> Friso
>
>
>
> On 25 jan 2011, at 21:34, Peter Haidinyak wrote:
>
>> Hi, I am kind of in a pickle right now. I enabled LZO compression on a table without having the correct lzo jar. Now HBase doesn't startup because of the missing jar. I am trying to build the LZO jar but Ivy, for some reason, can't connect to
>>
>> http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
>>
>> to download the jar. I have tried running the build in offline mode but the Ivy gets stuck at..
>>
>> ivy-resolve-common:
>> [ivy:resolve] :: resolving dependencies :: com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
>> [ivy:resolve]   confs: [common]
>>
>> Would anyone have a pre-build 32bit LZO jar available?
>>
>> Thanks
>>
>> -Pete
>
>

Re: Building LZO jar

Posted by Friso van Vollenhoven <fv...@xebia.com>.
Are you sure it is not a problem with the network on your side? Clicking the link and downloading that jar works fine for me.

I don't think jar files are either 32 or 64 bit, but the native libraries that go with LZO are. I would recommend building them on your target platform/architecture and not use libs that are built elsewhere.


Friso



On 25 jan 2011, at 21:34, Peter Haidinyak wrote:

> Hi, I am kind of in a pickle right now. I enabled LZO compression on a table without having the correct lzo jar. Now HBase doesn't startup because of the missing jar. I am trying to build the LZO jar but Ivy, for some reason, can't connect to 
> 
> http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
> 
> to download the jar. I have tried running the build in offline mode but the Ivy gets stuck at..
> 
> ivy-resolve-common:
> [ivy:resolve] :: resolving dependencies :: com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
> [ivy:resolve]   confs: [common]
> 
> Would anyone have a pre-build 32bit LZO jar available?
> 
> Thanks
> 
> -Pete


Building LZO jar

Posted by Peter Haidinyak <ph...@local.com>.
Hi, I am kind of in a pickle right now. I enabled LZO compression on a table without having the correct lzo jar. Now HBase doesn't startup because of the missing jar. I am trying to build the LZO jar but Ivy, for some reason, can't connect to 

http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar

to download the jar. I have tried running the build in offline mode but the Ivy gets stuck at..

ivy-resolve-common:
[ivy:resolve] :: resolving dependencies :: com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
[ivy:resolve]   confs: [common]

Would anyone have a pre-build 32bit LZO jar available?

Thanks

-Pete

Re: HBase performance evaluation master not running problem.

Posted by Wojciech Langiewicz <wl...@gmail.com>.
Hello,
I have found out what the problem was, and you were right, it was 
problem with maxClientCnxns setting, but I still don't know why setting 
this option in hbase-site.xml didn't affect anything.
HBase was reading zookeeper settings from /etc/zookeeper/zoo.cfg even 
when I set HBase to manage zookeeper.

Thanks for helping me out.

On 22.01.2011 20:18, Stack wrote:
> Does it work if you have<  30 mappers running but if you go>  30, some
> start to fail?  Perhaps it this config:
> hbase.zookeeper.property.maxClientCnxns.  Its default is 30 max
> connections per host.
> St.Ack
>
>
>
> On Sat, Jan 22, 2011 at 10:42 AM, Wojciech Langiewicz
> <wl...@gmail.com>  wrote:
>> I have changed it to 120000, and there's no change.
>> What is worth noticing is that some mappers finish task without any error,
>> and during the same job, the same server can fail some tasks, and finishes
>> other.
>>
>>
>> On 22.01.2011 18:56, Wojciech Langiewicz wrote:
>>>
>>> Hi,
>>> sessionTimeout=60000,
>>> I didn't change it, so it's default.
>>> What value do you recommend?
>>>
>>> --
>>> Wojciech Langiewicz
>>>
>>> On 22.01.2011 16:49, Ted Yu wrote:
>>>>
>>>> What's the value of 'zookeeper.session.timeout' ?
>>>>
>>>> Maybe you can tune it higher.
>>>>
>>>> On Sat, Jan 22, 2011 at 3:13 AM, Wojciech Langiewicz
>>>> <wl...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>> I have re-run test with 2 or more mappers and looked into logs more
>>>>> closely: The mapreduce job has finished correctly, some map attempts are
>>>>> killed, but eventually all of mappers finish. And there's no rule which
>>>>> mappers on which servers fail (re-run tests multiple times).
>>>>>
>>>>> So I suspect that this is not classpath problem, maybe there are
>>>>> settings
>>>>> that limit number of connections, because before MasterNotRunning I
>>>>> get this
>>>>> in logs:
>>>>>
>>>>> 2011-01-22 12:06:43,411 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>>>> client connection, connectString=hd-master:2181 sessionTimeout=60000
>>>>> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@3b5e234c
>>>>> 2011-01-22 12:06:43,483 INFO org.apache.zookeeper.ClientCnxn: Opening
>>>>> socket connection to server hd-master/10.6.75.212:2181
>>>>> 2011-01-22 12:06:43,484 INFO org.apache.zookeeper.ClientCnxn: Socket
>>>>> connection established to hd-master/10.6.75.212:2181, initiating session
>>>>> 2011-01-22 12:06:43,488 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>>> read additional data from server sessionid 0x0, likely server has closed
>>>>> socket, closing socket connection and attempting reconnect
>>>>> 2011-01-22 12:06:43,605 INFO
>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
>>>>> getMaster
>>>>> attempt 0 of 10 failed; retrying after sleep of 1000
>>>>> java.io.IOException:
>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
>>>>>
>>>>>
>>>>> at
>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>
>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>
>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>>> at
>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
>>>>>
>>>>> ... 16 more
>>>>>
>>>>>
>>>>> On 22.01.2011 01:36, Stack wrote:
>>>>>
>>>>>> Then its odd that PE fails. Can you figure difference between the two
>>>>>> environments? Perhaps your MR jobs are fat jars that include the conf
>>>>>> and all dependencies whereas PE is dump and expects the dependencies
>>>>>> and conf on CLASSPATH?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
>>>>>> <wl...@gmail.com>  wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I have other mapreduce tasks that are running on this cluster and
>>>>>>> using
>>>>>>> HBase and they are working correctly. All my servers have the same
>>>>>>> configuration.
>>>>>>>
>>>>>>> --
>>>>>>> Wojciech Langiewicz
>>>>>>>
>>>>>>> On 21.01.2011 19:20, Stack wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> When clients are>  1, then PE tries to run a mapreduce job to host the
>>>>>>>> loading clients.
>>>>>>>>
>>>>>>>> Is it possible that the client out in MR task is trying to connect to
>>>>>>>> wrong location? Perhaps the HBase conf dir is not available to the
>>>>>>>> running task? Have you seen
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
>>>>>>>>
>>>>>>>> ?
>>>>>>>> Perhaps this will help?
>>>>>>>>
>>>>>>>> St.Ack
>>>>>>>>
>>>>>>>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>>>>>>>> <wl...@gmail.com>  wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>> I have problem with running HBase performance tests from
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using
>>>>>>>>> version
>>>>>>>>> from CDH3,
>>>>>>>>> Tests are ok when argument nclients is 1, but in case of greater
>>>>>>>>> number,
>>>>>>>>> after mappers reaching 100% I get this exception (I didn't test all
>>>>>>>>> tests,
>>>>>>>>> but all of tested by me failed, 'scan' and 'randomRead' fail for
>>>>>>>>> sure):
>>>>>>>>>
>>>>>>>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>>>>>>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>>>>>>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>>>>>
>>>>>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>>>>>
>>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>>>>>>
>>>>>>>>> Do you have any ideas how to solve this?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Wojciech Langiewicz
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>>


Re: HBase performance evaluation master not running problem.

Posted by Stack <st...@duboce.net>.
Does it work if you have < 30 mappers running but if you go > 30, some
start to fail?  Perhaps it this config:
hbase.zookeeper.property.maxClientCnxns.  Its default is 30 max
connections per host.
St.Ack



On Sat, Jan 22, 2011 at 10:42 AM, Wojciech Langiewicz
<wl...@gmail.com> wrote:
> I have changed it to 120000, and there's no change.
> What is worth noticing is that some mappers finish task without any error,
> and during the same job, the same server can fail some tasks, and finishes
> other.
>
>
> On 22.01.2011 18:56, Wojciech Langiewicz wrote:
>>
>> Hi,
>> sessionTimeout=60000,
>> I didn't change it, so it's default.
>> What value do you recommend?
>>
>> --
>> Wojciech Langiewicz
>>
>> On 22.01.2011 16:49, Ted Yu wrote:
>>>
>>> What's the value of 'zookeeper.session.timeout' ?
>>>
>>> Maybe you can tune it higher.
>>>
>>> On Sat, Jan 22, 2011 at 3:13 AM, Wojciech Langiewicz
>>> <wl...@gmail.com>wrote:
>>>
>>>> Hi,
>>>> I have re-run test with 2 or more mappers and looked into logs more
>>>> closely: The mapreduce job has finished correctly, some map attempts are
>>>> killed, but eventually all of mappers finish. And there's no rule which
>>>> mappers on which servers fail (re-run tests multiple times).
>>>>
>>>> So I suspect that this is not classpath problem, maybe there are
>>>> settings
>>>> that limit number of connections, because before MasterNotRunning I
>>>> get this
>>>> in logs:
>>>>
>>>> 2011-01-22 12:06:43,411 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>>> client connection, connectString=hd-master:2181 sessionTimeout=60000
>>>> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@3b5e234c
>>>> 2011-01-22 12:06:43,483 INFO org.apache.zookeeper.ClientCnxn: Opening
>>>> socket connection to server hd-master/10.6.75.212:2181
>>>> 2011-01-22 12:06:43,484 INFO org.apache.zookeeper.ClientCnxn: Socket
>>>> connection established to hd-master/10.6.75.212:2181, initiating session
>>>> 2011-01-22 12:06:43,488 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>>> read additional data from server sessionid 0x0, likely server has closed
>>>> socket, closing socket connection and attempting reconnect
>>>> 2011-01-22 12:06:43,605 INFO
>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
>>>> getMaster
>>>> attempt 0 of 10 failed; retrying after sleep of 1000
>>>> java.io.IOException:
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
>>>>
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
>>>>
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
>>>>
>>>>
>>>> at
>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>
>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>>
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>
>>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>>>> at
>>>>
>>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
>>>>
>>>> ... 16 more
>>>>
>>>>
>>>> On 22.01.2011 01:36, Stack wrote:
>>>>
>>>>> Then its odd that PE fails. Can you figure difference between the two
>>>>> environments? Perhaps your MR jobs are fat jars that include the conf
>>>>> and all dependencies whereas PE is dump and expects the dependencies
>>>>> and conf on CLASSPATH?
>>>>>
>>>>> St.Ack
>>>>>
>>>>> On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
>>>>> <wl...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I have other mapreduce tasks that are running on this cluster and
>>>>>> using
>>>>>> HBase and they are working correctly. All my servers have the same
>>>>>> configuration.
>>>>>>
>>>>>> --
>>>>>> Wojciech Langiewicz
>>>>>>
>>>>>> On 21.01.2011 19:20, Stack wrote:
>>>>>>
>>>>>>>
>>>>>>> When clients are> 1, then PE tries to run a mapreduce job to host the
>>>>>>> loading clients.
>>>>>>>
>>>>>>> Is it possible that the client out in MR task is trying to connect to
>>>>>>> wrong location? Perhaps the HBase conf dir is not available to the
>>>>>>> running task? Have you seen
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
>>>>>>>
>>>>>>> ?
>>>>>>> Perhaps this will help?
>>>>>>>
>>>>>>> St.Ack
>>>>>>>
>>>>>>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>>>>>>> <wl...@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hello
>>>>>>>> I have problem with running HBase performance tests from
>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using
>>>>>>>> version
>>>>>>>> from CDH3,
>>>>>>>> Tests are ok when argument nclients is 1, but in case of greater
>>>>>>>> number,
>>>>>>>> after mappers reaching 100% I get this exception (I didn't test all
>>>>>>>> tests,
>>>>>>>> but all of tested by me failed, 'scan' and 'randomRead' fail for
>>>>>>>> sure):
>>>>>>>>
>>>>>>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>>>>>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>>>>>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>>>>
>>>>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>>>>
>>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>>>>>
>>>>>>>> Do you have any ideas how to solve this?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Wojciech Langiewicz
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>
>
>

Re: HBase performance evaluation master not running problem.

Posted by Wojciech Langiewicz <wl...@gmail.com>.
I have changed it to 120000, and there's no change.
What is worth noticing is that some mappers finish task without any 
error, and during the same job, the same server can fail some tasks, and 
finishes other.


On 22.01.2011 18:56, Wojciech Langiewicz wrote:
> Hi,
> sessionTimeout=60000,
> I didn't change it, so it's default.
> What value do you recommend?
>
> --
> Wojciech Langiewicz
>
> On 22.01.2011 16:49, Ted Yu wrote:
>> What's the value of 'zookeeper.session.timeout' ?
>>
>> Maybe you can tune it higher.
>>
>> On Sat, Jan 22, 2011 at 3:13 AM, Wojciech Langiewicz
>> <wl...@gmail.com>wrote:
>>
>>> Hi,
>>> I have re-run test with 2 or more mappers and looked into logs more
>>> closely: The mapreduce job has finished correctly, some map attempts are
>>> killed, but eventually all of mappers finish. And there's no rule which
>>> mappers on which servers fail (re-run tests multiple times).
>>>
>>> So I suspect that this is not classpath problem, maybe there are
>>> settings
>>> that limit number of connections, because before MasterNotRunning I
>>> get this
>>> in logs:
>>>
>>> 2011-01-22 12:06:43,411 INFO org.apache.zookeeper.ZooKeeper: Initiating
>>> client connection, connectString=hd-master:2181 sessionTimeout=60000
>>> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@3b5e234c
>>> 2011-01-22 12:06:43,483 INFO org.apache.zookeeper.ClientCnxn: Opening
>>> socket connection to server hd-master/10.6.75.212:2181
>>> 2011-01-22 12:06:43,484 INFO org.apache.zookeeper.ClientCnxn: Socket
>>> connection established to hd-master/10.6.75.212:2181, initiating session
>>> 2011-01-22 12:06:43,488 INFO org.apache.zookeeper.ClientCnxn: Unable to
>>> read additional data from server sessionid 0x0, likely server has closed
>>> socket, closing socket connection and attempting reconnect
>>> 2011-01-22 12:06:43,605 INFO
>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
>>> getMaster
>>> attempt 0 of 10 failed; retrying after sleep of 1000
>>> java.io.IOException:
>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
>>>
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
>>>
>>> at
>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
>>>
>>>
>>> at
>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>> at
>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>
>>> at
>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>
>>> at
>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>
>>> at
>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>
>>> at
>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>
>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>
>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
>>>
>>> ... 16 more
>>>
>>>
>>> On 22.01.2011 01:36, Stack wrote:
>>>
>>>> Then its odd that PE fails. Can you figure difference between the two
>>>> environments? Perhaps your MR jobs are fat jars that include the conf
>>>> and all dependencies whereas PE is dump and expects the dependencies
>>>> and conf on CLASSPATH?
>>>>
>>>> St.Ack
>>>>
>>>> On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
>>>> <wl...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I have other mapreduce tasks that are running on this cluster and
>>>>> using
>>>>> HBase and they are working correctly. All my servers have the same
>>>>> configuration.
>>>>>
>>>>> --
>>>>> Wojciech Langiewicz
>>>>>
>>>>> On 21.01.2011 19:20, Stack wrote:
>>>>>
>>>>>>
>>>>>> When clients are> 1, then PE tries to run a mapreduce job to host the
>>>>>> loading clients.
>>>>>>
>>>>>> Is it possible that the client out in MR task is trying to connect to
>>>>>> wrong location? Perhaps the HBase conf dir is not available to the
>>>>>> running task? Have you seen
>>>>>>
>>>>>>
>>>>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
>>>>>>
>>>>>> ?
>>>>>> Perhaps this will help?
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>>>>>> <wl...@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hello
>>>>>>> I have problem with running HBase performance tests from
>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using
>>>>>>> version
>>>>>>> from CDH3,
>>>>>>> Tests are ok when argument nclients is 1, but in case of greater
>>>>>>> number,
>>>>>>> after mappers reaching 100% I get this exception (I didn't test all
>>>>>>> tests,
>>>>>>> but all of tested by me failed, 'scan' and 'randomRead' fail for
>>>>>>> sure):
>>>>>>>
>>>>>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>>>>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>>>>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>>>
>>>>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>>>
>>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>>>>
>>>>>>> Do you have any ideas how to solve this?
>>>>>>>
>>>>>>> --
>>>>>>> Wojciech Langiewicz
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>
>


Re: HBase performance evaluation master not running problem.

Posted by Wojciech Langiewicz <wl...@gmail.com>.
Hi,
sessionTimeout=60000,
I didn't change it, so it's default.
What value do you recommend?

--
Wojciech Langiewicz

On 22.01.2011 16:49, Ted Yu wrote:
> What's the value of 'zookeeper.session.timeout' ?
>
> Maybe you can tune it higher.
>
> On Sat, Jan 22, 2011 at 3:13 AM, Wojciech Langiewicz
> <wl...@gmail.com>wrote:
>
>> Hi,
>> I have re-run test with 2 or more mappers and looked into logs more
>> closely: The mapreduce job has finished correctly, some map attempts are
>> killed, but eventually all of mappers finish. And there's no rule which
>> mappers on which servers fail (re-run tests multiple times).
>>
>> So I suspect that this is not classpath problem, maybe there are settings
>> that limit number of connections, because before MasterNotRunning I get this
>> in logs:
>>
>> 2011-01-22 12:06:43,411 INFO org.apache.zookeeper.ZooKeeper: Initiating
>> client connection, connectString=hd-master:2181 sessionTimeout=60000
>> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@3b5e234c
>> 2011-01-22 12:06:43,483 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server hd-master/10.6.75.212:2181
>> 2011-01-22 12:06:43,484 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection established to hd-master/10.6.75.212:2181, initiating session
>> 2011-01-22 12:06:43,488 INFO org.apache.zookeeper.ClientCnxn: Unable to
>> read additional data from server sessionid 0x0, likely server has closed
>> socket, closing socket connection and attempting reconnect
>> 2011-01-22 12:06:43,605 INFO
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster
>> attempt 0 of 10 failed; retrying after sleep of 1000
>> java.io.IOException:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>         at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
>>         at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
>>
>>         at
>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>         at org.apache.hadoop.mapred.Child.main(Child.java:211)
>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase/master
>>         at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>         at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>>         at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
>>         ... 16 more
>>
>>
>> On 22.01.2011 01:36, Stack wrote:
>>
>>> Then its odd that PE fails.  Can you figure difference between the two
>>> environments?  Perhaps your MR jobs are fat jars that include the conf
>>> and all dependencies whereas PE is dump and expects the dependencies
>>> and conf on CLASSPATH?
>>>
>>> St.Ack
>>>
>>> On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
>>> <wl...@gmail.com>   wrote:
>>>
>>>> Hi,
>>>> I have other mapreduce tasks that are running on this cluster and using
>>>> HBase and they are working correctly. All my servers have the same
>>>> configuration.
>>>>
>>>> --
>>>> Wojciech Langiewicz
>>>>
>>>> On 21.01.2011 19:20, Stack wrote:
>>>>
>>>>>
>>>>> When clients are>     1, then PE tries to run a mapreduce job to host the
>>>>> loading clients.
>>>>>
>>>>> Is it possible that the client out in MR task is trying to connect to
>>>>> wrong location?  Perhaps the HBase conf dir is not available to the
>>>>> running task?  Have you seen
>>>>>
>>>>>
>>>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
>>>>> ?
>>>>>   Perhaps this will help?
>>>>>
>>>>> St.Ack
>>>>>
>>>>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>>>>> <wl...@gmail.com>     wrote:
>>>>>
>>>>>>
>>>>>> Hello
>>>>>> I have problem with running HBase performance tests from
>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using
>>>>>> version
>>>>>> from CDH3,
>>>>>> Tests are ok when argument nclients is 1, but in case of greater
>>>>>> number,
>>>>>> after mappers reaching 100% I get this exception (I didn't test all
>>>>>> tests,
>>>>>> but all of tested by me failed, 'scan' and 'randomRead' fail for sure):
>>>>>>
>>>>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>>>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>>>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>>>>         at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>>>>         at
>>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>>>         at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>>         at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>>         at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>>         at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>>         at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>>>         at
>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>         at
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>>         at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>>>
>>>>>> Do you have any ideas how to solve this?
>>>>>>
>>>>>> --
>>>>>> Wojciech Langiewicz
>>>>>>
>>>>>>
>>>>
>>>>
>>
>


Re: HBase performance evaluation master not running problem.

Posted by Ted Yu <yu...@gmail.com>.
What's the value of 'zookeeper.session.timeout' ?

Maybe you can tune it higher.

On Sat, Jan 22, 2011 at 3:13 AM, Wojciech Langiewicz
<wl...@gmail.com>wrote:

> Hi,
> I have re-run test with 2 or more mappers and looked into logs more
> closely: The mapreduce job has finished correctly, some map attempts are
> killed, but eventually all of mappers finish. And there's no rule which
> mappers on which servers fail (re-run tests multiple times).
>
> So I suspect that this is not classpath problem, maybe there are settings
> that limit number of connections, because before MasterNotRunning I get this
> in logs:
>
> 2011-01-22 12:06:43,411 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=hd-master:2181 sessionTimeout=60000
> watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@3b5e234c
> 2011-01-22 12:06:43,483 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server hd-master/10.6.75.212:2181
> 2011-01-22 12:06:43,484 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to hd-master/10.6.75.212:2181, initiating session
> 2011-01-22 12:06:43,488 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x0, likely server has closed
> socket, closing socket connection and attempting reconnect
> 2011-01-22 12:06:43,605 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: getMaster
> attempt 0 of 10 failed; retrying after sleep of 1000
> java.io.IOException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
>
>        at
> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>        at org.apache.hadoop.mapred.Child.main(Child.java:211)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
>        at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
>        ... 16 more
>
>
> On 22.01.2011 01:36, Stack wrote:
>
>> Then its odd that PE fails.  Can you figure difference between the two
>> environments?  Perhaps your MR jobs are fat jars that include the conf
>> and all dependencies whereas PE is dump and expects the dependencies
>> and conf on CLASSPATH?
>>
>> St.Ack
>>
>> On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
>> <wl...@gmail.com>  wrote:
>>
>>> Hi,
>>> I have other mapreduce tasks that are running on this cluster and using
>>> HBase and they are working correctly. All my servers have the same
>>> configuration.
>>>
>>> --
>>> Wojciech Langiewicz
>>>
>>> On 21.01.2011 19:20, Stack wrote:
>>>
>>>>
>>>> When clients are>    1, then PE tries to run a mapreduce job to host the
>>>> loading clients.
>>>>
>>>> Is it possible that the client out in MR task is trying to connect to
>>>> wrong location?  Perhaps the HBase conf dir is not available to the
>>>> running task?  Have you seen
>>>>
>>>>
>>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description
>>>> ?
>>>>  Perhaps this will help?
>>>>
>>>> St.Ack
>>>>
>>>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>>>> <wl...@gmail.com>    wrote:
>>>>
>>>>>
>>>>> Hello
>>>>> I have problem with running HBase performance tests from
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using
>>>>> version
>>>>> from CDH3,
>>>>> Tests are ok when argument nclients is 1, but in case of greater
>>>>> number,
>>>>> after mappers reaching 100% I get this exception (I didn't test all
>>>>> tests,
>>>>> but all of tested by me failed, 'scan' and 'randomRead' fail for sure):
>>>>>
>>>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>>>        at
>>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>>        at
>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>>        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>>>        at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>>        at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>>
>>>>> Do you have any ideas how to solve this?
>>>>>
>>>>> --
>>>>> Wojciech Langiewicz
>>>>>
>>>>>
>>>
>>>
>

Re: HBase performance evaluation master not running problem.

Posted by Wojciech Langiewicz <wl...@gmail.com>.
Hi,
I have re-run test with 2 or more mappers and looked into logs more 
closely: The mapreduce job has finished correctly, some map attempts are 
killed, but eventually all of mappers finish. And there's no rule which 
mappers on which servers fail (re-run tests multiple times).

So I suspect that this is not classpath problem, maybe there are 
settings that limit number of connections, because before 
MasterNotRunning I get this in logs:

2011-01-22 12:06:43,411 INFO org.apache.zookeeper.ZooKeeper: Initiating 
client connection, connectString=hd-master:2181 sessionTimeout=60000 
watcher=org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper@3b5e234c
2011-01-22 12:06:43,483 INFO org.apache.zookeeper.ClientCnxn: Opening 
socket connection to server hd-master/10.6.75.212:2181
2011-01-22 12:06:43,484 INFO org.apache.zookeeper.ClientCnxn: Socket 
connection established to hd-master/10.6.75.212:2181, initiating session
2011-01-22 12:06:43,488 INFO org.apache.zookeeper.ClientCnxn: Unable to 
read additional data from server sessionid 0x0, likely server has closed 
socket, closing socket connection and attempting reconnect
2011-01-22 12:06:43,605 INFO 
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: 
getMaster attempt 0 of 10 failed; retrying after sleep of 1000
java.io.IOException: 
org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /hbase/master
	at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:481)
	at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:377)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:381)
	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
	at 
org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
	at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /hbase/master
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
	at 
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:477)
	... 16 more

On 22.01.2011 01:36, Stack wrote:
> Then its odd that PE fails.  Can you figure difference between the two
> environments?  Perhaps your MR jobs are fat jars that include the conf
> and all dependencies whereas PE is dump and expects the dependencies
> and conf on CLASSPATH?
>
> St.Ack
>
> On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
> <wl...@gmail.com>  wrote:
>> Hi,
>> I have other mapreduce tasks that are running on this cluster and using
>> HBase and they are working correctly. All my servers have the same
>> configuration.
>>
>> --
>> Wojciech Langiewicz
>>
>> On 21.01.2011 19:20, Stack wrote:
>>>
>>> When clients are>    1, then PE tries to run a mapreduce job to host the
>>> loading clients.
>>>
>>> Is it possible that the client out in MR task is trying to connect to
>>> wrong location?  Perhaps the HBase conf dir is not available to the
>>> running task?  Have you seen
>>>
>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description?
>>>   Perhaps this will help?
>>>
>>> St.Ack
>>>
>>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>>> <wl...@gmail.com>    wrote:
>>>>
>>>> Hello
>>>> I have problem with running HBase performance tests from
>>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using version
>>>> from CDH3,
>>>> Tests are ok when argument nclients is 1, but in case of greater number,
>>>> after mappers reaching 100% I get this exception (I didn't test all
>>>> tests,
>>>> but all of tested by me failed, 'scan' and 'randomRead' fail for sure):
>>>>
>>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>>         at
>>>>
>>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>>         at
>>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>>         at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>>         at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>>         at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>>         at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>>         at
>>>>
>>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>         at
>>>>
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>>         at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>>
>>>> Do you have any ideas how to solve this?
>>>>
>>>> --
>>>> Wojciech Langiewicz
>>>>
>>
>>


Re: HBase performance evaluation master not running problem.

Posted by Stack <st...@duboce.net>.
Then its odd that PE fails.  Can you figure difference between the two
environments?  Perhaps your MR jobs are fat jars that include the conf
and all dependencies whereas PE is dump and expects the dependencies
and conf on CLASSPATH?

St.Ack

On Fri, Jan 21, 2011 at 11:56 AM, Wojciech Langiewicz
<wl...@gmail.com> wrote:
> Hi,
> I have other mapreduce tasks that are running on this cluster and using
> HBase and they are working correctly. All my servers have the same
> configuration.
>
> --
> Wojciech Langiewicz
>
> On 21.01.2011 19:20, Stack wrote:
>>
>> When clients are>  1, then PE tries to run a mapreduce job to host the
>> loading clients.
>>
>> Is it possible that the client out in MR task is trying to connect to
>> wrong location?  Perhaps the HBase conf dir is not available to the
>> running task?  Have you seen
>>
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description?
>>  Perhaps this will help?
>>
>> St.Ack
>>
>> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
>> <wl...@gmail.com>  wrote:
>>>
>>> Hello
>>> I have problem with running HBase performance tests from
>>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using version
>>> from CDH3,
>>> Tests are ok when argument nclients is 1, but in case of greater number,
>>> after mappers reaching 100% I get this exception (I didn't test all
>>> tests,
>>> but all of tested by me failed, 'scan' and 'randomRead' fail for sure):
>>>
>>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>>> org.apache.hadoop.hbase.MasterNotRunningException
>>>        at
>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>>        at
>>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>>        at
>>>
>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>>        at
>>>
>>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>>        at
>>>
>>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>>        at
>>>
>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>>        at
>>>
>>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>>        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at javax.security.auth.Subject.doAs(Subject.java:396)
>>>        at
>>>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>>        at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>>
>>> Do you have any ideas how to solve this?
>>>
>>> --
>>> Wojciech Langiewicz
>>>
>
>

Re: HBase performance evaluation master not running problem.

Posted by Wojciech Langiewicz <wl...@gmail.com>.
Hi,
I have other mapreduce tasks that are running on this cluster and using 
HBase and they are working correctly. All my servers have the same 
configuration.

--
Wojciech Langiewicz

On 21.01.2011 19:20, Stack wrote:
> When clients are>  1, then PE tries to run a mapreduce job to host the
> loading clients.
>
> Is it possible that the client out in MR task is trying to connect to
> wrong location?  Perhaps the HBase conf dir is not available to the
> running task?  Have you seen
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description?
>   Perhaps this will help?
>
> St.Ack
>
> On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
> <wl...@gmail.com>  wrote:
>> Hello
>> I have problem with running HBase performance tests from
>> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using version
>> from CDH3,
>> Tests are ok when argument nclients is 1, but in case of greater number,
>> after mappers reaching 100% I get this exception (I didn't test all tests,
>> but all of tested by me failed, 'scan' and 'randomRead' fail for sure):
>>
>> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
>> attempt_201101211442_0005_m_000000_0, Status : FAILED
>> org.apache.hadoop.hbase.MasterNotRunningException
>>         at
>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>>         at
>> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>>         at
>> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>>         at org.apache.hadoop.mapred.Child.main(Child.java:211)
>>
>> Do you have any ideas how to solve this?
>>
>> --
>> Wojciech Langiewicz
>>


Re: HBase performance evaluation master not running problem.

Posted by Stack <st...@duboce.net>.
When clients are > 1, then PE tries to run a mapreduce job to host the
loading clients.

Is it possible that the client out in MR task is trying to connect to
wrong location?  Perhaps the HBase conf dir is not available to the
running task?  Have you seen
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description?
 Perhaps this will help?

St.Ack

On Fri, Jan 21, 2011 at 8:12 AM, Wojciech Langiewicz
<wl...@gmail.com> wrote:
> Hello
> I have problem with running HBase performance tests from
> org.apache.hadoop.hbase.PerformanceEvaluation package. I'm using version
> from CDH3,
> Tests are ok when argument nclients is 1, but in case of greater number,
> after mappers reaching 100% I get this exception (I didn't test all tests,
> but all of tested by me failed, 'scan' and 'randomRead' fail for sure):
>
> 11/01/21 17:05:19 INFO mapred.JobClient: Task Id :
> attempt_201101211442_0005_m_000000_0, Status : FAILED
> org.apache.hadoop.hbase.MasterNotRunningException
>        at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:416)
>        at
> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:78)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.testSetup(PerformanceEvaluation.java:745)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:764)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1097)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:446)
>        at
> org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:399)
>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
>        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>        at org.apache.hadoop.mapred.Child.main(Child.java:211)
>
> Do you have any ideas how to solve this?
>
> --
> Wojciech Langiewicz
>