You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by lulynn_2008 <lu...@163.com> on 2012/09/12 09:39:48 UTC
[pig-0.10.0 failed to load hbase table]Re:Re: Re: Re: pig-0.10.0:
TestJobSubmission failed with: Internal error creating job configuration.
Hi Cheolsoo,
Current TestJobSubmission and TestHBaseStorage passed. But if I run following scripts with hbase-0.94/zookeeper-3.4.3 cluster:
the same issue happened.
2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /127.0.0.1:2181
2012-09-12 00:30:07,199 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 22394@bdvm071.svl.ibm.com
2012-09-12 00:30:07,212 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-09-12 00:30:07,213 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-09-12 00:30:07,221 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
Reproduce steps:
Create hbase table:
./hbase shell:
create 'employees', 'SN', 'department', 'address'
put 'employees', 'Hong', 'address:country', 'China'
Run following pig commands:
./pig
A = load 'hbase://employees' using org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'address:country', '-loadKey true') as (SN:bytearray,country:bytearray);
B = filter A by SN == 'Hong';
dump B;
At 2012-08-22 05:51:38,"Cheolsoo Park" <ch...@cloudera.com> wrote:
>OK, I got TestJobSubmission passing. Please apply the following diff to
>your Pig:
>
>
>diff --git test/org/apache/pig/test/TestJobSubmission.java
>test/org/apache/pig/test/TestJobSubmission.java
>index 89ac132..211c4ae 100644
>--- test/org/apache/pig/test/TestJobSubmission.java
>+++ test/org/apache/pig/test/TestJobSubmission.java
>@@ -26,8 +26,10 @@ import java.util.Random;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseTestingUtility;
>*+import org.apache.hadoop.hbase.HConstants;*
> import org.apache.hadoop.hbase.MiniHBaseCluster;
> import org.apache.hadoop.hbase.util.Bytes;
>*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
> import org.apache.hadoop.mapred.jobcontrol.Job;
> import org.apache.hadoop.mapred.jobcontrol.JobControl;
> import org.apache.pig.ExecType;
>@@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
>*- util.startMiniZKCluster();*
>*+ int clientPort = util.startMiniZKCluster().getClientPort();*
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
>@@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
>"100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
>*+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
>Integer.toString(clientPort));*
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
>
>
>There are several changes to MiniZooKeeperCluster in HBase 0.94 compared to
>0.90. In particular, the ZK client port is no longer 21818 by default. In
>fact, it is randomized in 0.94. So the client port has to be explicitly set
>in PigContext at runtime.
>
>Now I am able to run both TestJobSubmission and TestHBaseStorage. Please
>let me know if this works for you. I am going to file a jira to capture
>what I found, so we can make use of it when bumping the Hbase version in
>the future.
>
>Thanks,
>Cheolsoo
>
>
>On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <ch...@cloudera.com>wrote:
>
>> Hi,
>>
>> TestJobSubmission doesn't pass for me either. But it doesn't throw the
>> error that you're seeing. Instead, it hangs at line 583
>> in TestJobSubmission.java. I am not sure yet what's happening though.
>>
>> Thanks,
>> Cheolsoo
>>
>> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <lu...@163.com> wrote:
>>
>>> Hi Cheolsoo,
>>> Please help to check whether TestJobSunbmission passed in your
>>> environment. Thanks
>>>
>>>
>>> Hi Cheolsoo,
>>> TestHBaseStorage passed now with your fix. Thanks.
>>> But TestJobSubmission still failed with the same previous error with
>>> TestHBaseStorage. Please help to check. Thanks
>>>
>>>
>>>
>>>
>>>
>>> At 2012-08-18 14:26:32,"Cheolsoo Park" <ch...@cloudera.com> wrote:
>>> >Hi,
>>> >
>>> >I tried the same version of libraries as you're using and was able to get
>>> >TestHBaseStorage passing in trunk. Here is what I did:
>>> >
>>> >1) Updated the version of hadoop, hbase, and zookeeper in
>>> >ivy/libraries.properties:
>>> >
>>> >hadoop-core.version=1.0.3
>>> >> hadoop-test.version=1.0.3
>>> >> hbase.version=0.94.0
>>> >> zookeeper.version=3.4.3
>>> >
>>> >
>>> >2) Updated TestHBaseStorage.java since it doesn't compile with
>>> hbase-0.94.0:
>>> >
>>> >HTable table = new HTable(tableName); => HTable table = new HTable(conf,
>>> >> tableName);
>>> >
>>> >
>>> >There are 6 places to fix.
>>> >
>>> >3) Added the following lines to ivy.xml:
>>> >
>>> ><dependency org="com.github.stephenc.high-scale-lib"
>>> name="high-scale-lib"
>>> >> rev="${high-scale-lib.version}" conf="test->default"/>
>>> >> <dependency org="com.google.protobuf" name="protobuf-java"
>>> >> rev="${protobuf-java.version}" conf="test->default"/>
>>> >
>>> >
>>> >I also added the following line to ivy/libraries.properties:
>>> >
>>> >high-scale-lib.version=1.1.1
>>> >
>>> >
>>> >In particular, I was able to reproduce your error without this line in
>>> >ivy.xml:
>>> >
>>> ><dependency org="com.google.protobuf" name="protobuf-java"
>>> >> rev="${protobuf-java.version}" conf="test->default"/>
>>> >
>>> >
>>> >With all these changes, TestHBaseStorage passes for me. Please let me
>>> know
>>> >if this works for you.
>>> >
>>> >Thanks,
>>> >Cheolsoo
>>> >
>>> >On Thu, Aug 16, 2012 at 11:02 PM, lulynn_2008 <lu...@163.com>
>>> wrote:
>>> >
>>> >> BTW, I am using hadoop-1.0.3, hbase-0.94.0 and zookeeper-3.4.3.
>>> >>
>>> >>
>>> >> Hi Cheolsoo,
>>> >> You are right. TestHBaseStorage also failed with the same error.
>>> >>
>>> >> I got following region server information, and seems hbase region has
>>> been
>>> >> started successfully. But hbase can not connect to zookeeper
>>> >>
>>> >> 12/08/13 06:34:12 INFO server.PrepRequestProcessor: Got user-level
>>> >> KeeperException when processing sessionid:0x1391cf90aed0000 type:delete
>>> >> cxid:0x24 zxid:0x10 txntype:-1 reqpath:n/a Error
>>> >> Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for
>>> >> /hbase/root-region-server
>>> >>
>>> >> 12/08/13 06:34:16 INFO regionserver.HRegion: creating HRegion
>>> pigtable_1
>>> >> HTD == {NAME => 'pigtable_1', FAMILIES => [{NAME => 'pig',
>>> >> REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS => 'false', COMPRESSION =>
>>> >> 'NONE', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true', MIN_VERSIONS =>
>>> >> '0', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'false', BLOOMFILTER
>>> =>
>>> >> 'NONE', TTL => '2147483647', VERSIONS => '3', BLOCKSIZE => '65536'}]}
>>> >> RootDir = hdfs://shihc015:54670/user/root/hbase Table name ==
>>> pigtable_1
>>> >>
>>> >> 12/08/13 06:34:43 INFO zookeeper.ClientCnxn: Opening socket connection
>>> to
>>> >> server shihc015/127.0.0.1:21818
>>> >> 12/08/13 06:34:43 WARN client.ZooKeeperSaslClient: SecurityException:
>>> >> java.lang.SecurityException: Unable to locate a login configuration
>>> >> occurred when trying to find JAAS configuration.
>>> >> 12/08/13 06:34:43 INFO client.ZooKeeperSaslClient: Client will not
>>> >> SASL-authenticate because the default JAAS configuration section
>>> 'Client'
>>> >> could not be found. If you are not using SASL, you may ignore this. On
>>> the
>>> >> other hand, if you expected SASL to work, please fix your JAAS
>>> >> configuration.
>>> >> 12/08/13 06:34:43 WARN zookeeper.ClientCnxn: Session 0x0 for server
>>> null,
>>> >> unexpected error, closing socket connection and attempting reconnect
>>> >> java.net.ConnectException: Connection refused
>>> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> >> at
>>> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
>>> >> at
>>> >>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
>>> >> at
>>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>>> >> 12/08/13 06:34:43 WARN zookeeper.RecoverableZooKeeper: Possibly
>>> transient
>>> >> ZooKeeper exception:
>>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
>>> >>
>>> >>
>>> >> 12/08/16 22:33:05 ERROR zookeeper.ZooKeeperWatcher: hconnection
>>> Received
>>> >> unexpected KeeperException, re-throwing exception
>>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
>>> >> at
>>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>> >> at
>>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>> >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>>> >> at
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>>> >> at
>>> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>>> >> at
>>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>>> >> at
>>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
>>> >> at
>>> >>
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:551)
>>> >> at
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385)
>>> >> at
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
>>> >> at
>>> >>
>>> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:492)
>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> >> at
>>> >>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
>>> >> at
>>> >>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
>>> >> at java.lang.reflect.Method.invoke(Method.java:611)
>>> >> at
>>> >>
>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>>> >> at
>>> >>
>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>> >> at
>>> >>
>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>>> >> at
>>> >>
>>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>>> >> at
>>> >>
>>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>>> >> at
>>> >>
>>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>>> >> at
>>> >>
>>> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
>>> >> at
>>> >>
>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> At 2012-08-16 01:30:33,"Cheolsoo Park" <ch...@cloudera.com> wrote:
>>> >> >Hi,
>>> >> >
>>> >> >Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException:
>>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries.
>>> >> >
>>> >> >
>>> >> >It looks like the region server is not available.
>>> >> >1) Does TestHBaseStorage pass? I guess that it fails with the same
>>> error.
>>> >> >2) In your log, do you see any errors related to the HBase region
>>> server?
>>> >> >For example, java.lang.IllegalStateException: Could not instantiate a
>>> >> >region instance.
>>> >> >
>>> >> >If the region server failed to start, you will see this failure.
>>> >> >
>>> >> >Thanks,
>>> >> >Cheolsoo
>>> >> >
>>> >> >
>>> >> >On Wed, Aug 15, 2012 at 12:56 AM, lulynn_2008 <lu...@163.com>
>>> >> wrote:
>>> >> >
>>> >> >> Hi All,
>>> >> >>
>>> >> >> In TestJobSubmission, testReducerNumEstimation failed with following
>>> >> error
>>> >> >> information. Please give a glance to check what the problem is.
>>> Thanks.
>>> >> >>
>>> >> >> Error information:
>>> >> >> Testcase: testReducerNumEstimation took 1,809.419 sec
>>> >> >> Caused an ERROR
>>> >> >> Internal error creating job configuration.
>>> >> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>>> >> >> ERROR 2017: Internal error creating job configuration.
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:491)
>>> >> >> Caused by:
>>> org.apache.hadoop.hbase.client.NoServerForRegionException:
>>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries.
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:926)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>>> >> >> at
>>> >> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>>> >> >> at
>>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>>> >> >> at
>>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:545)
>>> >> >> at
>>> >> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:384)
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>>
>>
Re: Re: [pig-0.10.0 failed to load hbase table]Re:Re: Re: Re:
pig-0.10.0: TestJobSubmission failed with: Internal error creating job configuration.
Posted by Cheolsoo Park <ch...@cloudera.com>.
It's a known issue with Pig 0.10.0 as far as I know, so I usually recommend
the workaround that I suggested to you.
I haven't verified this by myself, but I believe that the recent change to
HBaseStorage (PIG-2821 <https://issues.apache.org/jira/browse/PIG-2821>)
fixes this problem (PIG-2822<https://issues.apache.org/jira/browse/PIG-2822>)
- i.e. hbase-site.xml not being packed into job.xml.
Regarding Pig 0.9.1, I haven't used that version, so I can't answer your
question.
Thanks,
Cheolsoo
On Thu, Sep 13, 2012 at 1:10 AM, lulynn_2008 <lu...@163.com> wrote:
> Hi Cheolsoo,
>
> Yes, you are right. I am running the ZK quorum on remote machines and you
> way works.
> -- But I have put hbase-site.xml(which include hbase.zookeeper.quorum) in
> pig classpath. But seems pig did not store these hbase configurations to
> job. Is this the design of pig-0.10.0?
-- Seems pig just puts hadoop configurations(hadoop-site.xml,
> core-site.xml...) into job.xml during create job. Is this correct?
> -- I remembered that pig-0.9.1 store hbase configurations into job.xml.
> Did I miss anything?
>
> Thanks.
>
>
>
>
>
>
> At 2012-09-13 02:24:18,"Cheolsoo Park" <ch...@cloudera.com> wrote:
> >Hi,
> >
> >2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening
> >> socket connection to server /127.0.0.1:2181
> >
> >
> >This message seems wrong. I assume that you're running the ZK quorum on
> >remote machines, but it is trying to connect to localhost. Can you try to
> >set "hbase.zookeeper.quorum" in "pig.properties" as follows:
> >"hbase.zookeeper.quorum=<your
> >ZK quorum host:port>" ?
> >
> >Thanks,
> >Cheolsoo
> >
> >On Wed, Sep 12, 2012 at 12:39 AM, lulynn_2008 <lu...@163.com>
> wrote:
> >
> >> Hi Cheolsoo,
> >> Current TestJobSubmission and TestHBaseStorage passed. But if I run
> >> following scripts with hbase-0.94/zookeeper-3.4.3 cluster:
> >>
> >> the same issue happened.
> >>
> >> 2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening
> >> socket connection to server /127.0.0.1:2181
> >> 2012-09-12 00:30:07,199 INFO
> >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier
> of
> >> this process is 22394@bdvm071.svl.ibm.com
> >> 2012-09-12 00:30:07,212 WARN
> >> org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException:
> >> java.lang.SecurityException: Unable to locate a login configuration
> >> occurred when trying to find JAAS configuration.
> >> 2012-09-12 00:30:07,213 INFO
> >> org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not
> >> SASL-authenticate because the default JAAS configuration section
> 'Client'
> >> could not be found. If you are not using SASL, you may ignore this. On
> the
> >> other hand, if you expected SASL to work, please fix your JAAS
> >> configuration.
> >> 2012-09-12 00:30:07,221 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x0
> >> for server null, unexpected error, closing socket connection and
> attempting
> >> reconnect
> >> java.net.ConnectException: Connection refused
> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >> at
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
> >> at
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >> at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >>
> >> Reproduce steps:
> >>
> >> Create hbase table:
> >> ./hbase shell:
> >> create 'employees', 'SN', 'department', 'address'
> >> put 'employees', 'Hong', 'address:country', 'China'
> >>
> >> Run following pig commands:
> >> ./pig
> >> A = load 'hbase://employees' using
> >> org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'address:country',
> >> '-loadKey true') as (SN:bytearray,country:bytearray);
> >> B = filter A by SN == 'Hong';
> >> dump B;
> >>
> >>
> >>
> >>
> >>
> >> At 2012-08-22 05:51:38,"Cheolsoo Park" <ch...@cloudera.com> wrote:
> >> >OK, I got TestJobSubmission passing. Please apply the following diff to
> >> >your Pig:
> >> >
> >> >
> >> >diff --git test/org/apache/pig/test/TestJobSubmission.java
> >> >test/org/apache/pig/test/TestJobSubmission.java
> >> >index 89ac132..211c4ae 100644
> >> >--- test/org/apache/pig/test/TestJobSubmission.java
> >> >+++ test/org/apache/pig/test/TestJobSubmission.java
> >> >@@ -26,8 +26,10 @@ import java.util.Random;
> >> >
> >> > import org.apache.hadoop.conf.Configuration;
> >> > import org.apache.hadoop.hbase.HBaseTestingUtility;
> >> >*+import org.apache.hadoop.hbase.HConstants;*
> >> > import org.apache.hadoop.hbase.MiniHBaseCluster;
> >> > import org.apache.hadoop.hbase.util.Bytes;
> >> >*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
> >> > import org.apache.hadoop.mapred.jobcontrol.Job;
> >> > import org.apache.hadoop.mapred.jobcontrol.JobControl;
> >> > import org.apache.pig.ExecType;
> >> >@@ -541,7 +543,7 @@ public class TestJobSubmission {
> >> > // use the estimation
> >> > Configuration conf = cluster.getConfiguration();
> >> > HBaseTestingUtility util = new HBaseTestingUtility(conf);
> >> >*- util.startMiniZKCluster();*
> >> >*+ int clientPort = util.startMiniZKCluster().getClientPort();*
> >> > util.startMiniHBaseCluster(1, 1);
> >> >
> >> > String query = "a = load '/passwd';" +
> >> >@@ -553,6 +555,7 @@ public class TestJobSubmission {
> >> >
> >> >
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
> >> >"100");
> >> > pc.getConf().setProperty("pig.exec.reducers.max", "10");
> >> >*+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
> >> >Integer.toString(clientPort));*
> >> >
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> >> > conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> >> > JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> >> >
> >> >
> >> >There are several changes to MiniZooKeeperCluster in HBase 0.94
> compared
> >> to
> >> >0.90. In particular, the ZK client port is no longer 21818 by default.
> In
> >> >fact, it is randomized in 0.94. So the client port has to be explicitly
> >> set
> >> >in PigContext at runtime.
> >> >
> >> >Now I am able to run both TestJobSubmission and TestHBaseStorage.
> Please
> >> >let me know if this works for you. I am going to file a jira to capture
> >> >what I found, so we can make use of it when bumping the Hbase version
> in
> >> >the future.
> >> >
> >> >Thanks,
> >> >Cheolsoo
> >> >
> >> >
> >> >On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <cheolsoo@cloudera.com
> >> >wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> TestJobSubmission doesn't pass for me either. But it doesn't throw
> the
> >> >> error that you're seeing. Instead, it hangs at line 583
> >> >> in TestJobSubmission.java. I am not sure yet what's happening though.
> >> >>
> >> >> Thanks,
> >> >> Cheolsoo
> >> >>
> >> >> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <lu...@163.com>
> >> wrote:
> >> >>
> >> >>> Hi Cheolsoo,
> >> >>> Please help to check whether TestJobSunbmission passed in your
> >> >>> environment. Thanks
> >> >>>
> >> >>>
> >> >>> Hi Cheolsoo,
> >> >>> TestHBaseStorage passed now with your fix. Thanks.
> >> >>> But TestJobSubmission still failed with the same previous error with
> >> >>> TestHBaseStorage. Please help to check. Thanks
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> At 2012-08-18 14:26:32,"Cheolsoo Park" <ch...@cloudera.com>
> wrote:
> >> >>> >Hi,
> >> >>> >
> >> >>> >I tried the same version of libraries as you're using and was able
> to
> >> get
> >> >>> >TestHBaseStorage passing in trunk. Here is what I did:
> >> >>> >
> >> >>> >1) Updated the version of hadoop, hbase, and zookeeper in
> >> >>> >ivy/libraries.properties:
> >> >>> >
> >> >>> >hadoop-core.version=1.0.3
> >> >>> >> hadoop-test.version=1.0.3
> >> >>> >> hbase.version=0.94.0
> >> >>> >> zookeeper.version=3.4.3
> >> >>> >
> >> >>> >
> >> >>> >2) Updated TestHBaseStorage.java since it doesn't compile with
> >> >>> hbase-0.94.0:
> >> >>> >
> >> >>> >HTable table = new HTable(tableName); => HTable table = new
> >> HTable(conf,
> >> >>> >> tableName);
> >> >>> >
> >> >>> >
> >> >>> >There are 6 places to fix.
> >> >>> >
> >> >>> >3) Added the following lines to ivy.xml:
> >> >>> >
> >> >>> ><dependency org="com.github.stephenc.high-scale-lib"
> >> >>> name="high-scale-lib"
> >> >>> >> rev="${high-scale-lib.version}" conf="test->default"/>
> >> >>> >> <dependency org="com.google.protobuf" name="protobuf-java"
> >> >>> >> rev="${protobuf-java.version}" conf="test->default"/>
> >> >>> >
> >> >>> >
> >> >>> >I also added the following line to ivy/libraries.properties:
> >> >>> >
> >> >>> >high-scale-lib.version=1.1.1
> >> >>> >
> >> >>> >
> >> >>> >In particular, I was able to reproduce your error without this
> line in
> >> >>> >ivy.xml:
> >> >>> >
> >> >>> ><dependency org="com.google.protobuf" name="protobuf-java"
> >> >>> >> rev="${protobuf-java.version}" conf="test->default"/>
> >> >>> >
> >> >>> >
> >> >>> >With all these changes, TestHBaseStorage passes for me. Please let
> me
> >> >>> know
> >> >>> >if this works for you.
> >> >>> >
> >> >>> >Thanks,
> >> >>> >Cheolsoo
> >> >>> >
> >> >>> >On Thu, Aug 16, 2012 at 11:02 PM, lulynn_2008 <lulynn_2008@163.com
> >
> >> >>> wrote:
> >> >>> >
> >> >>> >> BTW, I am using hadoop-1.0.3, hbase-0.94.0 and zookeeper-3.4.3.
> >> >>> >>
> >> >>> >>
> >> >>> >> Hi Cheolsoo,
> >> >>> >> You are right. TestHBaseStorage also failed with the same error.
> >> >>> >>
> >> >>> >> I got following region server information, and seems hbase region
> >> has
> >> >>> been
> >> >>> >> started successfully. But hbase can not connect to zookeeper
> >> >>> >>
> >> >>> >> 12/08/13 06:34:12 INFO server.PrepRequestProcessor: Got
> user-level
> >> >>> >> KeeperException when processing sessionid:0x1391cf90aed0000
> >> type:delete
> >> >>> >> cxid:0x24 zxid:0x10 txntype:-1 reqpath:n/a Error
> >> >>> >> Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for
> >> >>> >> /hbase/root-region-server
> >> >>> >>
> >> >>> >> 12/08/13 06:34:16 INFO regionserver.HRegion: creating HRegion
> >> >>> pigtable_1
> >> >>> >> HTD == {NAME => 'pigtable_1', FAMILIES => [{NAME => 'pig',
> >> >>> >> REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS => 'false',
> >> COMPRESSION =>
> >> >>> >> 'NONE', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true',
> >> MIN_VERSIONS =>
> >> >>> >> '0', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'false',
> >> BLOOMFILTER
> >> >>> =>
> >> >>> >> 'NONE', TTL => '2147483647', VERSIONS => '3', BLOCKSIZE =>
> >> '65536'}]}
> >> >>> >> RootDir = hdfs://shihc015:54670/user/root/hbase Table name ==
> >> >>> pigtable_1
> >> >>> >>
> >> >>> >> 12/08/13 06:34:43 INFO zookeeper.ClientCnxn: Opening socket
> >> connection
> >> >>> to
> >> >>> >> server shihc015/127.0.0.1:21818
> >> >>> >> 12/08/13 06:34:43 WARN client.ZooKeeperSaslClient:
> >> SecurityException:
> >> >>> >> java.lang.SecurityException: Unable to locate a login
> configuration
> >> >>> >> occurred when trying to find JAAS configuration.
> >> >>> >> 12/08/13 06:34:43 INFO client.ZooKeeperSaslClient: Client will
> not
> >> >>> >> SASL-authenticate because the default JAAS configuration section
> >> >>> 'Client'
> >> >>> >> could not be found. If you are not using SASL, you may ignore
> this.
> >> On
> >> >>> the
> >> >>> >> other hand, if you expected SASL to work, please fix your JAAS
> >> >>> >> configuration.
> >> >>> >> 12/08/13 06:34:43 WARN zookeeper.ClientCnxn: Session 0x0 for
> server
> >> >>> null,
> >> >>> >> unexpected error, closing socket connection and attempting
> reconnect
> >> >>> >> java.net.ConnectException: Connection refused
> >> >>> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
> Method)
> >> >>> >> at
> >> >>> >>
> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >> >>> >> at
> >> >>> >>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >> >>> >> 12/08/13 06:34:43 WARN zookeeper.RecoverableZooKeeper: Possibly
> >> >>> transient
> >> >>> >> ZooKeeper exception:
> >> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> >> >>> >>
> >> >>> >>
> >> >>> >> 12/08/16 22:33:05 ERROR zookeeper.ZooKeeperWatcher: hconnection
> >> >>> Received
> >> >>> >> unexpected KeeperException, re-throwing exception
> >> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> >> >>> >> at
> >> >>> >>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >> >>> >> at
> >> >>> >>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >> >>> >> at
> >> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >> >>> >> at
> >> >>> >>
> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> >> >>> >> at
> >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> >> >>> >> at
> >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:551)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:492)
> >> >>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> Method)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
> >> >>> >> at java.lang.reflect.Method.invoke(Method.java:611)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> >> >>> >> at
> >> >>> >>
> >> >>>
> >>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> At 2012-08-16 01:30:33,"Cheolsoo Park" <ch...@cloudera.com>
> >> wrote:
> >> >>> >> >Hi,
> >> >>> >> >
> >> >>> >> >Caused by:
> >> org.apache.hadoop.hbase.client.NoServerForRegionException:
> >> >>> >> >> Unable to find region for passwd,,99999999999999 after 10
> tries.
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >It looks like the region server is not available.
> >> >>> >> >1) Does TestHBaseStorage pass? I guess that it fails with the
> same
> >> >>> error.
> >> >>> >> >2) In your log, do you see any errors related to the HBase
> region
> >> >>> server?
> >> >>> >> >For example, java.lang.IllegalStateException: Could not
> >> instantiate a
> >> >>> >> >region instance.
> >> >>> >> >
> >> >>> >> >If the region server failed to start, you will see this failure.
> >> >>> >> >
> >> >>> >> >Thanks,
> >> >>> >> >Cheolsoo
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >On Wed, Aug 15, 2012 at 12:56 AM, lulynn_2008 <
> lulynn_2008@163.com
> >> >
> >> >>> >> wrote:
> >> >>> >> >
> >> >>> >> >> Hi All,
> >> >>> >> >>
> >> >>> >> >> In TestJobSubmission, testReducerNumEstimation failed with
> >> following
> >> >>> >> error
> >> >>> >> >> information. Please give a glance to check what the problem
> is.
> >> >>> Thanks.
> >> >>> >> >>
> >> >>> >> >> Error information:
> >> >>> >> >> Testcase: testReducerNumEstimation took 1,809.419 sec
> >> >>> >> >> Caused an ERROR
> >> >>> >> >> Internal error creating job configuration.
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
> >> >>> >> >> ERROR 2017: Internal error creating job configuration.
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:491)
> >> >>> >> >> Caused by:
> >> >>> org.apache.hadoop.hbase.client.NoServerForRegionException:
> >> >>> >> >> Unable to find region for passwd,,99999999999999 after 10
> tries.
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:926)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >> >>> >> >> at
> >> >>> >> >>
> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> >> >>> >> >> at
> >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> >> >>> >> >> at
> >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:545)
> >> >>> >> >> at
> >> >>> >> >>
> >> >>> >>
> >> >>>
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:384)
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >>
>
Re:Re: [pig-0.10.0 failed to load hbase table]Re:Re: Re: Re:
pig-0.10.0: TestJobSubmission failed with: Internal error creating job
configuration.
Posted by lulynn_2008 <lu...@163.com>.
Hi Cheolsoo,
Yes, you are right. I am running the ZK quorum on remote machines and you way works.
-- But I have put hbase-site.xml(which include hbase.zookeeper.quorum) in pig classpath. But seems pig did not store these hbase configurations to job. Is this the design of pig-0.10.0?
-- Seems pig just puts hadoop configurations(hadoop-site.xml, core-site.xml...) into job.xml during create job. Is this correct?
-- I remembered that pig-0.9.1 store hbase configurations into job.xml. Did I miss anything?
Thanks.
At 2012-09-13 02:24:18,"Cheolsoo Park" <ch...@cloudera.com> wrote:
>Hi,
>
>2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server /127.0.0.1:2181
>
>
>This message seems wrong. I assume that you're running the ZK quorum on
>remote machines, but it is trying to connect to localhost. Can you try to
>set "hbase.zookeeper.quorum" in "pig.properties" as follows:
>"hbase.zookeeper.quorum=<your
>ZK quorum host:port>" ?
>
>Thanks,
>Cheolsoo
>
>On Wed, Sep 12, 2012 at 12:39 AM, lulynn_2008 <lu...@163.com> wrote:
>
>> Hi Cheolsoo,
>> Current TestJobSubmission and TestHBaseStorage passed. But if I run
>> following scripts with hbase-0.94/zookeeper-3.4.3 cluster:
>>
>> the same issue happened.
>>
>> 2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server /127.0.0.1:2181
>> 2012-09-12 00:30:07,199 INFO
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of
>> this process is 22394@bdvm071.svl.ibm.com
>> 2012-09-12 00:30:07,212 WARN
>> org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException:
>> java.lang.SecurityException: Unable to locate a login configuration
>> occurred when trying to find JAAS configuration.
>> 2012-09-12 00:30:07,213 INFO
>> org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not
>> SASL-authenticate because the default JAAS configuration section 'Client'
>> could not be found. If you are not using SASL, you may ignore this. On the
>> other hand, if you expected SASL to work, please fix your JAAS
>> configuration.
>> 2012-09-12 00:30:07,221 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
>> for server null, unexpected error, closing socket connection and attempting
>> reconnect
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
>> at
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>>
>> Reproduce steps:
>>
>> Create hbase table:
>> ./hbase shell:
>> create 'employees', 'SN', 'department', 'address'
>> put 'employees', 'Hong', 'address:country', 'China'
>>
>> Run following pig commands:
>> ./pig
>> A = load 'hbase://employees' using
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'address:country',
>> '-loadKey true') as (SN:bytearray,country:bytearray);
>> B = filter A by SN == 'Hong';
>> dump B;
>>
>>
>>
>>
>>
>> At 2012-08-22 05:51:38,"Cheolsoo Park" <ch...@cloudera.com> wrote:
>> >OK, I got TestJobSubmission passing. Please apply the following diff to
>> >your Pig:
>> >
>> >
>> >diff --git test/org/apache/pig/test/TestJobSubmission.java
>> >test/org/apache/pig/test/TestJobSubmission.java
>> >index 89ac132..211c4ae 100644
>> >--- test/org/apache/pig/test/TestJobSubmission.java
>> >+++ test/org/apache/pig/test/TestJobSubmission.java
>> >@@ -26,8 +26,10 @@ import java.util.Random;
>> >
>> > import org.apache.hadoop.conf.Configuration;
>> > import org.apache.hadoop.hbase.HBaseTestingUtility;
>> >*+import org.apache.hadoop.hbase.HConstants;*
>> > import org.apache.hadoop.hbase.MiniHBaseCluster;
>> > import org.apache.hadoop.hbase.util.Bytes;
>> >*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
>> > import org.apache.hadoop.mapred.jobcontrol.Job;
>> > import org.apache.hadoop.mapred.jobcontrol.JobControl;
>> > import org.apache.pig.ExecType;
>> >@@ -541,7 +543,7 @@ public class TestJobSubmission {
>> > // use the estimation
>> > Configuration conf = cluster.getConfiguration();
>> > HBaseTestingUtility util = new HBaseTestingUtility(conf);
>> >*- util.startMiniZKCluster();*
>> >*+ int clientPort = util.startMiniZKCluster().getClientPort();*
>> > util.startMiniHBaseCluster(1, 1);
>> >
>> > String query = "a = load '/passwd';" +
>> >@@ -553,6 +555,7 @@ public class TestJobSubmission {
>> >
>> > pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
>> >"100");
>> > pc.getConf().setProperty("pig.exec.reducers.max", "10");
>> >*+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
>> >Integer.toString(clientPort));*
>> > ConfigurationValidator.validatePigProperties(pc.getProperties());
>> > conf = ConfigurationUtil.toConfiguration(pc.getProperties());
>> > JobControlCompiler jcc = new JobControlCompiler(pc, conf);
>> >
>> >
>> >There are several changes to MiniZooKeeperCluster in HBase 0.94 compared
>> to
>> >0.90. In particular, the ZK client port is no longer 21818 by default. In
>> >fact, it is randomized in 0.94. So the client port has to be explicitly
>> set
>> >in PigContext at runtime.
>> >
>> >Now I am able to run both TestJobSubmission and TestHBaseStorage. Please
>> >let me know if this works for you. I am going to file a jira to capture
>> >what I found, so we can make use of it when bumping the Hbase version in
>> >the future.
>> >
>> >Thanks,
>> >Cheolsoo
>> >
>> >
>> >On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <cheolsoo@cloudera.com
>> >wrote:
>> >
>> >> Hi,
>> >>
>> >> TestJobSubmission doesn't pass for me either. But it doesn't throw the
>> >> error that you're seeing. Instead, it hangs at line 583
>> >> in TestJobSubmission.java. I am not sure yet what's happening though.
>> >>
>> >> Thanks,
>> >> Cheolsoo
>> >>
>> >> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <lu...@163.com>
>> wrote:
>> >>
>> >>> Hi Cheolsoo,
>> >>> Please help to check whether TestJobSunbmission passed in your
>> >>> environment. Thanks
>> >>>
>> >>>
>> >>> Hi Cheolsoo,
>> >>> TestHBaseStorage passed now with your fix. Thanks.
>> >>> But TestJobSubmission still failed with the same previous error with
>> >>> TestHBaseStorage. Please help to check. Thanks
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> At 2012-08-18 14:26:32,"Cheolsoo Park" <ch...@cloudera.com> wrote:
>> >>> >Hi,
>> >>> >
>> >>> >I tried the same version of libraries as you're using and was able to
>> get
>> >>> >TestHBaseStorage passing in trunk. Here is what I did:
>> >>> >
>> >>> >1) Updated the version of hadoop, hbase, and zookeeper in
>> >>> >ivy/libraries.properties:
>> >>> >
>> >>> >hadoop-core.version=1.0.3
>> >>> >> hadoop-test.version=1.0.3
>> >>> >> hbase.version=0.94.0
>> >>> >> zookeeper.version=3.4.3
>> >>> >
>> >>> >
>> >>> >2) Updated TestHBaseStorage.java since it doesn't compile with
>> >>> hbase-0.94.0:
>> >>> >
>> >>> >HTable table = new HTable(tableName); => HTable table = new
>> HTable(conf,
>> >>> >> tableName);
>> >>> >
>> >>> >
>> >>> >There are 6 places to fix.
>> >>> >
>> >>> >3) Added the following lines to ivy.xml:
>> >>> >
>> >>> ><dependency org="com.github.stephenc.high-scale-lib"
>> >>> name="high-scale-lib"
>> >>> >> rev="${high-scale-lib.version}" conf="test->default"/>
>> >>> >> <dependency org="com.google.protobuf" name="protobuf-java"
>> >>> >> rev="${protobuf-java.version}" conf="test->default"/>
>> >>> >
>> >>> >
>> >>> >I also added the following line to ivy/libraries.properties:
>> >>> >
>> >>> >high-scale-lib.version=1.1.1
>> >>> >
>> >>> >
>> >>> >In particular, I was able to reproduce your error without this line in
>> >>> >ivy.xml:
>> >>> >
>> >>> ><dependency org="com.google.protobuf" name="protobuf-java"
>> >>> >> rev="${protobuf-java.version}" conf="test->default"/>
>> >>> >
>> >>> >
>> >>> >With all these changes, TestHBaseStorage passes for me. Please let me
>> >>> know
>> >>> >if this works for you.
>> >>> >
>> >>> >Thanks,
>> >>> >Cheolsoo
>> >>> >
>> >>> >On Thu, Aug 16, 2012 at 11:02 PM, lulynn_2008 <lu...@163.com>
>> >>> wrote:
>> >>> >
>> >>> >> BTW, I am using hadoop-1.0.3, hbase-0.94.0 and zookeeper-3.4.3.
>> >>> >>
>> >>> >>
>> >>> >> Hi Cheolsoo,
>> >>> >> You are right. TestHBaseStorage also failed with the same error.
>> >>> >>
>> >>> >> I got following region server information, and seems hbase region
>> has
>> >>> been
>> >>> >> started successfully. But hbase can not connect to zookeeper
>> >>> >>
>> >>> >> 12/08/13 06:34:12 INFO server.PrepRequestProcessor: Got user-level
>> >>> >> KeeperException when processing sessionid:0x1391cf90aed0000
>> type:delete
>> >>> >> cxid:0x24 zxid:0x10 txntype:-1 reqpath:n/a Error
>> >>> >> Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for
>> >>> >> /hbase/root-region-server
>> >>> >>
>> >>> >> 12/08/13 06:34:16 INFO regionserver.HRegion: creating HRegion
>> >>> pigtable_1
>> >>> >> HTD == {NAME => 'pigtable_1', FAMILIES => [{NAME => 'pig',
>> >>> >> REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS => 'false',
>> COMPRESSION =>
>> >>> >> 'NONE', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true',
>> MIN_VERSIONS =>
>> >>> >> '0', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'false',
>> BLOOMFILTER
>> >>> =>
>> >>> >> 'NONE', TTL => '2147483647', VERSIONS => '3', BLOCKSIZE =>
>> '65536'}]}
>> >>> >> RootDir = hdfs://shihc015:54670/user/root/hbase Table name ==
>> >>> pigtable_1
>> >>> >>
>> >>> >> 12/08/13 06:34:43 INFO zookeeper.ClientCnxn: Opening socket
>> connection
>> >>> to
>> >>> >> server shihc015/127.0.0.1:21818
>> >>> >> 12/08/13 06:34:43 WARN client.ZooKeeperSaslClient:
>> SecurityException:
>> >>> >> java.lang.SecurityException: Unable to locate a login configuration
>> >>> >> occurred when trying to find JAAS configuration.
>> >>> >> 12/08/13 06:34:43 INFO client.ZooKeeperSaslClient: Client will not
>> >>> >> SASL-authenticate because the default JAAS configuration section
>> >>> 'Client'
>> >>> >> could not be found. If you are not using SASL, you may ignore this.
>> On
>> >>> the
>> >>> >> other hand, if you expected SASL to work, please fix your JAAS
>> >>> >> configuration.
>> >>> >> 12/08/13 06:34:43 WARN zookeeper.ClientCnxn: Session 0x0 for server
>> >>> null,
>> >>> >> unexpected error, closing socket connection and attempting reconnect
>> >>> >> java.net.ConnectException: Connection refused
>> >>> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>> >> at
>> >>> >>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
>> >>> >> at
>> >>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>> >>> >> 12/08/13 06:34:43 WARN zookeeper.RecoverableZooKeeper: Possibly
>> >>> transient
>> >>> >> ZooKeeper exception:
>> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
>> >>> >>
>> >>> >>
>> >>> >> 12/08/16 22:33:05 ERROR zookeeper.ZooKeeperWatcher: hconnection
>> >>> Received
>> >>> >> unexpected KeeperException, re-throwing exception
>> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
>> >>> >> at
>> >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> >>> >> at
>> >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> >>> >> at
>> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>> >>> >> at
>> >>> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>> >>> >> at
>> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>> >>> >> at
>> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:551)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
>> >>> >> at
>> >>> >>
>> >>>
>> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:492)
>> >>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >>> >> at
>> >>> >>
>> >>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
>> >>> >> at
>> >>> >>
>> >>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
>> >>> >> at java.lang.reflect.Method.invoke(Method.java:611)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
>> >>> >> at
>> >>> >>
>> >>>
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> At 2012-08-16 01:30:33,"Cheolsoo Park" <ch...@cloudera.com>
>> wrote:
>> >>> >> >Hi,
>> >>> >> >
>> >>> >> >Caused by:
>> org.apache.hadoop.hbase.client.NoServerForRegionException:
>> >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries.
>> >>> >> >
>> >>> >> >
>> >>> >> >It looks like the region server is not available.
>> >>> >> >1) Does TestHBaseStorage pass? I guess that it fails with the same
>> >>> error.
>> >>> >> >2) In your log, do you see any errors related to the HBase region
>> >>> server?
>> >>> >> >For example, java.lang.IllegalStateException: Could not
>> instantiate a
>> >>> >> >region instance.
>> >>> >> >
>> >>> >> >If the region server failed to start, you will see this failure.
>> >>> >> >
>> >>> >> >Thanks,
>> >>> >> >Cheolsoo
>> >>> >> >
>> >>> >> >
>> >>> >> >On Wed, Aug 15, 2012 at 12:56 AM, lulynn_2008 <lulynn_2008@163.com
>> >
>> >>> >> wrote:
>> >>> >> >
>> >>> >> >> Hi All,
>> >>> >> >>
>> >>> >> >> In TestJobSubmission, testReducerNumEstimation failed with
>> following
>> >>> >> error
>> >>> >> >> information. Please give a glance to check what the problem is.
>> >>> Thanks.
>> >>> >> >>
>> >>> >> >> Error information:
>> >>> >> >> Testcase: testReducerNumEstimation took 1,809.419 sec
>> >>> >> >> Caused an ERROR
>> >>> >> >> Internal error creating job configuration.
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
>> >>> >> >> ERROR 2017: Internal error creating job configuration.
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:491)
>> >>> >> >> Caused by:
>> >>> org.apache.hadoop.hbase.client.NoServerForRegionException:
>> >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries.
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:926)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
>> >>> >> >> at
>> >>> >> >>
>> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>> >>> >> >> at
>> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>> >>> >> >> at
>> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:545)
>> >>> >> >> at
>> >>> >> >>
>> >>> >>
>> >>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:384)
>> >>> >> >>
>> >>> >> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>
>>
Re: [pig-0.10.0 failed to load hbase table]Re:Re: Re: Re: pig-0.10.0:
TestJobSubmission failed with: Internal error creating job configuration.
Posted by Cheolsoo Park <ch...@cloudera.com>.
Hi,
2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server /127.0.0.1:2181
This message seems wrong. I assume that you're running the ZK quorum on
remote machines, but it is trying to connect to localhost. Can you try to
set "hbase.zookeeper.quorum" in "pig.properties" as follows:
"hbase.zookeeper.quorum=<your
ZK quorum host:port>" ?
Thanks,
Cheolsoo
On Wed, Sep 12, 2012 at 12:39 AM, lulynn_2008 <lu...@163.com> wrote:
> Hi Cheolsoo,
> Current TestJobSubmission and TestHBaseStorage passed. But if I run
> following scripts with hbase-0.94/zookeeper-3.4.3 cluster:
>
> the same issue happened.
>
> 2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server /127.0.0.1:2181
> 2012-09-12 00:30:07,199 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of
> this process is 22394@bdvm071.svl.ibm.com
> 2012-09-12 00:30:07,212 WARN
> org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> 2012-09-12 00:30:07,213 INFO
> org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section 'Client'
> could not be found. If you are not using SASL, you may ignore this. On the
> other hand, if you expected SASL to work, please fix your JAAS
> configuration.
> 2012-09-12 00:30:07,221 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
> for server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
>
> Reproduce steps:
>
> Create hbase table:
> ./hbase shell:
> create 'employees', 'SN', 'department', 'address'
> put 'employees', 'Hong', 'address:country', 'China'
>
> Run following pig commands:
> ./pig
> A = load 'hbase://employees' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'address:country',
> '-loadKey true') as (SN:bytearray,country:bytearray);
> B = filter A by SN == 'Hong';
> dump B;
>
>
>
>
>
> At 2012-08-22 05:51:38,"Cheolsoo Park" <ch...@cloudera.com> wrote:
> >OK, I got TestJobSubmission passing. Please apply the following diff to
> >your Pig:
> >
> >
> >diff --git test/org/apache/pig/test/TestJobSubmission.java
> >test/org/apache/pig/test/TestJobSubmission.java
> >index 89ac132..211c4ae 100644
> >--- test/org/apache/pig/test/TestJobSubmission.java
> >+++ test/org/apache/pig/test/TestJobSubmission.java
> >@@ -26,8 +26,10 @@ import java.util.Random;
> >
> > import org.apache.hadoop.conf.Configuration;
> > import org.apache.hadoop.hbase.HBaseTestingUtility;
> >*+import org.apache.hadoop.hbase.HConstants;*
> > import org.apache.hadoop.hbase.MiniHBaseCluster;
> > import org.apache.hadoop.hbase.util.Bytes;
> >*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;*
> > import org.apache.hadoop.mapred.jobcontrol.Job;
> > import org.apache.hadoop.mapred.jobcontrol.JobControl;
> > import org.apache.pig.ExecType;
> >@@ -541,7 +543,7 @@ public class TestJobSubmission {
> > // use the estimation
> > Configuration conf = cluster.getConfiguration();
> > HBaseTestingUtility util = new HBaseTestingUtility(conf);
> >*- util.startMiniZKCluster();*
> >*+ int clientPort = util.startMiniZKCluster().getClientPort();*
> > util.startMiniHBaseCluster(1, 1);
> >
> > String query = "a = load '/passwd';" +
> >@@ -553,6 +555,7 @@ public class TestJobSubmission {
> >
> > pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer",
> >"100");
> > pc.getConf().setProperty("pig.exec.reducers.max", "10");
> >*+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT,
> >Integer.toString(clientPort));*
> > ConfigurationValidator.validatePigProperties(pc.getProperties());
> > conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> > JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> >
> >
> >There are several changes to MiniZooKeeperCluster in HBase 0.94 compared
> to
> >0.90. In particular, the ZK client port is no longer 21818 by default. In
> >fact, it is randomized in 0.94. So the client port has to be explicitly
> set
> >in PigContext at runtime.
> >
> >Now I am able to run both TestJobSubmission and TestHBaseStorage. Please
> >let me know if this works for you. I am going to file a jira to capture
> >what I found, so we can make use of it when bumping the Hbase version in
> >the future.
> >
> >Thanks,
> >Cheolsoo
> >
> >
> >On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <cheolsoo@cloudera.com
> >wrote:
> >
> >> Hi,
> >>
> >> TestJobSubmission doesn't pass for me either. But it doesn't throw the
> >> error that you're seeing. Instead, it hangs at line 583
> >> in TestJobSubmission.java. I am not sure yet what's happening though.
> >>
> >> Thanks,
> >> Cheolsoo
> >>
> >> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <lu...@163.com>
> wrote:
> >>
> >>> Hi Cheolsoo,
> >>> Please help to check whether TestJobSunbmission passed in your
> >>> environment. Thanks
> >>>
> >>>
> >>> Hi Cheolsoo,
> >>> TestHBaseStorage passed now with your fix. Thanks.
> >>> But TestJobSubmission still failed with the same previous error with
> >>> TestHBaseStorage. Please help to check. Thanks
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> At 2012-08-18 14:26:32,"Cheolsoo Park" <ch...@cloudera.com> wrote:
> >>> >Hi,
> >>> >
> >>> >I tried the same version of libraries as you're using and was able to
> get
> >>> >TestHBaseStorage passing in trunk. Here is what I did:
> >>> >
> >>> >1) Updated the version of hadoop, hbase, and zookeeper in
> >>> >ivy/libraries.properties:
> >>> >
> >>> >hadoop-core.version=1.0.3
> >>> >> hadoop-test.version=1.0.3
> >>> >> hbase.version=0.94.0
> >>> >> zookeeper.version=3.4.3
> >>> >
> >>> >
> >>> >2) Updated TestHBaseStorage.java since it doesn't compile with
> >>> hbase-0.94.0:
> >>> >
> >>> >HTable table = new HTable(tableName); => HTable table = new
> HTable(conf,
> >>> >> tableName);
> >>> >
> >>> >
> >>> >There are 6 places to fix.
> >>> >
> >>> >3) Added the following lines to ivy.xml:
> >>> >
> >>> ><dependency org="com.github.stephenc.high-scale-lib"
> >>> name="high-scale-lib"
> >>> >> rev="${high-scale-lib.version}" conf="test->default"/>
> >>> >> <dependency org="com.google.protobuf" name="protobuf-java"
> >>> >> rev="${protobuf-java.version}" conf="test->default"/>
> >>> >
> >>> >
> >>> >I also added the following line to ivy/libraries.properties:
> >>> >
> >>> >high-scale-lib.version=1.1.1
> >>> >
> >>> >
> >>> >In particular, I was able to reproduce your error without this line in
> >>> >ivy.xml:
> >>> >
> >>> ><dependency org="com.google.protobuf" name="protobuf-java"
> >>> >> rev="${protobuf-java.version}" conf="test->default"/>
> >>> >
> >>> >
> >>> >With all these changes, TestHBaseStorage passes for me. Please let me
> >>> know
> >>> >if this works for you.
> >>> >
> >>> >Thanks,
> >>> >Cheolsoo
> >>> >
> >>> >On Thu, Aug 16, 2012 at 11:02 PM, lulynn_2008 <lu...@163.com>
> >>> wrote:
> >>> >
> >>> >> BTW, I am using hadoop-1.0.3, hbase-0.94.0 and zookeeper-3.4.3.
> >>> >>
> >>> >>
> >>> >> Hi Cheolsoo,
> >>> >> You are right. TestHBaseStorage also failed with the same error.
> >>> >>
> >>> >> I got following region server information, and seems hbase region
> has
> >>> been
> >>> >> started successfully. But hbase can not connect to zookeeper
> >>> >>
> >>> >> 12/08/13 06:34:12 INFO server.PrepRequestProcessor: Got user-level
> >>> >> KeeperException when processing sessionid:0x1391cf90aed0000
> type:delete
> >>> >> cxid:0x24 zxid:0x10 txntype:-1 reqpath:n/a Error
> >>> >> Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for
> >>> >> /hbase/root-region-server
> >>> >>
> >>> >> 12/08/13 06:34:16 INFO regionserver.HRegion: creating HRegion
> >>> pigtable_1
> >>> >> HTD == {NAME => 'pigtable_1', FAMILIES => [{NAME => 'pig',
> >>> >> REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS => 'false',
> COMPRESSION =>
> >>> >> 'NONE', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true',
> MIN_VERSIONS =>
> >>> >> '0', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'false',
> BLOOMFILTER
> >>> =>
> >>> >> 'NONE', TTL => '2147483647', VERSIONS => '3', BLOCKSIZE =>
> '65536'}]}
> >>> >> RootDir = hdfs://shihc015:54670/user/root/hbase Table name ==
> >>> pigtable_1
> >>> >>
> >>> >> 12/08/13 06:34:43 INFO zookeeper.ClientCnxn: Opening socket
> connection
> >>> to
> >>> >> server shihc015/127.0.0.1:21818
> >>> >> 12/08/13 06:34:43 WARN client.ZooKeeperSaslClient:
> SecurityException:
> >>> >> java.lang.SecurityException: Unable to locate a login configuration
> >>> >> occurred when trying to find JAAS configuration.
> >>> >> 12/08/13 06:34:43 INFO client.ZooKeeperSaslClient: Client will not
> >>> >> SASL-authenticate because the default JAAS configuration section
> >>> 'Client'
> >>> >> could not be found. If you are not using SASL, you may ignore this.
> On
> >>> the
> >>> >> other hand, if you expected SASL to work, please fix your JAAS
> >>> >> configuration.
> >>> >> 12/08/13 06:34:43 WARN zookeeper.ClientCnxn: Session 0x0 for server
> >>> null,
> >>> >> unexpected error, closing socket connection and attempting reconnect
> >>> >> java.net.ConnectException: Connection refused
> >>> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>> >> at
> >>> >>
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610)
> >>> >> at
> >>> >>
> >>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> >>> >> at
> >>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)
> >>> >> 12/08/13 06:34:43 WARN zookeeper.RecoverableZooKeeper: Possibly
> >>> transient
> >>> >> ZooKeeper exception:
> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> >>> >>
> >>> >>
> >>> >> 12/08/16 22:33:05 ERROR zookeeper.ZooKeeperWatcher: hconnection
> >>> Received
> >>> >> unexpected KeeperException, re-throwing exception
> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
> >>> >> at
> >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>> >> at
> >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>> >> at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
> >>> >> at
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >>> >> at
> >>> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> >>> >> at
> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> >>> >> at
> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> >>> >> at
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:551)
> >>> >> at
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385)
> >>> >> at
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
> >>> >> at
> >>> >>
> >>>
> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:492)
> >>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >>> >> at
> >>> >>
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
> >>> >> at
> >>> >>
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
> >>> >> at java.lang.reflect.Method.invoke(Method.java:611)
> >>> >> at
> >>> >>
> >>>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> >>> >> at
> >>> >>
> >>>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> >>> >> at
> >>> >>
> >>>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> >>> >> at
> >>> >>
> >>>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> >>> >> at
> >>> >>
> >>>
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> >>> >> at
> >>> >>
> >>>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> >>> >> at
> >>> >>
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> >>> >> at
> >>> >>
> >>>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> At 2012-08-16 01:30:33,"Cheolsoo Park" <ch...@cloudera.com>
> wrote:
> >>> >> >Hi,
> >>> >> >
> >>> >> >Caused by:
> org.apache.hadoop.hbase.client.NoServerForRegionException:
> >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries.
> >>> >> >
> >>> >> >
> >>> >> >It looks like the region server is not available.
> >>> >> >1) Does TestHBaseStorage pass? I guess that it fails with the same
> >>> error.
> >>> >> >2) In your log, do you see any errors related to the HBase region
> >>> server?
> >>> >> >For example, java.lang.IllegalStateException: Could not
> instantiate a
> >>> >> >region instance.
> >>> >> >
> >>> >> >If the region server failed to start, you will see this failure.
> >>> >> >
> >>> >> >Thanks,
> >>> >> >Cheolsoo
> >>> >> >
> >>> >> >
> >>> >> >On Wed, Aug 15, 2012 at 12:56 AM, lulynn_2008 <lulynn_2008@163.com
> >
> >>> >> wrote:
> >>> >> >
> >>> >> >> Hi All,
> >>> >> >>
> >>> >> >> In TestJobSubmission, testReducerNumEstimation failed with
> following
> >>> >> error
> >>> >> >> information. Please give a glance to check what the problem is.
> >>> Thanks.
> >>> >> >>
> >>> >> >> Error information:
> >>> >> >> Testcase: testReducerNumEstimation took 1,809.419 sec
> >>> >> >> Caused an ERROR
> >>> >> >> Internal error creating job configuration.
> >>> >> >>
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException:
> >>> >> >> ERROR 2017: Internal error creating job configuration.
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:491)
> >>> >> >> Caused by:
> >>> org.apache.hadoop.hbase.client.NoServerForRegionException:
> >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries.
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:926)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> >>> >> >> at
> >>> >> >>
> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> >>> >> >> at
> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> >>> >> >> at
> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:545)
> >>> >> >> at
> >>> >> >>
> >>> >>
> >>>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:384)
> >>> >> >>
> >>> >> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>>
> >>>
> >>>
> >>>
> >>
>