You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by George Pang <p0...@gmail.com> on 2009/05/25 20:43:28 UTC
Pig running hadoop mode
Hi users,
When I run pig in hadoop mode, before the grunt shell appears there is
something on the command prompt not quite right:
"Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> "
If I use grunt> ls , it only displays the local files, not on hdfs. Could
anyone help with these?
George
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
This is how I set the environment variables, and how I get the error:
wpang@wpang-desktop:~$ echo *$PIGDIR*
/home/wpang/pigtmp
wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
/home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
wpang@wpang-desktop:~$ echo *$JAVA_HOME*
/usr/lib/jvm/jdk1.6.0_13
wpang@wpang-desktop:~/pig-0.2.0$ cd ~/pig-0.2.0/bin
wpang@wpang-desktop:~/pig-0.2.0/bin$* pig*
2009-05-26 16:52:38,785 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - *Connecting
to hadoop file system at: file:///*
2009-05-26 16:52:38,909 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId=
grunt>
Thanks,
George
2009/5/26 Naber, Chad <CN...@edmunds.com>
> George,
>
> Set your PIG_CLASSPATH to the hadoop/conf directory to run Grunt in
> mapreduce mode.
>
> Chad
>
> -----Original Message-----
> From: George Pang [mailto:p0941p@gmail.com]
> Sent: Tuesday, May 26, 2009 3:12 PM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig running hadoop mode
>
> Hi Alan,
> I can run the examples for pig local mode and pig hadoop mode from pig
> tutorial
> http://hadoop.apache.org/pig/docs/r0.2.0/tutorial.html#Pig+Installation,
> with the results displayed in the hadoop cluster.
>
> My questions are on using Grunt Shell, according to the "quick start" guide
> (http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html)<
> http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html>
>
> 1) Following the instruction from link above, which reads : To run Pig's
> Grunt shell in local mode, follow these instructions.
>
> First, point $PIG_CLASSPATH to the pig.jar file (in your current working
> directory):
>
> $ export PIG_CLASSPATH=./pig.jar
>
> From your current working directory, run:
>
> $ pig -x local
>
> I got this :"pig is not installed. " ie., system can't find it from a
> directory with only pig.jar, the sample code files and a .txt data file.
>
> 2) If I try to run the grunt shell from /pig-0.2.0/bin, the Grunt shell
> will run. However, it only run locally. If I run -x mapreduce, I got
> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> grunt> "
> I set the following environment variables: PIGDIR, HADOOPDIR,
> PIG_CLASSPATH, JAVA_HOME. Are there more environment variables to set?
> Thank you.
>
> George
>
> 2009/5/26 Alan Gates <ga...@yahoo-inc.com>
>
> > This means you're connecting to your local file system instead of your
> > hadoop cluster. Check out
> > http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html, especially
> > the various last section on how to connect to your hadoop cluster.
> >
> > Alan.
> >
> >
> > On May 25, 2009, at 11:43 AM, George Pang wrote:
> >
> > Hi users,
> >>
> >> When I run pig in hadoop mode, before the grunt shell appears there
> >> is something on the command prompt not quite right:
> >> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> >> grunt> "
> >>
> >> If I use grunt> ls , it only displays the local files, not on hdfs.
> >> Could anyone help with these?
> >>
> >> George
> >>
> >
> >
>
>
RE: Pig running hadoop mode
Posted by "Naber, Chad" <CN...@edmunds.com>.
George,
Set your PIG_CLASSPATH to the hadoop/conf directory to run Grunt in mapreduce mode.
Chad
-----Original Message-----
From: George Pang [mailto:p0941p@gmail.com]
Sent: Tuesday, May 26, 2009 3:12 PM
To: pig-user@hadoop.apache.org
Subject: Re: Pig running hadoop mode
Hi Alan,
I can run the examples for pig local mode and pig hadoop mode from pig tutorial http://hadoop.apache.org/pig/docs/r0.2.0/tutorial.html#Pig+Installation,
with the results displayed in the hadoop cluster.
My questions are on using Grunt Shell, according to the "quick start" guide (http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html)<http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html>
1) Following the instruction from link above, which reads : To run Pig's Grunt shell in local mode, follow these instructions.
First, point $PIG_CLASSPATH to the pig.jar file (in your current working
directory):
$ export PIG_CLASSPATH=./pig.jar
>From your current working directory, run:
$ pig -x local
I got this :"pig is not installed. " ie., system can't find it from a directory with only pig.jar, the sample code files and a .txt data file.
2) If I try to run the grunt shell from /pig-0.2.0/bin, the Grunt shell will run. However, it only run locally. If I run -x mapreduce, I got "Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> "
I set the following environment variables: PIGDIR, HADOOPDIR, PIG_CLASSPATH, JAVA_HOME. Are there more environment variables to set?
Thank you.
George
2009/5/26 Alan Gates <ga...@yahoo-inc.com>
> This means you're connecting to your local file system instead of your
> hadoop cluster. Check out
> http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html, especially
> the various last section on how to connect to your hadoop cluster.
>
> Alan.
>
>
> On May 25, 2009, at 11:43 AM, George Pang wrote:
>
> Hi users,
>>
>> When I run pig in hadoop mode, before the grunt shell appears there
>> is something on the command prompt not quite right:
>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> grunt> "
>>
>> If I use grunt> ls , it only displays the local files, not on hdfs.
>> Could anyone help with these?
>>
>> George
>>
>
>
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
Hi Alan,
I can run the examples for pig local mode and pig hadoop mode from pig
tutorial
http://hadoop.apache.org/pig/docs/r0.2.0/tutorial.html#Pig+Installation,
with the results displayed in the hadoop cluster.
My questions are on using Grunt Shell, according to the "quick start" guide
(http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html)<http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html>
1) Following the instruction from link above, which reads : To run Pig’s
Grunt shell in local mode, follow these instructions.
First, point $PIG_CLASSPATH to the pig.jar file (in your current working
directory):
$ export PIG_CLASSPATH=./pig.jar
>From your current working directory, run:
$ pig -x local
I got this :"pig is not installed. " ie., system can't find it from a
directory with only pig.jar, the sample code files and a .txt data file.
2) If I try to run the grunt shell from /pig-0.2.0/bin, the Grunt shell
will run. However, it only run locally. If I run -x mapreduce, I
got "Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> "
I set the following environment variables: PIGDIR, HADOOPDIR, PIG_CLASSPATH,
JAVA_HOME. Are there more environment variables to set?
Thank you.
George
2009/5/26 Alan Gates <ga...@yahoo-inc.com>
> This means you're connecting to your local file system instead of your
> hadoop cluster. Check out
> http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html, especially the
> various last section on how to connect to your hadoop cluster.
>
> Alan.
>
>
> On May 25, 2009, at 11:43 AM, George Pang wrote:
>
> Hi users,
>>
>> When I run pig in hadoop mode, before the grunt shell appears there is
>> something on the command prompt not quite right:
>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> grunt> "
>>
>> If I use grunt> ls , it only displays the local files, not on hdfs. Could
>> anyone help with these?
>>
>> George
>>
>
>
Re: Pig running hadoop mode
Posted by Alan Gates <ga...@yahoo-inc.com>.
This means you're connecting to your local file system instead of your
hadoop cluster. Check out http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html
, especially the various last section on how to connect to your hadoop
cluster.
Alan.
On May 25, 2009, at 11:43 AM, George Pang wrote:
> Hi users,
>
> When I run pig in hadoop mode, before the grunt shell appears there is
> something on the command prompt not quite right:
> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> grunt> "
>
> If I use grunt> ls , it only displays the local files, not on hdfs.
> Could
> anyone help with these?
>
> George
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
Hi,
I added only this, and it works now.
*% export PIG_HADOOP_VERSION=18*
George
2009/5/28 George Pang <p0...@gmail.com>
> Obviously my program has problem going to this line:
>
> 481 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting to map-reduce job tracker at: localhost:54311
>
> I got this line from Cloudera's VM.
>
> George
>
>
> 2009/5/27 George Pang <p0...@gmail.com>
>
>> Correction: the port data seems strange. After I
>> restart Ubuntu, it is like:
>>
>> wpang2@wpang2-desktop:~$ sudo netstat -plten | grep java
>> tcp6 0 0 127.0.0.1:48961 :::*
>> LISTEN 1000 19407 6067/java
>> tcp6 0 0 :::50020 :::*
>> LISTEN 1000 19428 5826/java
>> tcp6 0 0 127.0.0.1:54310 :::*
>> LISTEN 1000 18602 5745/java
>> tcp6 0 0 127.0.0.1:54311 :::*
>> LISTEN 1000 18979 5979/java
>> tcp6 0 0 :::59271 :::*
>> LISTEN 1000 18849 5911/java
>> tcp6 0 0 :::50090 :::*
>> LISTEN 1000 19421 5911/java
>> tcp6 0 0 :::49067 :::*
>> LISTEN 1000 18754 5826/java
>> tcp6 0 0 :::50060 :::*
>> LISTEN 1000 19402 6067/java
>> tcp6 0 0 :::50030 :::*
>> LISTEN 1000 19341 5979/java
>> tcp6 0 0 :::40047 :::*
>> LISTEN 1000 18932 5979/java
>> tcp6 0 0 :::50070 :::*
>> LISTEN 1000 19002 5745/java
>> tcp6 0 0 :::40887 :::*
>> LISTEN 1000 18507 5745/java
>> tcp6 0 0 :::50010 :::*
>> LISTEN 1000 19059 5826/java
>> tcp6 0 0 :::50075 :::*
>> LISTEN 1000 19422 5826/java
>>
>> George
>>
>> 2009/5/27 George Pang <p0...@gmail.com>
>>
>> Hi Chad,
>>>
>>> I think you maybe right. I run fsck, and get this:
>>> fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'
>>>
>>> Since I am a Windows-turned linux user, I don't know much about this
>>> error. In an initial search online, I think it's something wrong with my
>>> disk partition. However I run my Ubuntu on a Virtual Machine. I hope this
>>> won't make things more complex.
>>>
>>> Other than this, my hadoop is now a single-node cluster. Some statuss
>>> are as follows:
>>>
>>> wpang2@wpang2-desktop:~$* jps*
>>> 6039 NameNode
>>> 6103 DataNode
>>> 6180 SecondaryNameNode
>>> 6332 TaskTracker
>>> 6254 JobTracker
>>> 8139 Jps
>>>
>>> wpang2@wpang2-desktop:~$ sudo *netstat -plten | grep java*
>>> [sudo] password for wpang2:
>>> tcp6 0 0 :::50020 :::*
>>> LISTEN 1000 21585 6103/java
>>> tcp6 0 0 127.0.0.1:59621 :::*
>>> LISTEN 1000 21319 6332/java
>>> tcp6 0 0 127.0.0.1:9000 :::*
>>> LISTEN 1000 20964 6039/java
>>> tcp6 0 0 127.0.0.1:9001 :::*
>>> LISTEN 1000 20962 6254/java
>>> tcp6 0 0 :::50090 :::*
>>> LISTEN 1000 21567 6180/java
>>> tcp6 0 0 :::54539 :::*
>>> LISTEN 1000 20937 6103/java
>>> tcp6 0 0 :::50060 :::*
>>> LISTEN 1000 21313 6332/java
>>> tcp6 0 0 :::48237 :::*
>>> LISTEN 1000 20941 6180/java
>>> tcp6 0 0 :::50030 :::*
>>> LISTEN 1000 21315 6254/java
>>> tcp6 0 0 :::46511 :::*
>>> LISTEN 1000 20943 6039/java
>>> tcp6 0 0 :::43189 :::*
>>> LISTEN 1000 20939 6254/java
>>> tcp6 0 0 :::50070 :::*
>>> LISTEN 1000 21318 6039/java
>>> tcp6 0 0 :::50010 :::*
>>> LISTEN 1000 21485 6103/java
>>> tcp6 0 0 :::50075 :::*
>>> LISTEN 1000 21569 6103/java
>>>
>>> Thank you,
>>>
>>> George
>>>
>>>
>>> 2009/5/27 Naber, Chad <CN...@edmunds.com>
>>>
>>>
>>>> George,
>>>>
>>>> I would bet that there are some problems with the HDFS
>>>> 'hdfs://localhost:54310' you are connecting to. Have you verified that the
>>>> HDFS is up and is healthy? Maybe do an FSCK on the blocks to verify that
>>>> there are no corrupt blocks.
>>>>
>>>> Chad
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: George Pang [mailto:p0941p@gmail.com]
>>>> Sent: Wednesday, May 27, 2009 12:40 PM
>>>> To: pig-user@hadoop.apache.org
>>>> Subject: Re: Pig running hadoop mode
>>>>
>>>> Hi Alan,
>>>> I set these 3 environment variables,
>>>>
>>>> wpang@wpang-desktop:~$ echo *$PIGDIR*
>>>> /home/wpang/pigtmp
>>>> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
>>>> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
>>>> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
>>>> /usr/lib/jvm/jdk1.6.0_13
>>>>
>>>> So the hadoop version is 0.18.3.
>>>>
>>>> I used to download the pig.jar from "pig tutorial" wiki page and
>>>> pig-0.2.0 release from the release page. Now I try to build the pig from
>>>> subversion, and it gave me the same thing.
>>>>
>>>> Any idea will be much appreciated.
>>>>
>>>> George
>>>>
>>>>
>>>>
>>>> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>>>>
>>>> > What version of hadoop are you using? Pig, as currently released,
>>>> > runs on 0.18.x hadoop. If you're running a different version you made
>>>> > need to apply a patch to make it run with your version of hadoop.
>>>> >
>>>> > Alan.
>>>> >
>>>> >
>>>> > On May 26, 2009, at 8:39 PM, George Pang wrote:
>>>> >
>>>> > Dear users,
>>>> >>
>>>> >> Now my pig can connect to my hdfs, as on the screeen:
>>>> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>>>> >> 2009-05-26 20:22:29,243 [main] INFO
>>>> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>>> >> Connecting to hadoop file system at: hdfs://localhost:54310
>>>> >> ..........
>>>> >>
>>>> >> But then it failed and generate an error log:
>>>> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>>>> >> java.lang.RuntimeException: Failed to create DataStorage
>>>> >> at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>>>> >> at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>>>> >> at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>>>> >> at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>>>> >> at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>>>> >> at org.apache.pig.PigServer.<init>(PigServer.java:144)
>>>> >> at org.apache.pig.PigServer.<init>(PigServer.java:132)
>>>> >> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>>>> >> at org.apache.pig.Main.main(Main.java:318)
>>>> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>>>> >> response
>>>> >> at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>>> >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> >> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>>>> >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>>>> >> at
>>>> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>>>> >> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>>>> >> at
>>>> >>
>>>> >>
>>>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>>>> >> at
>>>> >>
>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>>>> >> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>>>> >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>>>> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>>>> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>>>> >> at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>>>> >> ... 8 more
>>>> >>
>>>> >> Your advice is much appreciated!
>>>> >> George
>>>> >>
>>>> >> 2009/5/25 George Pang <p0...@gmail.com>
>>>> >>
>>>> >> Hi users,
>>>> >>>
>>>> >>> When I run pig in hadoop mode, before the grunt shell appears there
>>>> >>> is something on the command prompt not quite right:
>>>> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>>> >>> grunt> "
>>>> >>>
>>>> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>>> >>> Could
>>>> >>> anyone help with these?
>>>> >>>
>>>> >>> George
>>>> >>>
>>>> >>>
>>>> >
>>>>
>>>>
>>>
>>
>
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
Obviously my program has problem going to this line:
481 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to map-reduce job tracker at: localhost:54311
I got this line from Cloudera's VM.
George
2009/5/27 George Pang <p0...@gmail.com>
> Correction: the port data seems strange. After I
> restart Ubuntu, it is like:
>
> wpang2@wpang2-desktop:~$ sudo netstat -plten | grep java
> tcp6 0 0 127.0.0.1:48961 :::*
> LISTEN 1000 19407 6067/java
> tcp6 0 0 :::50020 :::*
> LISTEN 1000 19428 5826/java
> tcp6 0 0 127.0.0.1:54310 :::*
> LISTEN 1000 18602 5745/java
> tcp6 0 0 127.0.0.1:54311 :::*
> LISTEN 1000 18979 5979/java
> tcp6 0 0 :::59271 :::*
> LISTEN 1000 18849 5911/java
> tcp6 0 0 :::50090 :::*
> LISTEN 1000 19421 5911/java
> tcp6 0 0 :::49067 :::*
> LISTEN 1000 18754 5826/java
> tcp6 0 0 :::50060 :::*
> LISTEN 1000 19402 6067/java
> tcp6 0 0 :::50030 :::*
> LISTEN 1000 19341 5979/java
> tcp6 0 0 :::40047 :::*
> LISTEN 1000 18932 5979/java
> tcp6 0 0 :::50070 :::*
> LISTEN 1000 19002 5745/java
> tcp6 0 0 :::40887 :::*
> LISTEN 1000 18507 5745/java
> tcp6 0 0 :::50010 :::*
> LISTEN 1000 19059 5826/java
> tcp6 0 0 :::50075 :::*
> LISTEN 1000 19422 5826/java
>
> George
>
> 2009/5/27 George Pang <p0...@gmail.com>
>
> Hi Chad,
>>
>> I think you maybe right. I run fsck, and get this:
>> fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'
>>
>> Since I am a Windows-turned linux user, I don't know much about this
>> error. In an initial search online, I think it's something wrong with my
>> disk partition. However I run my Ubuntu on a Virtual Machine. I hope this
>> won't make things more complex.
>>
>> Other than this, my hadoop is now a single-node cluster. Some statuss are
>> as follows:
>>
>> wpang2@wpang2-desktop:~$* jps*
>> 6039 NameNode
>> 6103 DataNode
>> 6180 SecondaryNameNode
>> 6332 TaskTracker
>> 6254 JobTracker
>> 8139 Jps
>>
>> wpang2@wpang2-desktop:~$ sudo *netstat -plten | grep java*
>> [sudo] password for wpang2:
>> tcp6 0 0 :::50020 :::*
>> LISTEN 1000 21585 6103/java
>> tcp6 0 0 127.0.0.1:59621 :::*
>> LISTEN 1000 21319 6332/java
>> tcp6 0 0 127.0.0.1:9000 :::*
>> LISTEN 1000 20964 6039/java
>> tcp6 0 0 127.0.0.1:9001 :::*
>> LISTEN 1000 20962 6254/java
>> tcp6 0 0 :::50090 :::*
>> LISTEN 1000 21567 6180/java
>> tcp6 0 0 :::54539 :::*
>> LISTEN 1000 20937 6103/java
>> tcp6 0 0 :::50060 :::*
>> LISTEN 1000 21313 6332/java
>> tcp6 0 0 :::48237 :::*
>> LISTEN 1000 20941 6180/java
>> tcp6 0 0 :::50030 :::*
>> LISTEN 1000 21315 6254/java
>> tcp6 0 0 :::46511 :::*
>> LISTEN 1000 20943 6039/java
>> tcp6 0 0 :::43189 :::*
>> LISTEN 1000 20939 6254/java
>> tcp6 0 0 :::50070 :::*
>> LISTEN 1000 21318 6039/java
>> tcp6 0 0 :::50010 :::*
>> LISTEN 1000 21485 6103/java
>> tcp6 0 0 :::50075 :::*
>> LISTEN 1000 21569 6103/java
>>
>> Thank you,
>>
>> George
>>
>>
>> 2009/5/27 Naber, Chad <CN...@edmunds.com>
>>
>>
>>> George,
>>>
>>> I would bet that there are some problems with the HDFS
>>> 'hdfs://localhost:54310' you are connecting to. Have you verified that the
>>> HDFS is up and is healthy? Maybe do an FSCK on the blocks to verify that
>>> there are no corrupt blocks.
>>>
>>> Chad
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: George Pang [mailto:p0941p@gmail.com]
>>> Sent: Wednesday, May 27, 2009 12:40 PM
>>> To: pig-user@hadoop.apache.org
>>> Subject: Re: Pig running hadoop mode
>>>
>>> Hi Alan,
>>> I set these 3 environment variables,
>>>
>>> wpang@wpang-desktop:~$ echo *$PIGDIR*
>>> /home/wpang/pigtmp
>>> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
>>> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
>>> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
>>> /usr/lib/jvm/jdk1.6.0_13
>>>
>>> So the hadoop version is 0.18.3.
>>>
>>> I used to download the pig.jar from "pig tutorial" wiki page and
>>> pig-0.2.0 release from the release page. Now I try to build the pig from
>>> subversion, and it gave me the same thing.
>>>
>>> Any idea will be much appreciated.
>>>
>>> George
>>>
>>>
>>>
>>> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>>>
>>> > What version of hadoop are you using? Pig, as currently released,
>>> > runs on 0.18.x hadoop. If you're running a different version you made
>>> > need to apply a patch to make it run with your version of hadoop.
>>> >
>>> > Alan.
>>> >
>>> >
>>> > On May 26, 2009, at 8:39 PM, George Pang wrote:
>>> >
>>> > Dear users,
>>> >>
>>> >> Now my pig can connect to my hdfs, as on the screeen:
>>> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>>> >> 2009-05-26 20:22:29,243 [main] INFO
>>> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> >> Connecting to hadoop file system at: hdfs://localhost:54310
>>> >> ..........
>>> >>
>>> >> But then it failed and generate an error log:
>>> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>>> >> java.lang.RuntimeException: Failed to create DataStorage
>>> >> at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>>> >> at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>>> >> at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>>> >> at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>>> >> at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>>> >> at org.apache.pig.PigServer.<init>(PigServer.java:144)
>>> >> at org.apache.pig.PigServer.<init>(PigServer.java:132)
>>> >> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>>> >> at org.apache.pig.Main.main(Main.java:318)
>>> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>>> >> response
>>> >> at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>> >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> >> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>>> >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>>> >> at
>>> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>>> >> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>>> >> at
>>> >>
>>> >>
>>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>>> >> at
>>> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>>> >> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>>> >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>>> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>>> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>>> >> at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>>> >> ... 8 more
>>> >>
>>> >> Your advice is much appreciated!
>>> >> George
>>> >>
>>> >> 2009/5/25 George Pang <p0...@gmail.com>
>>> >>
>>> >> Hi users,
>>> >>>
>>> >>> When I run pig in hadoop mode, before the grunt shell appears there
>>> >>> is something on the command prompt not quite right:
>>> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>> >>> grunt> "
>>> >>>
>>> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>> >>> Could
>>> >>> anyone help with these?
>>> >>>
>>> >>> George
>>> >>>
>>> >>>
>>> >
>>>
>>>
>>
>
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
Correction: the port data seems strange. After I
restart Ubuntu, it is like:
wpang2@wpang2-desktop:~$ sudo netstat -plten | grep java
tcp6 0 0 127.0.0.1:48961 :::*
LISTEN 1000 19407 6067/java
tcp6 0 0 :::50020 :::*
LISTEN 1000 19428 5826/java
tcp6 0 0 127.0.0.1:54310 :::*
LISTEN 1000 18602 5745/java
tcp6 0 0 127.0.0.1:54311 :::*
LISTEN 1000 18979 5979/java
tcp6 0 0 :::59271 :::*
LISTEN 1000 18849 5911/java
tcp6 0 0 :::50090 :::*
LISTEN 1000 19421 5911/java
tcp6 0 0 :::49067 :::*
LISTEN 1000 18754 5826/java
tcp6 0 0 :::50060 :::*
LISTEN 1000 19402 6067/java
tcp6 0 0 :::50030 :::*
LISTEN 1000 19341 5979/java
tcp6 0 0 :::40047 :::*
LISTEN 1000 18932 5979/java
tcp6 0 0 :::50070 :::*
LISTEN 1000 19002 5745/java
tcp6 0 0 :::40887 :::*
LISTEN 1000 18507 5745/java
tcp6 0 0 :::50010 :::*
LISTEN 1000 19059 5826/java
tcp6 0 0 :::50075 :::*
LISTEN 1000 19422 5826/java
George
2009/5/27 George Pang <p0...@gmail.com>
> Hi Chad,
>
> I think you maybe right. I run fsck, and get this:
> fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'
>
> Since I am a Windows-turned linux user, I don't know much about this error.
> In an initial search online, I think it's something wrong with my disk
> partition. However I run my Ubuntu on a Virtual Machine. I hope this
> won't make things more complex.
>
> Other than this, my hadoop is now a single-node cluster. Some statuss are
> as follows:
>
> wpang2@wpang2-desktop:~$* jps*
> 6039 NameNode
> 6103 DataNode
> 6180 SecondaryNameNode
> 6332 TaskTracker
> 6254 JobTracker
> 8139 Jps
>
> wpang2@wpang2-desktop:~$ sudo *netstat -plten | grep java*
> [sudo] password for wpang2:
> tcp6 0 0 :::50020 :::*
> LISTEN 1000 21585 6103/java
> tcp6 0 0 127.0.0.1:59621 :::*
> LISTEN 1000 21319 6332/java
> tcp6 0 0 127.0.0.1:9000 :::*
> LISTEN 1000 20964 6039/java
> tcp6 0 0 127.0.0.1:9001 :::*
> LISTEN 1000 20962 6254/java
> tcp6 0 0 :::50090 :::*
> LISTEN 1000 21567 6180/java
> tcp6 0 0 :::54539 :::*
> LISTEN 1000 20937 6103/java
> tcp6 0 0 :::50060 :::*
> LISTEN 1000 21313 6332/java
> tcp6 0 0 :::48237 :::*
> LISTEN 1000 20941 6180/java
> tcp6 0 0 :::50030 :::*
> LISTEN 1000 21315 6254/java
> tcp6 0 0 :::46511 :::*
> LISTEN 1000 20943 6039/java
> tcp6 0 0 :::43189 :::*
> LISTEN 1000 20939 6254/java
> tcp6 0 0 :::50070 :::*
> LISTEN 1000 21318 6039/java
> tcp6 0 0 :::50010 :::*
> LISTEN 1000 21485 6103/java
> tcp6 0 0 :::50075 :::*
> LISTEN 1000 21569 6103/java
>
> Thank you,
>
> George
>
>
> 2009/5/27 Naber, Chad <CN...@edmunds.com>
>
>
>> George,
>>
>> I would bet that there are some problems with the HDFS
>> 'hdfs://localhost:54310' you are connecting to. Have you verified that the
>> HDFS is up and is healthy? Maybe do an FSCK on the blocks to verify that
>> there are no corrupt blocks.
>>
>> Chad
>>
>>
>>
>>
>> -----Original Message-----
>> From: George Pang [mailto:p0941p@gmail.com]
>> Sent: Wednesday, May 27, 2009 12:40 PM
>> To: pig-user@hadoop.apache.org
>> Subject: Re: Pig running hadoop mode
>>
>> Hi Alan,
>> I set these 3 environment variables,
>>
>> wpang@wpang-desktop:~$ echo *$PIGDIR*
>> /home/wpang/pigtmp
>> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
>> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
>> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
>> /usr/lib/jvm/jdk1.6.0_13
>>
>> So the hadoop version is 0.18.3.
>>
>> I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0
>> release from the release page. Now I try to build the pig from subversion,
>> and it gave me the same thing.
>>
>> Any idea will be much appreciated.
>>
>> George
>>
>>
>>
>> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>>
>> > What version of hadoop are you using? Pig, as currently released,
>> > runs on 0.18.x hadoop. If you're running a different version you made
>> > need to apply a patch to make it run with your version of hadoop.
>> >
>> > Alan.
>> >
>> >
>> > On May 26, 2009, at 8:39 PM, George Pang wrote:
>> >
>> > Dear users,
>> >>
>> >> Now my pig can connect to my hdfs, as on the screeen:
>> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>> >> 2009-05-26 20:22:29,243 [main] INFO
>> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> >> Connecting to hadoop file system at: hdfs://localhost:54310
>> >> ..........
>> >>
>> >> But then it failed and generate an error log:
>> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>> >> java.lang.RuntimeException: Failed to create DataStorage
>> >> at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>> >> at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>> >> at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>> >> at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>> >> at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>> >> at org.apache.pig.PigServer.<init>(PigServer.java:144)
>> >> at org.apache.pig.PigServer.<init>(PigServer.java:132)
>> >> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>> >> at org.apache.pig.Main.main(Main.java:318)
>> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>> >> response
>> >> at org.apache.hadoop.ipc.Client.call(Client.java:559)
>> >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>> >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>> >> at
>> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>> >> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>> >> at
>> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>> >> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>> >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>> >> at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>> >> ... 8 more
>> >>
>> >> Your advice is much appreciated!
>> >> George
>> >>
>> >> 2009/5/25 George Pang <p0...@gmail.com>
>> >>
>> >> Hi users,
>> >>>
>> >>> When I run pig in hadoop mode, before the grunt shell appears there
>> >>> is something on the command prompt not quite right:
>> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> >>> grunt> "
>> >>>
>> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
>> >>> Could
>> >>> anyone help with these?
>> >>>
>> >>> George
>> >>>
>> >>>
>> >
>>
>>
>
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
Hi Chad,
I think you maybe right. I run fsck, and get this:
fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'
Since I am a Windows-turned linux user, I don't know much about this error.
In an initial search online, I think it's something wrong with my disk
partition. However I run my Ubuntu on a Virtual Machine. I hope this
won't make things more complex.
Other than this, my hadoop is now a single-node cluster. Some statuss are
as follows:
wpang2@wpang2-desktop:~$* jps*
6039 NameNode
6103 DataNode
6180 SecondaryNameNode
6332 TaskTracker
6254 JobTracker
8139 Jps
wpang2@wpang2-desktop:~$ sudo *netstat -plten | grep java*
[sudo] password for wpang2:
tcp6 0 0 :::50020 :::*
LISTEN 1000 21585 6103/java
tcp6 0 0 127.0.0.1:59621 :::*
LISTEN 1000 21319 6332/java
tcp6 0 0 127.0.0.1:9000 :::*
LISTEN 1000 20964 6039/java
tcp6 0 0 127.0.0.1:9001 :::*
LISTEN 1000 20962 6254/java
tcp6 0 0 :::50090 :::*
LISTEN 1000 21567 6180/java
tcp6 0 0 :::54539 :::*
LISTEN 1000 20937 6103/java
tcp6 0 0 :::50060 :::*
LISTEN 1000 21313 6332/java
tcp6 0 0 :::48237 :::*
LISTEN 1000 20941 6180/java
tcp6 0 0 :::50030 :::*
LISTEN 1000 21315 6254/java
tcp6 0 0 :::46511 :::*
LISTEN 1000 20943 6039/java
tcp6 0 0 :::43189 :::*
LISTEN 1000 20939 6254/java
tcp6 0 0 :::50070 :::*
LISTEN 1000 21318 6039/java
tcp6 0 0 :::50010 :::*
LISTEN 1000 21485 6103/java
tcp6 0 0 :::50075 :::*
LISTEN 1000 21569 6103/java
Thank you,
George
2009/5/27 Naber, Chad <CN...@edmunds.com>
>
> George,
>
> I would bet that there are some problems with the HDFS
> 'hdfs://localhost:54310' you are connecting to. Have you verified that the
> HDFS is up and is healthy? Maybe do an FSCK on the blocks to verify that
> there are no corrupt blocks.
>
> Chad
>
>
>
>
> -----Original Message-----
> From: George Pang [mailto:p0941p@gmail.com]
> Sent: Wednesday, May 27, 2009 12:40 PM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig running hadoop mode
>
> Hi Alan,
> I set these 3 environment variables,
>
> wpang@wpang-desktop:~$ echo *$PIGDIR*
> /home/wpang/pigtmp
> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
> /usr/lib/jvm/jdk1.6.0_13
>
> So the hadoop version is 0.18.3.
>
> I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0
> release from the release page. Now I try to build the pig from subversion,
> and it gave me the same thing.
>
> Any idea will be much appreciated.
>
> George
>
>
>
> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>
> > What version of hadoop are you using? Pig, as currently released,
> > runs on 0.18.x hadoop. If you're running a different version you made
> > need to apply a patch to make it run with your version of hadoop.
> >
> > Alan.
> >
> >
> > On May 26, 2009, at 8:39 PM, George Pang wrote:
> >
> > Dear users,
> >>
> >> Now my pig can connect to my hdfs, as on the screeen:
> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
> >> 2009-05-26 20:22:29,243 [main] INFO
> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> >> Connecting to hadoop file system at: hdfs://localhost:54310
> >> ..........
> >>
> >> But then it failed and generate an error log:
> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
> >> java.lang.RuntimeException: Failed to create DataStorage
> >> at
> >>
> >>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
> >> at
> >>
> >>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
> >> at
> >>
> >>
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
> >> at
> >>
> >>
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
> >> at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
> >> at org.apache.pig.PigServer.<init>(PigServer.java:144)
> >> at org.apache.pig.PigServer.<init>(PigServer.java:132)
> >> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
> >> at org.apache.pig.Main.main(Main.java:318)
> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
> >> response
> >> at org.apache.hadoop.ipc.Client.call(Client.java:559)
> >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
> >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
> >> at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
> >> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
> >> at
> >>
> >>
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
> >> at
> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
> >> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
> >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
> >> at
> >>
> >>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
> >> ... 8 more
> >>
> >> Your advice is much appreciated!
> >> George
> >>
> >> 2009/5/25 George Pang <p0...@gmail.com>
> >>
> >> Hi users,
> >>>
> >>> When I run pig in hadoop mode, before the grunt shell appears there
> >>> is something on the command prompt not quite right:
> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> >>> grunt> "
> >>>
> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
> >>> Could
> >>> anyone help with these?
> >>>
> >>> George
> >>>
> >>>
> >
>
>
RE: Pig running hadoop mode
Posted by "Naber, Chad" <CN...@edmunds.com>.
George,
I would bet that there are some problems with the HDFS 'hdfs://localhost:54310' you are connecting to. Have you verified that the HDFS is up and is healthy? Maybe do an FSCK on the blocks to verify that there are no corrupt blocks.
Chad
-----Original Message-----
From: George Pang [mailto:p0941p@gmail.com]
Sent: Wednesday, May 27, 2009 12:40 PM
To: pig-user@hadoop.apache.org
Subject: Re: Pig running hadoop mode
Hi Alan,
I set these 3 environment variables,
wpang@wpang-desktop:~$ echo *$PIGDIR*
/home/wpang/pigtmp
wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH* /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
wpang@wpang-desktop:~$ echo *$JAVA_HOME*
/usr/lib/jvm/jdk1.6.0_13
So the hadoop version is 0.18.3.
I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0 release from the release page. Now I try to build the pig from subversion, and it gave me the same thing.
Any idea will be much appreciated.
George
2009/5/27 Alan Gates <ga...@yahoo-inc.com>
> What version of hadoop are you using? Pig, as currently released,
> runs on 0.18.x hadoop. If you're running a different version you made
> need to apply a patch to make it run with your version of hadoop.
>
> Alan.
>
>
> On May 26, 2009, at 8:39 PM, George Pang wrote:
>
> Dear users,
>>
>> Now my pig can connect to my hdfs, as on the screeen:
>> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>> 2009-05-26 20:22:29,243 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting to hadoop file system at: hdfs://localhost:54310
>> ..........
>>
>> But then it failed and generate an error log:
>> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>> java.lang.RuntimeException: Failed to create DataStorage
>> at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>> at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>> at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>> at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>> at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>> at org.apache.pig.PigServer.<init>(PigServer.java:144)
>> at org.apache.pig.PigServer.<init>(PigServer.java:132)
>> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>> at org.apache.pig.Main.main(Main.java:318)
>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>> response
>> at org.apache.hadoop.ipc.Client.call(Client.java:559)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>> at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>> at
>>
>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>> at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>> ... 8 more
>>
>> Your advice is much appreciated!
>> George
>>
>> 2009/5/25 George Pang <p0...@gmail.com>
>>
>> Hi users,
>>>
>>> When I run pig in hadoop mode, before the grunt shell appears there
>>> is something on the command prompt not quite right:
>>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>> grunt> "
>>>
>>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>> Could
>>> anyone help with these?
>>>
>>> George
>>>
>>>
>
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
Hi Alan,
I set these 3 environment variables,
wpang@wpang-desktop:~$ echo *$PIGDIR*
/home/wpang/pigtmp
wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
/home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
wpang@wpang-desktop:~$ echo *$JAVA_HOME*
/usr/lib/jvm/jdk1.6.0_13
So the hadoop version is 0.18.3.
I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0
release from the release page. Now I try to build the pig from subversion,
and it gave me the same thing.
Any idea will be much appreciated.
George
2009/5/27 Alan Gates <ga...@yahoo-inc.com>
> What version of hadoop are you using? Pig, as currently released, runs on
> 0.18.x hadoop. If you're running a different version you made need to apply
> a patch to make it run with your version of hadoop.
>
> Alan.
>
>
> On May 26, 2009, at 8:39 PM, George Pang wrote:
>
> Dear users,
>>
>> Now my pig can connect to my hdfs, as on the screeen:
>> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>> 2009-05-26 20:22:29,243 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting
>> to hadoop file system at: hdfs://localhost:54310
>> ..........
>>
>> But then it failed and generate an error log:
>> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>> java.lang.RuntimeException: Failed to create DataStorage
>> at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>> at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>> at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>> at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>> at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>> at org.apache.pig.PigServer.<init>(PigServer.java:144)
>> at org.apache.pig.PigServer.<init>(PigServer.java:132)
>> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>> at org.apache.pig.Main.main(Main.java:318)
>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>> response
>> at org.apache.hadoop.ipc.Client.call(Client.java:559)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>> at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>> at
>>
>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>> at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>> ... 8 more
>>
>> Your advice is much appreciated!
>> George
>>
>> 2009/5/25 George Pang <p0...@gmail.com>
>>
>> Hi users,
>>>
>>> When I run pig in hadoop mode, before the grunt shell appears there is
>>> something on the command prompt not quite right:
>>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>> grunt> "
>>>
>>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>> Could
>>> anyone help with these?
>>>
>>> George
>>>
>>>
>
Re: Pig running hadoop mode
Posted by Alan Gates <ga...@yahoo-inc.com>.
What version of hadoop are you using? Pig, as currently released,
runs on 0.18.x hadoop. If you're running a different version you made
need to apply a patch to make it run with your version of hadoop.
Alan.
On May 26, 2009, at 8:39 PM, George Pang wrote:
> Dear users,
>
> Now my pig can connect to my hdfs, as on the screeen:
> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
> 2009-05-26 20:22:29,243 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting
> to hadoop file system at: hdfs://localhost:54310
> ..........
>
> But then it failed and generate an error log:
> ERROR 2999: Unexpected internal error. Failed to create DataStorage
> java.lang.RuntimeException: Failed to create DataStorage
> at
> org
> .apache
> .pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:
> 75)
> at
> org
> .apache
> .pig
> .backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
> at
> org
> .apache
> .pig
> .backend
> .hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:
> 194)
> at
> org
> .apache
> .pig
> .backend
> .hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:
> 133)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
> at org.apache.pig.PigServer.<init>(PigServer.java:144)
> at org.apache.pig.PigServer.<init>(PigServer.java:132)
> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
> at org.apache.pig.Main.main(Main.java:318)
> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
> response
> at org.apache.hadoop.ipc.Client.call(Client.java:559)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
> at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
> at
> org
> .apache
> .hadoop
> .dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
> at
> org
> .apache
> .pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:
> 72)
> ... 8 more
>
> Your advice is much appreciated!
> George
>
> 2009/5/25 George Pang <p0...@gmail.com>
>
>> Hi users,
>>
>> When I run pig in hadoop mode, before the grunt shell appears there
>> is
>> something on the command prompt not quite right:
>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> grunt> "
>>
>> If I use grunt> ls , it only displays the local files, not on
>> hdfs. Could
>> anyone help with these?
>>
>> George
>>
Re: Pig running hadoop mode
Posted by George Pang <p0...@gmail.com>.
Dear users,
Now my pig can connect to my hdfs, as on the screeen:
wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
2009-05-26 20:22:29,243 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: hdfs://localhost:54310
..........
But then it failed and generate an error log:
ERROR 2999: Unexpected internal error. Failed to create DataStorage
java.lang.RuntimeException: Failed to create DataStorage
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
at org.apache.pig.PigServer.<init>(PigServer.java:144)
at org.apache.pig.PigServer.<init>(PigServer.java:132)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
at org.apache.pig.Main.main(Main.java:318)
Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
response
at org.apache.hadoop.ipc.Client.call(Client.java:559)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 8 more
Your advice is much appreciated!
George
2009/5/25 George Pang <p0...@gmail.com>
> Hi users,
>
> When I run pig in hadoop mode, before the grunt shell appears there is
> something on the command prompt not quite right:
> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> grunt> "
>
> If I use grunt> ls , it only displays the local files, not on hdfs. Could
> anyone help with these?
>
> George
>