You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by George Pang <p0...@gmail.com> on 2009/05/25 20:43:28 UTC

Pig running hadoop mode

Hi users,

When I run pig in hadoop mode, before the grunt shell appears there is
something on the command prompt not quite right:
"Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> "

If I use grunt> ls , it only displays the local files, not on hdfs.  Could
anyone help with these?

George

Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
This is how I set the environment variables, and how I get the error:

wpang@wpang-desktop:~$ echo *$PIGDIR*
/home/wpang/pigtmp
wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
/home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
wpang@wpang-desktop:~$ echo *$JAVA_HOME*
/usr/lib/jvm/jdk1.6.0_13

wpang@wpang-desktop:~/pig-0.2.0$ cd ~/pig-0.2.0/bin
wpang@wpang-desktop:~/pig-0.2.0/bin$* pig*
2009-05-26 16:52:38,785 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - *Connecting
to hadoop file system at: file:///*
2009-05-26 16:52:38,909 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId=
grunt>

Thanks,
George

2009/5/26 Naber, Chad <CN...@edmunds.com>

> George,
>
> Set your PIG_CLASSPATH to the hadoop/conf directory to run Grunt in
> mapreduce mode.
>
> Chad
>
> -----Original Message-----
> From: George Pang [mailto:p0941p@gmail.com]
> Sent: Tuesday, May 26, 2009 3:12 PM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig running hadoop mode
>
> Hi Alan,
> I can run the examples for pig local mode and pig hadoop mode from pig
> tutorial
> http://hadoop.apache.org/pig/docs/r0.2.0/tutorial.html#Pig+Installation,
> with the results displayed in the hadoop cluster.
>
> My questions are on using Grunt Shell, according to the "quick start" guide
> (http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html)<
> http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html>
>
> 1)  Following the instruction from link above, which reads : To run Pig's
> Grunt shell in local mode, follow these instructions.
>
> First, point $PIG_CLASSPATH to the pig.jar file (in your current working
> directory):
>
> $ export PIG_CLASSPATH=./pig.jar
>
> From your current working directory, run:
>
> $ pig -x local
>
> I got this :"pig is not installed. " ie., system can't find it from a
> directory with only pig.jar, the sample code files and a .txt data file.
>
> 2)  If I try to run the grunt shell from /pig-0.2.0/bin, the Grunt shell
> will run.  However, it only run locally.  If I run -x  mapreduce,  I got
> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> grunt> "
> I set the following environment variables: PIGDIR, HADOOPDIR,
> PIG_CLASSPATH, JAVA_HOME.  Are there more environment variables to set?
> Thank you.
>
> George
>
> 2009/5/26 Alan Gates <ga...@yahoo-inc.com>
>
> > This means you're connecting to your local file system instead of your
> > hadoop cluster.  Check out
> > http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html, especially
> > the various last section on how to connect to your hadoop cluster.
> >
> > Alan.
> >
> >
> > On May 25, 2009, at 11:43 AM, George Pang wrote:
> >
> >  Hi users,
> >>
> >> When I run pig in hadoop mode, before the grunt shell appears there
> >> is something on the command prompt not quite right:
> >> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> >> grunt> "
> >>
> >> If I use grunt> ls , it only displays the local files, not on hdfs.
> >> Could anyone help with these?
> >>
> >> George
> >>
> >
> >
>
>

RE: Pig running hadoop mode

Posted by "Naber, Chad" <CN...@edmunds.com>.
George,

Set your PIG_CLASSPATH to the hadoop/conf directory to run Grunt in mapreduce mode.

Chad 

-----Original Message-----
From: George Pang [mailto:p0941p@gmail.com] 
Sent: Tuesday, May 26, 2009 3:12 PM
To: pig-user@hadoop.apache.org
Subject: Re: Pig running hadoop mode

Hi Alan,
I can run the examples for pig local mode and pig hadoop mode from pig tutorial http://hadoop.apache.org/pig/docs/r0.2.0/tutorial.html#Pig+Installation,
with the results displayed in the hadoop cluster.

My questions are on using Grunt Shell, according to the "quick start" guide (http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html)<http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html>

1)  Following the instruction from link above, which reads : To run Pig's Grunt shell in local mode, follow these instructions.

First, point $PIG_CLASSPATH to the pig.jar file (in your current working
directory):

$ export PIG_CLASSPATH=./pig.jar

>From your current working directory, run:

$ pig -x local

I got this :"pig is not installed. " ie., system can't find it from a directory with only pig.jar, the sample code files and a .txt data file.

2)  If I try to run the grunt shell from /pig-0.2.0/bin, the Grunt shell will run.  However, it only run locally.  If I run -x  mapreduce,  I got "Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> "
I set the following environment variables: PIGDIR, HADOOPDIR, PIG_CLASSPATH, JAVA_HOME.  Are there more environment variables to set?
Thank you.

George

2009/5/26 Alan Gates <ga...@yahoo-inc.com>

> This means you're connecting to your local file system instead of your 
> hadoop cluster.  Check out 
> http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html, especially 
> the various last section on how to connect to your hadoop cluster.
>
> Alan.
>
>
> On May 25, 2009, at 11:43 AM, George Pang wrote:
>
>  Hi users,
>>
>> When I run pig in hadoop mode, before the grunt shell appears there 
>> is something on the command prompt not quite right:
>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> grunt> "
>>
>> If I use grunt> ls , it only displays the local files, not on hdfs.  
>> Could anyone help with these?
>>
>> George
>>
>
>


Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
Hi Alan,
I can run the examples for pig local mode and pig hadoop mode from pig
tutorial
http://hadoop.apache.org/pig/docs/r0.2.0/tutorial.html#Pig+Installation,
with the results displayed in the hadoop cluster.

My questions are on using Grunt Shell, according to the "quick start" guide
(http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html)<http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html>

1)  Following the instruction from link above, which reads : To run Pig’s
Grunt shell in local mode, follow these instructions.

First, point $PIG_CLASSPATH to the pig.jar file (in your current working
directory):

$ export PIG_CLASSPATH=./pig.jar

>From your current working directory, run:

$ pig -x local

I got this :"pig is not installed. " ie., system can't find it from a
directory with only pig.jar, the sample code files and a .txt data file.

2)  If I try to run the grunt shell from /pig-0.2.0/bin, the Grunt shell
will run.  However, it only run locally.  If I run -x  mapreduce,  I
got "Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> "
I set the following environment variables: PIGDIR, HADOOPDIR, PIG_CLASSPATH,
JAVA_HOME.  Are there more environment variables to set?
Thank you.

George

2009/5/26 Alan Gates <ga...@yahoo-inc.com>

> This means you're connecting to your local file system instead of your
> hadoop cluster.  Check out
> http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html, especially the
> various last section on how to connect to your hadoop cluster.
>
> Alan.
>
>
> On May 25, 2009, at 11:43 AM, George Pang wrote:
>
>  Hi users,
>>
>> When I run pig in hadoop mode, before the grunt shell appears there is
>> something on the command prompt not quite right:
>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> grunt> "
>>
>> If I use grunt> ls , it only displays the local files, not on hdfs.  Could
>> anyone help with these?
>>
>> George
>>
>
>

Re: Pig running hadoop mode

Posted by Alan Gates <ga...@yahoo-inc.com>.
This means you're connecting to your local file system instead of your  
hadoop cluster.  Check out http://hadoop.apache.org/pig/docs/r0.2.0/quickstart.html 
, especially the various last section on how to connect to your hadoop  
cluster.

Alan.

On May 25, 2009, at 11:43 AM, George Pang wrote:

> Hi users,
>
> When I run pig in hadoop mode, before the grunt shell appears there is
> something on the command prompt not quite right:
> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> grunt> "
>
> If I use grunt> ls , it only displays the local files, not on hdfs.   
> Could
> anyone help with these?
>
> George


Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
Hi,
I added only this, and it works now.

*% export PIG_HADOOP_VERSION=18*

George


2009/5/28 George Pang <p0...@gmail.com>

> Obviously my program has problem going to this line:
>
> 481  [main] INFO
>  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  -
> Connecting to map-reduce job tracker at: localhost:54311
>
> I got this line from Cloudera's VM.
>
> George
>
>
> 2009/5/27 George Pang <p0...@gmail.com>
>
>> Correction: the port data seems strange.  After I
>> restart Ubuntu, it is like:
>>
>> wpang2@wpang2-desktop:~$ sudo netstat -plten | grep java
>> tcp6       0      0 127.0.0.1:48961         :::*
>> LISTEN      1000       19407       6067/java
>> tcp6       0      0 :::50020                :::*
>> LISTEN      1000       19428       5826/java
>> tcp6       0      0 127.0.0.1:54310         :::*
>> LISTEN      1000       18602       5745/java
>> tcp6       0      0 127.0.0.1:54311         :::*
>> LISTEN      1000       18979       5979/java
>> tcp6       0      0 :::59271                :::*
>> LISTEN      1000       18849       5911/java
>> tcp6       0      0 :::50090                :::*
>> LISTEN      1000       19421       5911/java
>> tcp6       0      0 :::49067                :::*
>> LISTEN      1000       18754       5826/java
>> tcp6       0      0 :::50060                :::*
>> LISTEN      1000       19402       6067/java
>> tcp6       0      0 :::50030                :::*
>> LISTEN      1000       19341       5979/java
>> tcp6       0      0 :::40047                :::*
>> LISTEN      1000       18932       5979/java
>> tcp6       0      0 :::50070                :::*
>> LISTEN      1000       19002       5745/java
>> tcp6       0      0 :::40887                :::*
>> LISTEN      1000       18507       5745/java
>> tcp6       0      0 :::50010                :::*
>> LISTEN      1000       19059       5826/java
>> tcp6       0      0 :::50075                :::*
>> LISTEN      1000       19422       5826/java
>>
>> George
>>
>> 2009/5/27 George Pang <p0...@gmail.com>
>>
>> Hi Chad,
>>>
>>> I think you maybe right.  I run fsck, and get this:
>>> fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'
>>>
>>> Since I am a Windows-turned linux user, I don't know much about this
>>> error. In an initial search online, I think it's something wrong with my
>>> disk partition.  However I run my Ubuntu on a Virtual Machine.   I hope this
>>> won't make things more complex.
>>>
>>> Other than this, my hadoop is now a single-node cluster.  Some statuss
>>> are as follows:
>>>
>>> wpang2@wpang2-desktop:~$* jps*
>>> 6039 NameNode
>>> 6103 DataNode
>>> 6180 SecondaryNameNode
>>> 6332 TaskTracker
>>> 6254 JobTracker
>>> 8139 Jps
>>>
>>>  wpang2@wpang2-desktop:~$  sudo *netstat -plten | grep java*
>>> [sudo] password for wpang2:
>>> tcp6       0      0 :::50020                :::*
>>> LISTEN      1000       21585       6103/java
>>> tcp6       0      0 127.0.0.1:59621         :::*
>>> LISTEN      1000       21319       6332/java
>>> tcp6       0      0 127.0.0.1:9000          :::*
>>> LISTEN      1000       20964       6039/java
>>> tcp6       0      0 127.0.0.1:9001          :::*
>>> LISTEN      1000       20962       6254/java
>>> tcp6       0      0 :::50090                :::*
>>> LISTEN      1000       21567       6180/java
>>> tcp6       0      0 :::54539                :::*
>>> LISTEN      1000       20937       6103/java
>>> tcp6       0      0 :::50060                :::*
>>> LISTEN      1000       21313       6332/java
>>> tcp6       0      0 :::48237                :::*
>>> LISTEN      1000       20941       6180/java
>>> tcp6       0      0 :::50030                :::*
>>> LISTEN      1000       21315       6254/java
>>> tcp6       0      0 :::46511                :::*
>>> LISTEN      1000       20943       6039/java
>>> tcp6       0      0 :::43189                :::*
>>> LISTEN      1000       20939       6254/java
>>> tcp6       0      0 :::50070                :::*
>>> LISTEN      1000       21318       6039/java
>>> tcp6       0      0 :::50010                :::*
>>> LISTEN      1000       21485       6103/java
>>> tcp6       0      0 :::50075                :::*
>>> LISTEN      1000       21569       6103/java
>>>
>>> Thank you,
>>>
>>> George
>>>
>>>
>>> 2009/5/27 Naber, Chad <CN...@edmunds.com>
>>>
>>>
>>>> George,
>>>>
>>>> I would bet that there are some problems with the HDFS
>>>> 'hdfs://localhost:54310' you are connecting to.  Have you verified that the
>>>> HDFS is up and is healthy?  Maybe do an FSCK on the blocks to verify that
>>>> there are no corrupt blocks.
>>>>
>>>> Chad
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: George Pang [mailto:p0941p@gmail.com]
>>>> Sent: Wednesday, May 27, 2009 12:40 PM
>>>> To: pig-user@hadoop.apache.org
>>>> Subject: Re: Pig running hadoop mode
>>>>
>>>> Hi Alan,
>>>> I set these 3 environment variables,
>>>>
>>>> wpang@wpang-desktop:~$ echo *$PIGDIR*
>>>> /home/wpang/pigtmp
>>>> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
>>>> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
>>>> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
>>>> /usr/lib/jvm/jdk1.6.0_13
>>>>
>>>> So the hadoop version is 0.18.3.
>>>>
>>>> I used to download the pig.jar from "pig tutorial" wiki page and
>>>> pig-0.2.0 release from the release page.  Now I try to build the pig from
>>>> subversion, and it gave me the same thing.
>>>>
>>>> Any idea will be much appreciated.
>>>>
>>>> George
>>>>
>>>>
>>>>
>>>> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>>>>
>>>> > What version of hadoop are you using?  Pig, as currently released,
>>>> > runs on 0.18.x hadoop.  If you're running a different version you made
>>>> > need to apply a patch to make it run with your version of hadoop.
>>>> >
>>>> > Alan.
>>>> >
>>>> >
>>>> > On May 26, 2009, at 8:39 PM, George Pang wrote:
>>>> >
>>>> >  Dear users,
>>>> >>
>>>> >> Now my pig can connect to my hdfs, as on the screeen:
>>>> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>>>> >> 2009-05-26 20:22:29,243 [main] INFO
>>>> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>>> >> Connecting to hadoop file system at: hdfs://localhost:54310
>>>> >> ..........
>>>> >>
>>>> >> But then it failed and generate an error log:
>>>> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>>>> >> java.lang.RuntimeException: Failed to create DataStorage
>>>> >>   at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>>>> >>   at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>>>> >>   at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>>>> >>   at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>>>> >>   at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>>>> >>   at org.apache.pig.PigServer.<init>(PigServer.java:144)
>>>> >>   at org.apache.pig.PigServer.<init>(PigServer.java:132)
>>>> >>   at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>>>> >>   at org.apache.pig.Main.main(Main.java:318)
>>>> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>>>> >> response
>>>> >>   at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>>> >>   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> >>   at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>>>> >>   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>>>> >>   at
>>>> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>>>> >>   at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>>>> >>   at
>>>> >>
>>>> >>
>>>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>>>> >>   at
>>>> >>
>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>>>> >>   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>>>> >>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>>>> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>>>> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>>>> >>   at
>>>> >>
>>>> >>
>>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>>>> >>   ... 8 more
>>>> >>
>>>> >> Your advice is much appreciated!
>>>> >> George
>>>> >>
>>>> >> 2009/5/25 George Pang <p0...@gmail.com>
>>>> >>
>>>> >>  Hi users,
>>>> >>>
>>>> >>> When I run pig in hadoop mode, before the grunt shell appears there
>>>> >>> is something on the command prompt not quite right:
>>>> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>>> >>> grunt> "
>>>> >>>
>>>> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>>> >>>  Could
>>>> >>> anyone help with these?
>>>> >>>
>>>> >>> George
>>>> >>>
>>>> >>>
>>>> >
>>>>
>>>>
>>>
>>
>

Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
Obviously my program has problem going to this line:

481  [main] INFO
 org.apache.pig.backend.hadoop.executionengine.HExecutionEngine  -
Connecting to map-reduce job tracker at: localhost:54311

I got this line from Cloudera's VM.

George


2009/5/27 George Pang <p0...@gmail.com>

> Correction: the port data seems strange.  After I
> restart Ubuntu, it is like:
>
> wpang2@wpang2-desktop:~$ sudo netstat -plten | grep java
> tcp6       0      0 127.0.0.1:48961         :::*
> LISTEN      1000       19407       6067/java
> tcp6       0      0 :::50020                :::*
> LISTEN      1000       19428       5826/java
> tcp6       0      0 127.0.0.1:54310         :::*
> LISTEN      1000       18602       5745/java
> tcp6       0      0 127.0.0.1:54311         :::*
> LISTEN      1000       18979       5979/java
> tcp6       0      0 :::59271                :::*
> LISTEN      1000       18849       5911/java
> tcp6       0      0 :::50090                :::*
> LISTEN      1000       19421       5911/java
> tcp6       0      0 :::49067                :::*
> LISTEN      1000       18754       5826/java
> tcp6       0      0 :::50060                :::*
> LISTEN      1000       19402       6067/java
> tcp6       0      0 :::50030                :::*
> LISTEN      1000       19341       5979/java
> tcp6       0      0 :::40047                :::*
> LISTEN      1000       18932       5979/java
> tcp6       0      0 :::50070                :::*
> LISTEN      1000       19002       5745/java
> tcp6       0      0 :::40887                :::*
> LISTEN      1000       18507       5745/java
> tcp6       0      0 :::50010                :::*
> LISTEN      1000       19059       5826/java
> tcp6       0      0 :::50075                :::*
> LISTEN      1000       19422       5826/java
>
> George
>
> 2009/5/27 George Pang <p0...@gmail.com>
>
> Hi Chad,
>>
>> I think you maybe right.  I run fsck, and get this:
>> fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'
>>
>> Since I am a Windows-turned linux user, I don't know much about this
>> error. In an initial search online, I think it's something wrong with my
>> disk partition.  However I run my Ubuntu on a Virtual Machine.   I hope this
>> won't make things more complex.
>>
>> Other than this, my hadoop is now a single-node cluster.  Some statuss are
>> as follows:
>>
>> wpang2@wpang2-desktop:~$* jps*
>> 6039 NameNode
>> 6103 DataNode
>> 6180 SecondaryNameNode
>> 6332 TaskTracker
>> 6254 JobTracker
>> 8139 Jps
>>
>>  wpang2@wpang2-desktop:~$  sudo *netstat -plten | grep java*
>> [sudo] password for wpang2:
>> tcp6       0      0 :::50020                :::*
>> LISTEN      1000       21585       6103/java
>> tcp6       0      0 127.0.0.1:59621         :::*
>> LISTEN      1000       21319       6332/java
>> tcp6       0      0 127.0.0.1:9000          :::*
>> LISTEN      1000       20964       6039/java
>> tcp6       0      0 127.0.0.1:9001          :::*
>> LISTEN      1000       20962       6254/java
>> tcp6       0      0 :::50090                :::*
>> LISTEN      1000       21567       6180/java
>> tcp6       0      0 :::54539                :::*
>> LISTEN      1000       20937       6103/java
>> tcp6       0      0 :::50060                :::*
>> LISTEN      1000       21313       6332/java
>> tcp6       0      0 :::48237                :::*
>> LISTEN      1000       20941       6180/java
>> tcp6       0      0 :::50030                :::*
>> LISTEN      1000       21315       6254/java
>> tcp6       0      0 :::46511                :::*
>> LISTEN      1000       20943       6039/java
>> tcp6       0      0 :::43189                :::*
>> LISTEN      1000       20939       6254/java
>> tcp6       0      0 :::50070                :::*
>> LISTEN      1000       21318       6039/java
>> tcp6       0      0 :::50010                :::*
>> LISTEN      1000       21485       6103/java
>> tcp6       0      0 :::50075                :::*
>> LISTEN      1000       21569       6103/java
>>
>> Thank you,
>>
>> George
>>
>>
>> 2009/5/27 Naber, Chad <CN...@edmunds.com>
>>
>>
>>> George,
>>>
>>> I would bet that there are some problems with the HDFS
>>> 'hdfs://localhost:54310' you are connecting to.  Have you verified that the
>>> HDFS is up and is healthy?  Maybe do an FSCK on the blocks to verify that
>>> there are no corrupt blocks.
>>>
>>> Chad
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: George Pang [mailto:p0941p@gmail.com]
>>> Sent: Wednesday, May 27, 2009 12:40 PM
>>> To: pig-user@hadoop.apache.org
>>> Subject: Re: Pig running hadoop mode
>>>
>>> Hi Alan,
>>> I set these 3 environment variables,
>>>
>>> wpang@wpang-desktop:~$ echo *$PIGDIR*
>>> /home/wpang/pigtmp
>>> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
>>> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
>>> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
>>> /usr/lib/jvm/jdk1.6.0_13
>>>
>>> So the hadoop version is 0.18.3.
>>>
>>> I used to download the pig.jar from "pig tutorial" wiki page and
>>> pig-0.2.0 release from the release page.  Now I try to build the pig from
>>> subversion, and it gave me the same thing.
>>>
>>> Any idea will be much appreciated.
>>>
>>> George
>>>
>>>
>>>
>>> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>>>
>>> > What version of hadoop are you using?  Pig, as currently released,
>>> > runs on 0.18.x hadoop.  If you're running a different version you made
>>> > need to apply a patch to make it run with your version of hadoop.
>>> >
>>> > Alan.
>>> >
>>> >
>>> > On May 26, 2009, at 8:39 PM, George Pang wrote:
>>> >
>>> >  Dear users,
>>> >>
>>> >> Now my pig can connect to my hdfs, as on the screeen:
>>> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>>> >> 2009-05-26 20:22:29,243 [main] INFO
>>> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> >> Connecting to hadoop file system at: hdfs://localhost:54310
>>> >> ..........
>>> >>
>>> >> But then it failed and generate an error log:
>>> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>>> >> java.lang.RuntimeException: Failed to create DataStorage
>>> >>   at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>>> >>   at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>>> >>   at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>>> >>   at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>>> >>   at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>>> >>   at org.apache.pig.PigServer.<init>(PigServer.java:144)
>>> >>   at org.apache.pig.PigServer.<init>(PigServer.java:132)
>>> >>   at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>>> >>   at org.apache.pig.Main.main(Main.java:318)
>>> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>>> >> response
>>> >>   at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>> >>   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> >>   at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>>> >>   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>>> >>   at
>>> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>>> >>   at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>>> >>   at
>>> >>
>>> >>
>>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>>> >>   at
>>> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>>> >>   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>>> >>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>>> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>>> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>>> >>   at
>>> >>
>>> >>
>>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>>> >>   ... 8 more
>>> >>
>>> >> Your advice is much appreciated!
>>> >> George
>>> >>
>>> >> 2009/5/25 George Pang <p0...@gmail.com>
>>> >>
>>> >>  Hi users,
>>> >>>
>>> >>> When I run pig in hadoop mode, before the grunt shell appears there
>>> >>> is something on the command prompt not quite right:
>>> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>> >>> grunt> "
>>> >>>
>>> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>> >>>  Could
>>> >>> anyone help with these?
>>> >>>
>>> >>> George
>>> >>>
>>> >>>
>>> >
>>>
>>>
>>
>

Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
Correction: the port data seems strange.  After I
restart Ubuntu, it is like:

wpang2@wpang2-desktop:~$ sudo netstat -plten | grep java
tcp6       0      0 127.0.0.1:48961         :::*
LISTEN      1000       19407       6067/java
tcp6       0      0 :::50020                :::*
LISTEN      1000       19428       5826/java
tcp6       0      0 127.0.0.1:54310         :::*
LISTEN      1000       18602       5745/java
tcp6       0      0 127.0.0.1:54311         :::*
LISTEN      1000       18979       5979/java
tcp6       0      0 :::59271                :::*
LISTEN      1000       18849       5911/java
tcp6       0      0 :::50090                :::*
LISTEN      1000       19421       5911/java
tcp6       0      0 :::49067                :::*
LISTEN      1000       18754       5826/java
tcp6       0      0 :::50060                :::*
LISTEN      1000       19402       6067/java
tcp6       0      0 :::50030                :::*
LISTEN      1000       19341       5979/java
tcp6       0      0 :::40047                :::*
LISTEN      1000       18932       5979/java
tcp6       0      0 :::50070                :::*
LISTEN      1000       19002       5745/java
tcp6       0      0 :::40887                :::*
LISTEN      1000       18507       5745/java
tcp6       0      0 :::50010                :::*
LISTEN      1000       19059       5826/java
tcp6       0      0 :::50075                :::*
LISTEN      1000       19422       5826/java

George

2009/5/27 George Pang <p0...@gmail.com>

> Hi Chad,
>
> I think you maybe right.  I run fsck, and get this:
> fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'
>
> Since I am a Windows-turned linux user, I don't know much about this error.
> In an initial search online, I think it's something wrong with my disk
> partition.  However I run my Ubuntu on a Virtual Machine.   I hope this
> won't make things more complex.
>
> Other than this, my hadoop is now a single-node cluster.  Some statuss are
> as follows:
>
> wpang2@wpang2-desktop:~$* jps*
> 6039 NameNode
> 6103 DataNode
> 6180 SecondaryNameNode
> 6332 TaskTracker
> 6254 JobTracker
> 8139 Jps
>
>  wpang2@wpang2-desktop:~$  sudo *netstat -plten | grep java*
> [sudo] password for wpang2:
> tcp6       0      0 :::50020                :::*
> LISTEN      1000       21585       6103/java
> tcp6       0      0 127.0.0.1:59621         :::*
> LISTEN      1000       21319       6332/java
> tcp6       0      0 127.0.0.1:9000          :::*
> LISTEN      1000       20964       6039/java
> tcp6       0      0 127.0.0.1:9001          :::*
> LISTEN      1000       20962       6254/java
> tcp6       0      0 :::50090                :::*
> LISTEN      1000       21567       6180/java
> tcp6       0      0 :::54539                :::*
> LISTEN      1000       20937       6103/java
> tcp6       0      0 :::50060                :::*
> LISTEN      1000       21313       6332/java
> tcp6       0      0 :::48237                :::*
> LISTEN      1000       20941       6180/java
> tcp6       0      0 :::50030                :::*
> LISTEN      1000       21315       6254/java
> tcp6       0      0 :::46511                :::*
> LISTEN      1000       20943       6039/java
> tcp6       0      0 :::43189                :::*
> LISTEN      1000       20939       6254/java
> tcp6       0      0 :::50070                :::*
> LISTEN      1000       21318       6039/java
> tcp6       0      0 :::50010                :::*
> LISTEN      1000       21485       6103/java
> tcp6       0      0 :::50075                :::*
> LISTEN      1000       21569       6103/java
>
> Thank you,
>
> George
>
>
> 2009/5/27 Naber, Chad <CN...@edmunds.com>
>
>
>> George,
>>
>> I would bet that there are some problems with the HDFS
>> 'hdfs://localhost:54310' you are connecting to.  Have you verified that the
>> HDFS is up and is healthy?  Maybe do an FSCK on the blocks to verify that
>> there are no corrupt blocks.
>>
>> Chad
>>
>>
>>
>>
>> -----Original Message-----
>> From: George Pang [mailto:p0941p@gmail.com]
>> Sent: Wednesday, May 27, 2009 12:40 PM
>> To: pig-user@hadoop.apache.org
>> Subject: Re: Pig running hadoop mode
>>
>> Hi Alan,
>> I set these 3 environment variables,
>>
>> wpang@wpang-desktop:~$ echo *$PIGDIR*
>> /home/wpang/pigtmp
>> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
>> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
>> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
>> /usr/lib/jvm/jdk1.6.0_13
>>
>> So the hadoop version is 0.18.3.
>>
>> I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0
>> release from the release page.  Now I try to build the pig from subversion,
>> and it gave me the same thing.
>>
>> Any idea will be much appreciated.
>>
>> George
>>
>>
>>
>> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>>
>> > What version of hadoop are you using?  Pig, as currently released,
>> > runs on 0.18.x hadoop.  If you're running a different version you made
>> > need to apply a patch to make it run with your version of hadoop.
>> >
>> > Alan.
>> >
>> >
>> > On May 26, 2009, at 8:39 PM, George Pang wrote:
>> >
>> >  Dear users,
>> >>
>> >> Now my pig can connect to my hdfs, as on the screeen:
>> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>> >> 2009-05-26 20:22:29,243 [main] INFO
>> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> >> Connecting to hadoop file system at: hdfs://localhost:54310
>> >> ..........
>> >>
>> >> But then it failed and generate an error log:
>> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>> >> java.lang.RuntimeException: Failed to create DataStorage
>> >>   at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>> >>   at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>> >>   at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>> >>   at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>> >>   at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>> >>   at org.apache.pig.PigServer.<init>(PigServer.java:144)
>> >>   at org.apache.pig.PigServer.<init>(PigServer.java:132)
>> >>   at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>> >>   at org.apache.pig.Main.main(Main.java:318)
>> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>> >> response
>> >>   at org.apache.hadoop.ipc.Client.call(Client.java:559)
>> >>   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>   at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>> >>   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>> >>   at
>> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>> >>   at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>> >>   at
>> >>
>> >>
>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>> >>   at
>> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>> >>   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>> >>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>> >>   at
>> >>
>> >>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>> >>   ... 8 more
>> >>
>> >> Your advice is much appreciated!
>> >> George
>> >>
>> >> 2009/5/25 George Pang <p0...@gmail.com>
>> >>
>> >>  Hi users,
>> >>>
>> >>> When I run pig in hadoop mode, before the grunt shell appears there
>> >>> is something on the command prompt not quite right:
>> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> >>> grunt> "
>> >>>
>> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
>> >>>  Could
>> >>> anyone help with these?
>> >>>
>> >>> George
>> >>>
>> >>>
>> >
>>
>>
>

Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
Hi Chad,

I think you maybe right.  I run fsck, and get this:
fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'

Since I am a Windows-turned linux user, I don't know much about this error.
In an initial search online, I think it's something wrong with my disk
partition.  However I run my Ubuntu on a Virtual Machine.   I hope this
won't make things more complex.

Other than this, my hadoop is now a single-node cluster.  Some statuss are
as follows:

wpang2@wpang2-desktop:~$* jps*
6039 NameNode
6103 DataNode
6180 SecondaryNameNode
6332 TaskTracker
6254 JobTracker
8139 Jps

 wpang2@wpang2-desktop:~$  sudo *netstat -plten | grep java*
[sudo] password for wpang2:
tcp6       0      0 :::50020                :::*
LISTEN      1000       21585       6103/java
tcp6       0      0 127.0.0.1:59621         :::*
LISTEN      1000       21319       6332/java
tcp6       0      0 127.0.0.1:9000          :::*
LISTEN      1000       20964       6039/java
tcp6       0      0 127.0.0.1:9001          :::*
LISTEN      1000       20962       6254/java
tcp6       0      0 :::50090                :::*
LISTEN      1000       21567       6180/java
tcp6       0      0 :::54539                :::*
LISTEN      1000       20937       6103/java
tcp6       0      0 :::50060                :::*
LISTEN      1000       21313       6332/java
tcp6       0      0 :::48237                :::*
LISTEN      1000       20941       6180/java
tcp6       0      0 :::50030                :::*
LISTEN      1000       21315       6254/java
tcp6       0      0 :::46511                :::*
LISTEN      1000       20943       6039/java
tcp6       0      0 :::43189                :::*
LISTEN      1000       20939       6254/java
tcp6       0      0 :::50070                :::*
LISTEN      1000       21318       6039/java
tcp6       0      0 :::50010                :::*
LISTEN      1000       21485       6103/java
tcp6       0      0 :::50075                :::*
LISTEN      1000       21569       6103/java

Thank you,

George


2009/5/27 Naber, Chad <CN...@edmunds.com>

>
> George,
>
> I would bet that there are some problems with the HDFS
> 'hdfs://localhost:54310' you are connecting to.  Have you verified that the
> HDFS is up and is healthy?  Maybe do an FSCK on the blocks to verify that
> there are no corrupt blocks.
>
> Chad
>
>
>
>
> -----Original Message-----
> From: George Pang [mailto:p0941p@gmail.com]
> Sent: Wednesday, May 27, 2009 12:40 PM
> To: pig-user@hadoop.apache.org
> Subject: Re: Pig running hadoop mode
>
> Hi Alan,
> I set these 3 environment variables,
>
> wpang@wpang-desktop:~$ echo *$PIGDIR*
> /home/wpang/pigtmp
> wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
> /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
> wpang@wpang-desktop:~$ echo *$JAVA_HOME*
> /usr/lib/jvm/jdk1.6.0_13
>
> So the hadoop version is 0.18.3.
>
> I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0
> release from the release page.  Now I try to build the pig from subversion,
> and it gave me the same thing.
>
> Any idea will be much appreciated.
>
> George
>
>
>
> 2009/5/27 Alan Gates <ga...@yahoo-inc.com>
>
> > What version of hadoop are you using?  Pig, as currently released,
> > runs on 0.18.x hadoop.  If you're running a different version you made
> > need to apply a patch to make it run with your version of hadoop.
> >
> > Alan.
> >
> >
> > On May 26, 2009, at 8:39 PM, George Pang wrote:
> >
> >  Dear users,
> >>
> >> Now my pig can connect to my hdfs, as on the screeen:
> >> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
> >> 2009-05-26 20:22:29,243 [main] INFO
> >> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> >> Connecting to hadoop file system at: hdfs://localhost:54310
> >> ..........
> >>
> >> But then it failed and generate an error log:
> >> ERROR 2999: Unexpected internal error. Failed to create DataStorage
> >> java.lang.RuntimeException: Failed to create DataStorage
> >>   at
> >>
> >>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
> >>   at
> >>
> >>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
> >>   at
> >>
> >>
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
> >>   at
> >>
> >>
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
> >>   at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
> >>   at org.apache.pig.PigServer.<init>(PigServer.java:144)
> >>   at org.apache.pig.PigServer.<init>(PigServer.java:132)
> >>   at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
> >>   at org.apache.pig.Main.main(Main.java:318)
> >> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
> >> response
> >>   at org.apache.hadoop.ipc.Client.call(Client.java:559)
> >>   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>   at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
> >>   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
> >>   at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
> >>   at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
> >>   at
> >>
> >>
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
> >>   at
> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
> >>   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
> >>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
> >>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
> >>   at
> >>
> >>
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
> >>   ... 8 more
> >>
> >> Your advice is much appreciated!
> >> George
> >>
> >> 2009/5/25 George Pang <p0...@gmail.com>
> >>
> >>  Hi users,
> >>>
> >>> When I run pig in hadoop mode, before the grunt shell appears there
> >>> is something on the command prompt not quite right:
> >>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> >>> grunt> "
> >>>
> >>> If I use grunt> ls , it only displays the local files, not on hdfs.
> >>>  Could
> >>> anyone help with these?
> >>>
> >>> George
> >>>
> >>>
> >
>
>

RE: Pig running hadoop mode

Posted by "Naber, Chad" <CN...@edmunds.com>.
 
George,

I would bet that there are some problems with the HDFS 'hdfs://localhost:54310' you are connecting to.  Have you verified that the HDFS is up and is healthy?  Maybe do an FSCK on the blocks to verify that there are no corrupt blocks.

Chad




-----Original Message-----
From: George Pang [mailto:p0941p@gmail.com] 
Sent: Wednesday, May 27, 2009 12:40 PM
To: pig-user@hadoop.apache.org
Subject: Re: Pig running hadoop mode

Hi Alan,
I set these 3 environment variables,

wpang@wpang-desktop:~$ echo *$PIGDIR*
/home/wpang/pigtmp
wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH* /home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
wpang@wpang-desktop:~$ echo *$JAVA_HOME*
/usr/lib/jvm/jdk1.6.0_13

So the hadoop version is 0.18.3.

I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0 release from the release page.  Now I try to build the pig from subversion, and it gave me the same thing.

Any idea will be much appreciated.

George



2009/5/27 Alan Gates <ga...@yahoo-inc.com>

> What version of hadoop are you using?  Pig, as currently released, 
> runs on 0.18.x hadoop.  If you're running a different version you made 
> need to apply a patch to make it run with your version of hadoop.
>
> Alan.
>
>
> On May 26, 2009, at 8:39 PM, George Pang wrote:
>
>  Dear users,
>>
>> Now my pig can connect to my hdfs, as on the screeen:
>> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>> 2009-05-26 20:22:29,243 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - 
>> Connecting to hadoop file system at: hdfs://localhost:54310 
>> ..........
>>
>> But then it failed and generate an error log:
>> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>> java.lang.RuntimeException: Failed to create DataStorage
>>   at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>>   at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>>   at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>>   at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>>   at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>>   at org.apache.pig.PigServer.<init>(PigServer.java:144)
>>   at org.apache.pig.PigServer.<init>(PigServer.java:132)
>>   at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>>   at org.apache.pig.Main.main(Main.java:318)
>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc 
>> response
>>   at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>   at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>>   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>>   at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>>   at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>>   at
>>
>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>>   at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>>   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>>   at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>>   ... 8 more
>>
>> Your advice is much appreciated!
>> George
>>
>> 2009/5/25 George Pang <p0...@gmail.com>
>>
>>  Hi users,
>>>
>>> When I run pig in hadoop mode, before the grunt shell appears there 
>>> is something on the command prompt not quite right:
>>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>> grunt> "
>>>
>>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>>  Could
>>> anyone help with these?
>>>
>>> George
>>>
>>>
>


Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
Hi Alan,
I set these 3 environment variables,

wpang@wpang-desktop:~$ echo *$PIGDIR*
/home/wpang/pigtmp
wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
/home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
wpang@wpang-desktop:~$ echo *$JAVA_HOME*
/usr/lib/jvm/jdk1.6.0_13

So the hadoop version is 0.18.3.

I used to download the pig.jar from "pig tutorial" wiki page and pig-0.2.0
release from the release page.  Now I try to build the pig from subversion,
and it gave me the same thing.

Any idea will be much appreciated.

George



2009/5/27 Alan Gates <ga...@yahoo-inc.com>

> What version of hadoop are you using?  Pig, as currently released, runs on
> 0.18.x hadoop.  If you're running a different version you made need to apply
> a patch to make it run with your version of hadoop.
>
> Alan.
>
>
> On May 26, 2009, at 8:39 PM, George Pang wrote:
>
>  Dear users,
>>
>> Now my pig can connect to my hdfs, as on the screeen:
>> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
>> 2009-05-26 20:22:29,243 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting
>> to hadoop file system at: hdfs://localhost:54310
>> ..........
>>
>> But then it failed and generate an error log:
>> ERROR 2999: Unexpected internal error. Failed to create DataStorage
>> java.lang.RuntimeException: Failed to create DataStorage
>>   at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
>>   at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>>   at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
>>   at
>>
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
>>   at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>>   at org.apache.pig.PigServer.<init>(PigServer.java:144)
>>   at org.apache.pig.PigServer.<init>(PigServer.java:132)
>>   at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>>   at org.apache.pig.Main.main(Main.java:318)
>> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
>> response
>>   at org.apache.hadoop.ipc.Client.call(Client.java:559)
>>   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>   at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>>   at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>>   at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>>   at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>>   at
>>
>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>>   at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>>   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>>   at
>>
>> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
>>   ... 8 more
>>
>> Your advice is much appreciated!
>> George
>>
>> 2009/5/25 George Pang <p0...@gmail.com>
>>
>>  Hi users,
>>>
>>> When I run pig in hadoop mode, before the grunt shell appears there is
>>> something on the command prompt not quite right:
>>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>>> grunt> "
>>>
>>> If I use grunt> ls , it only displays the local files, not on hdfs.
>>>  Could
>>> anyone help with these?
>>>
>>> George
>>>
>>>
>

Re: Pig running hadoop mode

Posted by Alan Gates <ga...@yahoo-inc.com>.
What version of hadoop are you using?  Pig, as currently released,  
runs on 0.18.x hadoop.  If you're running a different version you made  
need to apply a patch to make it run with your version of hadoop.

Alan.

On May 26, 2009, at 8:39 PM, George Pang wrote:

> Dear users,
>
> Now my pig can connect to my hdfs, as on the screeen:
> wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
> 2009-05-26 20:22:29,243 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -  
> Connecting
> to hadoop file system at: hdfs://localhost:54310
> ..........
>
> But then it failed and generate an error log:
> ERROR 2999: Unexpected internal error. Failed to create DataStorage
> java.lang.RuntimeException: Failed to create DataStorage
>    at
> org 
> .apache 
> .pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java: 
> 75)
>    at
> org 
> .apache 
> .pig 
> .backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
>    at
> org 
> .apache 
> .pig 
> .backend 
> .hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java: 
> 194)
>    at
> org 
> .apache 
> .pig 
> .backend 
> .hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java: 
> 133)
>    at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
>    at org.apache.pig.PigServer.<init>(PigServer.java:144)
>    at org.apache.pig.PigServer.<init>(PigServer.java:132)
>    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
>    at org.apache.pig.Main.main(Main.java:318)
> Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
> response
>    at org.apache.hadoop.ipc.Client.call(Client.java:559)
>    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>    at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>    at  
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>    at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>    at
> org 
> .apache 
> .hadoop 
> .dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>    at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>    at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>    at
> org 
> .apache 
> .pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java: 
> 72)
>    ... 8 more
>
> Your advice is much appreciated!
> George
>
> 2009/5/25 George Pang <p0...@gmail.com>
>
>> Hi users,
>>
>> When I run pig in hadoop mode, before the grunt shell appears there  
>> is
>> something on the command prompt not quite right:
>> "Initializing JVM Metrics with processName=JobTracker, sessionId=
>> grunt> "
>>
>> If I use grunt> ls , it only displays the local files, not on  
>> hdfs.  Could
>> anyone help with these?
>>
>> George
>>


Re: Pig running hadoop mode

Posted by George Pang <p0...@gmail.com>.
Dear users,

Now my pig can connect to my hdfs, as on the screeen:
wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
2009-05-26 20:22:29,243 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: hdfs://localhost:54310
..........

But then it failed and generate an error log:
ERROR 2999: Unexpected internal error. Failed to create DataStorage
java.lang.RuntimeException: Failed to create DataStorage
    at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
    at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
    at org.apache.pig.PigServer.<init>(PigServer.java:144)
    at org.apache.pig.PigServer.<init>(PigServer.java:132)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
    at org.apache.pig.Main.main(Main.java:318)
Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
response
    at org.apache.hadoop.ipc.Client.call(Client.java:559)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
    at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
    at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
    at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
    at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
    at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
    at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
    at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 8 more

Your advice is much appreciated!
George

2009/5/25 George Pang <p0...@gmail.com>

> Hi users,
>
> When I run pig in hadoop mode, before the grunt shell appears there is
> something on the command prompt not quite right:
> "Initializing JVM Metrics with processName=JobTracker, sessionId=
> grunt> "
>
> If I use grunt> ls , it only displays the local files, not on hdfs.  Could
> anyone help with these?
>
> George
>