You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by an...@nokia.com on 2011/01/21 03:09:00 UTC

embeded pig is not conecting to hdfs cluster

I can connect to hdfs while running in script mode but am having trouble doing so in embedded mode(local mode is fine)
I've tried all possible export settings, could someone please point out what I'm doing wrong in trying to connect to HDFS
Do I have to do anything else in the java file other than pigServer = new PigServer("mapreduce")

which hadoop
/usr/local/software/hadoop-0.20.2/bin/hadoop

pig -x mapreduce try1.pig                      is fine ... Connecting to hadoop file system at: hdfs://master-hadoop:54310
java -cp pig.jar:. test.java                   is fine too (pigServer = new PigServer("local"))

echo $HADOOPDIR gives /usr/local/software/hadoop-0.20.2/conf
however
java  -cp pig.jar:.:$HADOOPDIR  test  is not OK (pigServer = new PigServer("mapreduce")  Connecting to hadoop file system at: file:/// instead of hdfs://)
Also it shows: ERROR mapReduceLayer.MapReduceLauncher: java.io.IOException: sample2.txt does not exist

I have sample2.txt in local dir as well as in hdfs
ls -l sample2.txt
-rw-r--r-- 1 hadoop hadoop 225 2011-01-13 17:47 sample2.txt
hadoop dfs -ls sample2.txt
Found 1 items
-rw-r--r--   3 hadoop supergroup        225 2011-01-20 18:34 /user/hadoop/sample2.txt
-----
export JAVA_HOME=/usr/local/jdk1.6.0_18
HADOOP_HOME=/usr/local/software/hadoop-0.20.2
PIG_HOME=/home/hadoop/code/pig-0.7.0/pig-0.7.0
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$PIG_HOME/bin

export HADOOPDIR=$HADOOP_HOME/conf
export PIGDIR=$PIG_HOME
export PIG_HADOOP_VERSION=20
export PIG_CLASSPATH=$PIG_HOME/pig.jar:.:$HADOOPDIR

thanks a lot!