You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by R J <rj...@yahoo.com> on 2014/07/28 02:46:57 UTC
Cannot compaile a basic PutMerge.java program
Hi All,
I am new to programming on hadoop. I tried to compile the following program (example program from a hadoop book) on my linix server where I have Haddop installed:
I get the errors:
$javac PutMerge.java
PutMerge.java:2: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
PutMerge.java:3: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataInputStream;
^
PutMerge.java:4: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataOutputStream;
^
PutMerge.java:5: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileStatus;
^
PutMerge.java:6: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileSystem;
^
PutMerge.java:7: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
I have $HADOOP_HOME set u:
$echo $HADOOP_HOME
/usr/lib/hadoop
Could you please suggest how to compile this program? Thanks a lot.
Shu
====PutMerge.java=========
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class PutMerge {
public static void main(String[] args) throws IOException {
Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.get(conf);
FileSystem local = FileSystem.getLocal(conf);
Path inputDir = new Path(args[0]);
Path hdfsFile = new Path(args[1]);
try {
FileStatus[] inputFiles = local.listStatus(inputDir);
FSDataOutputStream out = hdfs.create(hdfsFile);
for (int i=0; i<inputFiles.length; i++) {
System.out.println(inputFiles[i].getPath().getName());
FSDataInputStream in = local.open(inputFiles[i].getPath());
byte buffer[] = new byte[256];
int bytesRead = 0;
while( (bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
in.close();
}
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
=============
Re: Cannot compaile a basic PutMerge.java program
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
I can probably help you out with that. I don¹t want to sound patronising
though. What is your IDE and have you included the hadoop libraries in
your jar ?
Regards,
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: R J <rj...@yahoo.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 01:46
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Cannot compaile a basic PutMerge.java program
Hi All,
I am new to programming on hadoop. I tried to compile the following
program (example program from a hadoop book) on my linix server where I
have Haddop installed:
I get the errors:
$javac PutMerge.java
PutMerge.java:2: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
PutMerge.java:3: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataInputStream;
^
PutMerge.java:4: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataOutputStream;
^
PutMerge.java:5: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileStatus;
^
PutMerge.java:6: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileSystem;
^
PutMerge.java:7: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
I have $HADOOP_HOME set u:
$echo $HADOOP_HOME
/usr/lib/hadoop
Could you please suggest how to compile this program? Thanks a lot.
Shu
====PutMerge.java=========
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class PutMerge {
public static void main(String[] args) throws IOException {
Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.get(conf);
FileSystem local = FileSystem.getLocal(conf);
Path inputDir = new Path(args[0]);
Path hdfsFile = new Path(args[1]);
try {
FileStatus[] inputFiles =
local.listStatus(inputDir);
FSDataOutputStream out = hdfs.create(hdfsFile);
for (int i=0; i<inputFiles.length; i++) {
System.out.println(inputFiles[i].getPath().getName());
FSDataInputStream in = local.open(inputFiles[i].getPath());
byte buffer[] = new byte[256];
int bytesRead = 0;
while( (bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
in.close();
}
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
=============
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thank you.
I compileted with the command:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$javac -cp $CLASSPATH PutMerge.java
$ls PutMerge.class
PutMerge.class
Now I tried:
$java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
14/07/28 16:33:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/28 16:33:40 INFO security.JniBasedUnixGroupsMapping: Bailing out since native library couldn't be loaded
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:951)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1002)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028)
at org.apache.hadoop.security.Groups.<init>(Groups.java:48)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:137)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:251)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:229)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:307)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:536)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:519)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1717)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1710)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1578)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
at PutMerge.main(PutMerge.java:12)
Caused by: java.lang.RuntimeException
at org.apache.hadoop.security.JniBasedUnixGroupsMapping.<clinit>(JniBasedUnixGroupsMapping.java:44)
... 18 more
If I try:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$java -cp $($CLASSPATH classpath):. PutMerge
-bash: /usr/lib/hive/lib/hive-serde-0.7.1-cdh3u4.jar:/usr/lib/hive/lib/hive-exec-0.7.1-cdh3u4.jar:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u4.jar: No such file or directory
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
On Monday, July 28, 2014 12:42 AM, Harsh J <ha...@cloudera.com> wrote:
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thank you.
I compileted with the command:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$javac -cp $CLASSPATH PutMerge.java
$ls PutMerge.class
PutMerge.class
Now I tried:
$java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
14/07/28 16:33:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/28 16:33:40 INFO security.JniBasedUnixGroupsMapping: Bailing out since native library couldn't be loaded
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:951)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1002)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028)
at org.apache.hadoop.security.Groups.<init>(Groups.java:48)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:137)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:251)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:229)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:307)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:536)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:519)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1717)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1710)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1578)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
at PutMerge.main(PutMerge.java:12)
Caused by: java.lang.RuntimeException
at org.apache.hadoop.security.JniBasedUnixGroupsMapping.<clinit>(JniBasedUnixGroupsMapping.java:44)
... 18 more
If I try:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$java -cp $($CLASSPATH classpath):. PutMerge
-bash: /usr/lib/hive/lib/hive-serde-0.7.1-cdh3u4.jar:/usr/lib/hive/lib/hive-exec-0.7.1-cdh3u4.jar:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u4.jar: No such file or directory
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
On Monday, July 28, 2014 12:42 AM, Harsh J <ha...@cloudera.com> wrote:
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thank you.
I compileted with the command:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$javac -cp $CLASSPATH PutMerge.java
$ls PutMerge.class
PutMerge.class
Now I tried:
$java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
14/07/28 16:33:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/28 16:33:40 INFO security.JniBasedUnixGroupsMapping: Bailing out since native library couldn't be loaded
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:951)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1002)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028)
at org.apache.hadoop.security.Groups.<init>(Groups.java:48)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:137)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:251)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:229)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:307)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:536)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:519)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1717)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1710)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1578)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
at PutMerge.main(PutMerge.java:12)
Caused by: java.lang.RuntimeException
at org.apache.hadoop.security.JniBasedUnixGroupsMapping.<clinit>(JniBasedUnixGroupsMapping.java:44)
... 18 more
If I try:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$java -cp $($CLASSPATH classpath):. PutMerge
-bash: /usr/lib/hive/lib/hive-serde-0.7.1-cdh3u4.jar:/usr/lib/hive/lib/hive-exec-0.7.1-cdh3u4.jar:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u4.jar: No such file or directory
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
On Monday, July 28, 2014 12:42 AM, Harsh J <ha...@cloudera.com> wrote:
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thank you.
I compileted with the command:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$javac -cp $CLASSPATH PutMerge.java
$ls PutMerge.class
PutMerge.class
Now I tried:
$java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
14/07/28 16:33:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/28 16:33:40 INFO security.JniBasedUnixGroupsMapping: Bailing out since native library couldn't be loaded
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:951)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1002)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028)
at org.apache.hadoop.security.Groups.<init>(Groups.java:48)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:137)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:251)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:229)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:307)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:536)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:519)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1717)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1710)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1578)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:111)
at PutMerge.main(PutMerge.java:12)
Caused by: java.lang.RuntimeException
at org.apache.hadoop.security.JniBasedUnixGroupsMapping.<clinit>(JniBasedUnixGroupsMapping.java:44)
... 18 more
If I try:
$CLASSPATH=$(ls $HIVE_HOME/lib/hive-serde-*.jar):$(ls $HIVE_HOME/lib/hive-exec-*.jar):$(ls $HADOOP_HOME/hadoop-core-*.jar)
$java -cp $($CLASSPATH classpath):. PutMerge
-bash: /usr/lib/hive/lib/hive-serde-0.7.1-cdh3u4.jar:/usr/lib/hive/lib/hive-exec-0.7.1-cdh3u4.jar:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u4.jar: No such file or directory
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
On Monday, July 28, 2014 12:42 AM, Harsh J <ha...@cloudera.com> wrote:
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
Please run it in the same style. The binary 'java' accepts a -cp param too:
java -cp $($HADOOP_HOME/bin/hadoop classpath):. PutMerge
On Mon, Jul 28, 2014 at 11:21 AM, R J <rj...@yahoo.com> wrote:
> Thanks a lot! I could compile with the added classpath:
> $javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
> The above created PutMerge.class file.
> Now I try to run:
> $java PutMerge
> Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
> Caused by: java.lang.ClassNotFoundException: PutMerge
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: PutMerge. Program will exit.
>
> I get the above error.
> I tried:
> $set CLASSPATH=/usr/lib/hadoop/bin/hadoop
> $java PutMerge
>
> I still get the error.
>
>
>
> On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
>
>
> The javac program can only find import dependencies referenced in a
> program if it is also supplied on the javac classpath. Setting
> HADOOP_HOME alone will not magically do this. Have you set an
> appropriate classpath?
>
> Try as below, perhaps:
>
> javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
>
> Alternatively, consider using a modern build helper tool such as
> Apache Maven for writing java applications, they make your work
> easier.
>
> On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
>> Hi All,
>>
>> I am new to programming on hadoop. I tried to compile the following
>> program
>> (example program from a hadoop book) on my linix server where I have
>> Haddop
>> installed:
>> I get the errors:
>> $javac PutMerge.java
>> PutMerge.java:2: package org.apache.hadoop.conf does not exist
>> import org.apache.hadoop.conf.Configuration;
>> ^
>> PutMerge.java:3: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataInputStream;
>> ^
>> PutMerge.java:4: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> ^
>> PutMerge.java:5: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileStatus;
>> ^
>> PutMerge.java:6: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.FileSystem;
>> ^
>> PutMerge.java:7: package org.apache.hadoop.fs does not exist
>> import org.apache.hadoop.fs.Path;
>>
>> I have $HADOOP_HOME set u:
>> $echo $HADOOP_HOME
>> /usr/lib/hadoop
>>
>> Could you please suggest how to compile this program? Thanks a lot.
>>
>> Shu
>>
>>
>> ====PutMerge.java=========
>> import java.io.IOException;
>> import org.apache.hadoop.conf.Configuration;
>> import org.apache.hadoop.fs.FSDataInputStream;
>> import org.apache.hadoop.fs.FSDataOutputStream;
>> import org.apache.hadoop.fs.FileStatus;
>> import org.apache.hadoop.fs.FileSystem;
>> import org.apache.hadoop.fs.Path;
>> public class PutMerge {
>>
>> public static void main(String[] args) throws IOException {
>> Configuration conf = new Configuration();
>> FileSystem hdfs = FileSystem.get(conf);
>> FileSystem local = FileSystem.getLocal(conf);
>>
>> Path inputDir = new Path(args[0]);
>> Path hdfsFile = new Path(args[1]);
>>
>> try {
>> FileStatus[] inputFiles = local.listStatus(inputDir);
>> FSDataOutputStream out = hdfs.create(hdfsFile);
>>
>> for (int i=0; i<inputFiles.length; i++) {
>> System.out.println(inputFiles[i].getPath().getName());
>> FSDataInputStream in = local.open(inputFiles[i].getPath());
>> byte buffer[] = new byte[256];
>> int bytesRead = 0;
>> while( (bytesRead = in.read(buffer)) > 0) {
>> out.write(buffer, 0, bytesRead);
>> }
>> in.close();
>> }
>> out.close();
>> } catch (IOException e) {
>> e.printStackTrace();
>
>> }
>> }
>> }
>> =============
>>
>
>
>
> --
> Harsh J
>
>
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thanks a lot! I could compile with the added classpath:
$javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
The above created PutMerge.class file.
Now I try to run:
$java PutMerge
Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
Caused by: java.lang.ClassNotFoundException: PutMerge
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
I get the above error.
I tried:
$set CLASSPATH=/usr/lib/hadoop/bin/hadoop
$java PutMerge
I still get the error.
On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thanks a lot! I could compile with the added classpath:
$javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
The above created PutMerge.class file.
Now I try to run:
$java PutMerge
Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
Caused by: java.lang.ClassNotFoundException: PutMerge
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
I get the above error.
I tried:
$set CLASSPATH=/usr/lib/hadoop/bin/hadoop
$java PutMerge
I still get the error.
On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thanks a lot! I could compile with the added classpath:
$javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
The above created PutMerge.class file.
Now I try to run:
$java PutMerge
Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
Caused by: java.lang.ClassNotFoundException: PutMerge
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
I get the above error.
I tried:
$set CLASSPATH=/usr/lib/hadoop/bin/hadoop
$java PutMerge
I still get the error.
On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by R J <rj...@yahoo.com>.
Thanks a lot! I could compile with the added classpath:
$javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
The above created PutMerge.class file.
Now I try to run:
$java PutMerge
Exception in thread "main" java.lang.NoClassDefFoundError: PutMerge
Caused by: java.lang.ClassNotFoundException: PutMerge
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: PutMerge. Program will exit.
I get the above error.
I tried:
$set CLASSPATH=/usr/lib/hadoop/bin/hadoop
$java PutMerge
I still get the error.
On Sunday, July 27, 2014 10:16 PM, Harsh J <ha...@cloudera.com> wrote:
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
I can probably help you out with that. I don¹t want to sound patronising
though. What is your IDE and have you included the hadoop libraries in
your jar ?
Regards,
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: R J <rj...@yahoo.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 01:46
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Cannot compaile a basic PutMerge.java program
Hi All,
I am new to programming on hadoop. I tried to compile the following
program (example program from a hadoop book) on my linix server where I
have Haddop installed:
I get the errors:
$javac PutMerge.java
PutMerge.java:2: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
PutMerge.java:3: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataInputStream;
^
PutMerge.java:4: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataOutputStream;
^
PutMerge.java:5: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileStatus;
^
PutMerge.java:6: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileSystem;
^
PutMerge.java:7: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
I have $HADOOP_HOME set u:
$echo $HADOOP_HOME
/usr/lib/hadoop
Could you please suggest how to compile this program? Thanks a lot.
Shu
====PutMerge.java=========
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class PutMerge {
public static void main(String[] args) throws IOException {
Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.get(conf);
FileSystem local = FileSystem.getLocal(conf);
Path inputDir = new Path(args[0]);
Path hdfsFile = new Path(args[1]);
try {
FileStatus[] inputFiles =
local.listStatus(inputDir);
FSDataOutputStream out = hdfs.create(hdfsFile);
for (int i=0; i<inputFiles.length; i++) {
System.out.println(inputFiles[i].getPath().getName());
FSDataInputStream in = local.open(inputFiles[i].getPath());
byte buffer[] = new byte[256];
int bytesRead = 0;
while( (bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
in.close();
}
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
=============
Re: Cannot compaile a basic PutMerge.java program
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
I can probably help you out with that. I don¹t want to sound patronising
though. What is your IDE and have you included the hadoop libraries in
your jar ?
Regards,
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: R J <rj...@yahoo.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 01:46
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Cannot compaile a basic PutMerge.java program
Hi All,
I am new to programming on hadoop. I tried to compile the following
program (example program from a hadoop book) on my linix server where I
have Haddop installed:
I get the errors:
$javac PutMerge.java
PutMerge.java:2: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
PutMerge.java:3: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataInputStream;
^
PutMerge.java:4: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataOutputStream;
^
PutMerge.java:5: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileStatus;
^
PutMerge.java:6: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileSystem;
^
PutMerge.java:7: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
I have $HADOOP_HOME set u:
$echo $HADOOP_HOME
/usr/lib/hadoop
Could you please suggest how to compile this program? Thanks a lot.
Shu
====PutMerge.java=========
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class PutMerge {
public static void main(String[] args) throws IOException {
Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.get(conf);
FileSystem local = FileSystem.getLocal(conf);
Path inputDir = new Path(args[0]);
Path hdfsFile = new Path(args[1]);
try {
FileStatus[] inputFiles =
local.listStatus(inputDir);
FSDataOutputStream out = hdfs.create(hdfsFile);
for (int i=0; i<inputFiles.length; i++) {
System.out.println(inputFiles[i].getPath().getName());
FSDataInputStream in = local.open(inputFiles[i].getPath());
byte buffer[] = new byte[256];
int bytesRead = 0;
while( (bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
in.close();
}
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
=============
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Harsh J <ha...@cloudera.com>.
The javac program can only find import dependencies referenced in a
program if it is also supplied on the javac classpath. Setting
HADOOP_HOME alone will not magically do this. Have you set an
appropriate classpath?
Try as below, perhaps:
javac -cp $($HADOOP_HOME/bin/hadoop classpath) PutMerge.java
Alternatively, consider using a modern build helper tool such as
Apache Maven for writing java applications, they make your work
easier.
On Mon, Jul 28, 2014 at 6:16 AM, R J <rj...@yahoo.com> wrote:
> Hi All,
>
> I am new to programming on hadoop. I tried to compile the following program
> (example program from a hadoop book) on my linix server where I have Haddop
> installed:
> I get the errors:
> $javac PutMerge.java
> PutMerge.java:2: package org.apache.hadoop.conf does not exist
> import org.apache.hadoop.conf.Configuration;
> ^
> PutMerge.java:3: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataInputStream;
> ^
> PutMerge.java:4: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FSDataOutputStream;
> ^
> PutMerge.java:5: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileStatus;
> ^
> PutMerge.java:6: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.FileSystem;
> ^
> PutMerge.java:7: package org.apache.hadoop.fs does not exist
> import org.apache.hadoop.fs.Path;
>
> I have $HADOOP_HOME set u:
> $echo $HADOOP_HOME
> /usr/lib/hadoop
>
> Could you please suggest how to compile this program? Thanks a lot.
>
> Shu
>
>
> ====PutMerge.java=========
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataInputStream;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> public class PutMerge {
>
> public static void main(String[] args) throws IOException {
> Configuration conf = new Configuration();
> FileSystem hdfs = FileSystem.get(conf);
> FileSystem local = FileSystem.getLocal(conf);
>
> Path inputDir = new Path(args[0]);
> Path hdfsFile = new Path(args[1]);
>
> try {
> FileStatus[] inputFiles = local.listStatus(inputDir);
> FSDataOutputStream out = hdfs.create(hdfsFile);
>
> for (int i=0; i<inputFiles.length; i++) {
> System.out.println(inputFiles[i].getPath().getName());
> FSDataInputStream in = local.open(inputFiles[i].getPath());
> byte buffer[] = new byte[256];
> int bytesRead = 0;
> while( (bytesRead = in.read(buffer)) > 0) {
> out.write(buffer, 0, bytesRead);
> }
> in.close();
> }
> out.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> =============
>
--
Harsh J
Re: Cannot compaile a basic PutMerge.java program
Posted by Chris MacKenzie <st...@chrismackenziephotography.co.uk>.
Hi,
I can probably help you out with that. I don¹t want to sound patronising
though. What is your IDE and have you included the hadoop libraries in
your jar ?
Regards,
Regards,
Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>
From: R J <rj...@yahoo.com>
Reply-To: <us...@hadoop.apache.org>
Date: Monday, 28 July 2014 01:46
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Cannot compaile a basic PutMerge.java program
Hi All,
I am new to programming on hadoop. I tried to compile the following
program (example program from a hadoop book) on my linix server where I
have Haddop installed:
I get the errors:
$javac PutMerge.java
PutMerge.java:2: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
^
PutMerge.java:3: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataInputStream;
^
PutMerge.java:4: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FSDataOutputStream;
^
PutMerge.java:5: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileStatus;
^
PutMerge.java:6: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.FileSystem;
^
PutMerge.java:7: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
I have $HADOOP_HOME set u:
$echo $HADOOP_HOME
/usr/lib/hadoop
Could you please suggest how to compile this program? Thanks a lot.
Shu
====PutMerge.java=========
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class PutMerge {
public static void main(String[] args) throws IOException {
Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.get(conf);
FileSystem local = FileSystem.getLocal(conf);
Path inputDir = new Path(args[0]);
Path hdfsFile = new Path(args[1]);
try {
FileStatus[] inputFiles =
local.listStatus(inputDir);
FSDataOutputStream out = hdfs.create(hdfsFile);
for (int i=0; i<inputFiles.length; i++) {
System.out.println(inputFiles[i].getPath().getName());
FSDataInputStream in = local.open(inputFiles[i].getPath());
byte buffer[] = new byte[256];
int bytesRead = 0;
while( (bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
in.close();
}
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
=============