You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sky USC <sk...@hotmail.com> on 2012/04/07 00:24:42 UTC
Reading properties file from command line + passing parameters from
main to mapper/reducer?
Dear friends I am new to hadoop on aws. I am using AWS Elastic mapreduce. I am trying to convert an old java program to elastic mapreduce. I would appreciate your help. My question is about:
1. How do I pass a "Properties" file which i used to pass via command line paramters - to elastic mapreduce?
2. how do I pass parameters from this properties file from main() to the Mapper and Reducer? Details:
1. properties file:
My program used to read a properties file as follows:
org.apache.commons.configuration.Configuration config = new PropertiesConfiguration("my_app.properties");
How do I read this in Amazon EMR?
I tried launching the program with :
jar file: s3n://my.bucket.name/myjar-job.jar
args: s3n://my.bucket.name/my_app.properties I get an stderr of:
org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source s3n://my.bucket.name/my_app.properties In order to debug, I tried to do the following:
String c = FileUtils.readFileToString(new File(remainingArgs[0]));
I got an exception:
Exception in thread "main" java.io.FileNotFoundException: File 's3n:/my.bucket.name/my_app.propertiess' does not exist
Notice that the file name - even though I entered s3n:// is shown as s3n:/ - could the // be getting dropped? 2. Passing the properties file from my main() to Mapper and Reducer. How do I do that? Or pass other generic parameters?
-Sky
Reading properties file from command line + passing parameters from
main to mapper/reducer?
Posted by Sky USC <sk...@hotmail.com>.
Dear friends
I am new to hadoop on aws. I am using AWS Elastic mapreduce. I am trying to convert an old java program to elastic mapreduce. I would appreciate your help.
My question is about:
1. How do I pass a "Properties" file which i used to pass via command line paramters - to elastic mapreduce?
2. how do I pass parameters from this properties file from main() to the Mapper and Reducer?
Details:
1. properties file:
My program used to read a properties file as follows:
org.apache.commons.configuration.Configuration config = new PropertiesConfiguration("my_app.properties");
How do I read this in Amazon EMR?
I tried launching the program with :
jar file: s3n://my.bucket.name/myjar-job.jar
args: s3n://my.bucket.name/my_app.properties
I get an stderr of:
org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source s3n://my.bucket.name/my_app.properties
In order to debug, I tried to do the following:
String c = FileUtils.readFileToString(new File(remainingArgs[0]));
I got an exception:
Exception in thread "main" java.io.FileNotFoundException: File 's3n:/my.bucket.name/my_app.propertiess' does not exist
Notice that the file name - even though I entered s3n:// is shown as s3n:/ - could the // be getting dropped?
2. Passing the properties file from my main() to Mapper and Reducer. How do I do that? Or pass other generic parameters?
-Sky
Reading properties file from command line + passing parameters from
main to mapper/reducer?
Posted by Sky USC <sk...@hotmail.com>.
Dear friends
I am new to hadoop on aws. I am using AWS Elastic mapreduce. I am trying to convert an old java program to elastic mapreduce. I would appreciate your help.
My question is about:
1. How do I pass a "Properties" file which i used to pass via command line paramters - to elastic mapreduce?
2. how do I pass parameters from this properties file from main() to the Mapper and Reducer?
Details:
1. properties file:
My program used to read a properties file as follows:
org.apache.commons.configuration.Configuration config = new PropertiesConfiguration("my_app.properties");
How do I read this in Amazon EMR?
I tried launching the program with :
jar file: s3n://my.bucket.name/myjar-job.jar
args: s3n://my.bucket.name/my_app.properties
I get an stderr of:
org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source s3n://my.bucket.name/my_app.properties
In order to debug, I tried to do the following:
String c = FileUtils.readFileToString(new File(remainingArgs[0]));
I got an exception:
Exception in thread "main" java.io.FileNotFoundException: File 's3n:/my.bucket.name/my_app.propertiess' does not exist
Notice that the file name - even though I entered s3n:// is shown as s3n:/ - could the // be getting dropped?
2. Passing the properties file from my main() to Mapper and Reducer. How do I do that? Or pass other generic parameters?
-Sky