You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Li Li <fa...@gmail.com> on 2014/03/28 09:33:38 UTC
when it's safe to read map-reduce result?
I have a program that do some map-reduce job and then read the result
of the job.
I learned that hdfs is not strong consistent. when it's safe to read the result?
as long as output/_SUCCESS exist?
Re: when it's safe to read map-reduce result?
Posted by Hardik Pandya <sm...@gmail.com>.
if the job complets without any failures exitCode should be 0 and safe
to read the result
public class MyApp extends Configured implements Tool {
public int run(String[] args) throws Exception {
// Configuration processed by ToolRunner
Configuration conf = getConf();
// Create a JobConf using the processed conf
JobConf job = new JobConf(conf, MyApp.class);
// Process custom command-line options
Path in = new Path(args[1]);
Path out = new Path(args[2]);
// Specify various job-specific parameters
job.setJobName("my-app");
job.setInputPath(in);
job.setOutputPath(out);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
// Submit the job, then poll for progress until the job is complete
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
// Let ToolRunner handle generic command-line options
int res = ToolRunner.run(new Configuration(), new MyApp(), args);
System.exit(res);
}
}
On Fri, Mar 28, 2014 at 4:41 AM, Li Li <fa...@gmail.com> wrote:
> thanks. is the following codes safe?
> int exitCode=ToolRunner.run()
> if(exitCode==0){
> //safe to read result
> }
>
> On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com>
> wrote:
> > _SUCCES implies that the job has succesfully terminated, so this seems
> like
> > a reasonable criterion.
> >
> > Regards, Dieter
> >
> >
> > 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> >
> >> I have a program that do some map-reduce job and then read the result
> >> of the job.
> >> I learned that hdfs is not strong consistent. when it's safe to read the
> >> result?
> >> as long as output/_SUCCESS exist?
> >
> >
>
Re: when it's safe to read map-reduce result?
Posted by Hardik Pandya <sm...@gmail.com>.
if the job complets without any failures exitCode should be 0 and safe
to read the result
public class MyApp extends Configured implements Tool {
public int run(String[] args) throws Exception {
// Configuration processed by ToolRunner
Configuration conf = getConf();
// Create a JobConf using the processed conf
JobConf job = new JobConf(conf, MyApp.class);
// Process custom command-line options
Path in = new Path(args[1]);
Path out = new Path(args[2]);
// Specify various job-specific parameters
job.setJobName("my-app");
job.setInputPath(in);
job.setOutputPath(out);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
// Submit the job, then poll for progress until the job is complete
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
// Let ToolRunner handle generic command-line options
int res = ToolRunner.run(new Configuration(), new MyApp(), args);
System.exit(res);
}
}
On Fri, Mar 28, 2014 at 4:41 AM, Li Li <fa...@gmail.com> wrote:
> thanks. is the following codes safe?
> int exitCode=ToolRunner.run()
> if(exitCode==0){
> //safe to read result
> }
>
> On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com>
> wrote:
> > _SUCCES implies that the job has succesfully terminated, so this seems
> like
> > a reasonable criterion.
> >
> > Regards, Dieter
> >
> >
> > 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> >
> >> I have a program that do some map-reduce job and then read the result
> >> of the job.
> >> I learned that hdfs is not strong consistent. when it's safe to read the
> >> result?
> >> as long as output/_SUCCESS exist?
> >
> >
>
Re: when it's safe to read map-reduce result?
Posted by Hardik Pandya <sm...@gmail.com>.
if the job complets without any failures exitCode should be 0 and safe
to read the result
public class MyApp extends Configured implements Tool {
public int run(String[] args) throws Exception {
// Configuration processed by ToolRunner
Configuration conf = getConf();
// Create a JobConf using the processed conf
JobConf job = new JobConf(conf, MyApp.class);
// Process custom command-line options
Path in = new Path(args[1]);
Path out = new Path(args[2]);
// Specify various job-specific parameters
job.setJobName("my-app");
job.setInputPath(in);
job.setOutputPath(out);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
// Submit the job, then poll for progress until the job is complete
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
// Let ToolRunner handle generic command-line options
int res = ToolRunner.run(new Configuration(), new MyApp(), args);
System.exit(res);
}
}
On Fri, Mar 28, 2014 at 4:41 AM, Li Li <fa...@gmail.com> wrote:
> thanks. is the following codes safe?
> int exitCode=ToolRunner.run()
> if(exitCode==0){
> //safe to read result
> }
>
> On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com>
> wrote:
> > _SUCCES implies that the job has succesfully terminated, so this seems
> like
> > a reasonable criterion.
> >
> > Regards, Dieter
> >
> >
> > 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> >
> >> I have a program that do some map-reduce job and then read the result
> >> of the job.
> >> I learned that hdfs is not strong consistent. when it's safe to read the
> >> result?
> >> as long as output/_SUCCESS exist?
> >
> >
>
Re: when it's safe to read map-reduce result?
Posted by Hardik Pandya <sm...@gmail.com>.
if the job complets without any failures exitCode should be 0 and safe
to read the result
public class MyApp extends Configured implements Tool {
public int run(String[] args) throws Exception {
// Configuration processed by ToolRunner
Configuration conf = getConf();
// Create a JobConf using the processed conf
JobConf job = new JobConf(conf, MyApp.class);
// Process custom command-line options
Path in = new Path(args[1]);
Path out = new Path(args[2]);
// Specify various job-specific parameters
job.setJobName("my-app");
job.setInputPath(in);
job.setOutputPath(out);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
// Submit the job, then poll for progress until the job is complete
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception {
// Let ToolRunner handle generic command-line options
int res = ToolRunner.run(new Configuration(), new MyApp(), args);
System.exit(res);
}
}
On Fri, Mar 28, 2014 at 4:41 AM, Li Li <fa...@gmail.com> wrote:
> thanks. is the following codes safe?
> int exitCode=ToolRunner.run()
> if(exitCode==0){
> //safe to read result
> }
>
> On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com>
> wrote:
> > _SUCCES implies that the job has succesfully terminated, so this seems
> like
> > a reasonable criterion.
> >
> > Regards, Dieter
> >
> >
> > 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> >
> >> I have a program that do some map-reduce job and then read the result
> >> of the job.
> >> I learned that hdfs is not strong consistent. when it's safe to read the
> >> result?
> >> as long as output/_SUCCESS exist?
> >
> >
>
Re: when it's safe to read map-reduce result?
Posted by Li Li <fa...@gmail.com>.
thanks. is the following codes safe?
int exitCode=ToolRunner.run()
if(exitCode==0){
//safe to read result
}
On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com> wrote:
> _SUCCES implies that the job has succesfully terminated, so this seems like
> a reasonable criterion.
>
> Regards, Dieter
>
>
> 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
>
>> I have a program that do some map-reduce job and then read the result
>> of the job.
>> I learned that hdfs is not strong consistent. when it's safe to read the
>> result?
>> as long as output/_SUCCESS exist?
>
>
Re: when it's safe to read map-reduce result?
Posted by Li Li <fa...@gmail.com>.
thanks. is the following codes safe?
int exitCode=ToolRunner.run()
if(exitCode==0){
//safe to read result
}
On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com> wrote:
> _SUCCES implies that the job has succesfully terminated, so this seems like
> a reasonable criterion.
>
> Regards, Dieter
>
>
> 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
>
>> I have a program that do some map-reduce job and then read the result
>> of the job.
>> I learned that hdfs is not strong consistent. when it's safe to read the
>> result?
>> as long as output/_SUCCESS exist?
>
>
Re: when it's safe to read map-reduce result?
Posted by Li Li <fa...@gmail.com>.
thanks. is the following codes safe?
int exitCode=ToolRunner.run()
if(exitCode==0){
//safe to read result
}
On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com> wrote:
> _SUCCES implies that the job has succesfully terminated, so this seems like
> a reasonable criterion.
>
> Regards, Dieter
>
>
> 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
>
>> I have a program that do some map-reduce job and then read the result
>> of the job.
>> I learned that hdfs is not strong consistent. when it's safe to read the
>> result?
>> as long as output/_SUCCESS exist?
>
>
Re: when it's safe to read map-reduce result?
Posted by Li Li <fa...@gmail.com>.
thanks. is the following codes safe?
int exitCode=ToolRunner.run()
if(exitCode==0){
//safe to read result
}
On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <dr...@gmail.com> wrote:
> _SUCCES implies that the job has succesfully terminated, so this seems like
> a reasonable criterion.
>
> Regards, Dieter
>
>
> 2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
>
>> I have a program that do some map-reduce job and then read the result
>> of the job.
>> I learned that hdfs is not strong consistent. when it's safe to read the
>> result?
>> as long as output/_SUCCESS exist?
>
>
Re: when it's safe to read map-reduce result?
Posted by Dieter De Witte <dr...@gmail.com>.
_SUCCES implies that the job has succesfully terminated, so this seems like
a reasonable criterion.
Regards, Dieter
2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> I have a program that do some map-reduce job and then read the result
> of the job.
> I learned that hdfs is not strong consistent. when it's safe to read the
> result?
> as long as output/_SUCCESS exist?
>
Re: when it's safe to read map-reduce result?
Posted by Dieter De Witte <dr...@gmail.com>.
_SUCCES implies that the job has succesfully terminated, so this seems like
a reasonable criterion.
Regards, Dieter
2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> I have a program that do some map-reduce job and then read the result
> of the job.
> I learned that hdfs is not strong consistent. when it's safe to read the
> result?
> as long as output/_SUCCESS exist?
>
Re: when it's safe to read map-reduce result?
Posted by Dieter De Witte <dr...@gmail.com>.
_SUCCES implies that the job has succesfully terminated, so this seems like
a reasonable criterion.
Regards, Dieter
2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> I have a program that do some map-reduce job and then read the result
> of the job.
> I learned that hdfs is not strong consistent. when it's safe to read the
> result?
> as long as output/_SUCCESS exist?
>
Re: when it's safe to read map-reduce result?
Posted by Dieter De Witte <dr...@gmail.com>.
_SUCCES implies that the job has succesfully terminated, so this seems like
a reasonable criterion.
Regards, Dieter
2014-03-28 9:33 GMT+01:00 Li Li <fa...@gmail.com>:
> I have a program that do some map-reduce job and then read the result
> of the job.
> I learned that hdfs is not strong consistent. when it's safe to read the
> result?
> as long as output/_SUCCESS exist?
>