You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Manoj Babu <ma...@gmail.com> on 2012/08/23 17:36:10 UTC
Reg: when failures on writing to DB from map\reduce
Hi All,
In Sqoop:
When exporting from HDFS to DB, If an export map task fails due to these or
other reasons, it will cause the export job to fail. The results of a
failed export are undefined. Each export map task operates in a separate
transaction. Furthermore, individual map tasks commit their current
transaction periodically. If a task fails, the current transaction will be
rolled back. Any previously-committed transactions will remain durable in
the database, leading to a partially-complete export.
When using DBOutputFormat in custom map\reduce program:
When emitting from multiple mappers or reducers and if any one of the
mapper or reducer got failure only the failed transaction will be rolled
back Any previously-committed transactions will remain durable in the
database, leading to a partially-complete export.
Is there any way to commit at last to avoid partially-complete?
Cheers!
Manoj.
Re: Reg: when failures on writing to DB from map\reduce
Posted by Alberto Andreotti <al...@gmail.com>.
unsubscribe
On 23 August 2012 12:36, Manoj Babu <ma...@gmail.com> wrote:
>
> Hi All,
>
> In Sqoop:
> When exporting from HDFS to DB, If an export map task fails due to these
> or other reasons, it will cause the export job to fail. The results of a
> failed export are undefined. Each export map task operates in a separate
> transaction. Furthermore, individual map tasks commit their current
> transaction periodically. If a task fails, the current transaction will be
> rolled back. Any previously-committed transactions will remain durable in
> the database, leading to a partially-complete export.
>
> When using DBOutputFormat in custom map\reduce program:
> When emitting from multiple mappers or reducers and if any one of the
> mapper or reducer got failure only the failed transaction will be rolled
> back Any previously-committed transactions will remain durable in the
> database, leading to a partially-complete export.
>
> Is there any way to commit at last to avoid partially-complete?
>
> Cheers!
> Manoj.
>
>
>
--
José Pablo Alberto Andreotti.
Tel: 54 351 4730292
Móvil: 54351156526363.
MSN: albertoandreotti@gmail.com
Skype: andreottialberto
Re: Reg: when failures on writing to DB from map\reduce
Posted by Alberto Andreotti <al...@gmail.com>.
unsubscribe
On 23 August 2012 12:36, Manoj Babu <ma...@gmail.com> wrote:
>
> Hi All,
>
> In Sqoop:
> When exporting from HDFS to DB, If an export map task fails due to these
> or other reasons, it will cause the export job to fail. The results of a
> failed export are undefined. Each export map task operates in a separate
> transaction. Furthermore, individual map tasks commit their current
> transaction periodically. If a task fails, the current transaction will be
> rolled back. Any previously-committed transactions will remain durable in
> the database, leading to a partially-complete export.
>
> When using DBOutputFormat in custom map\reduce program:
> When emitting from multiple mappers or reducers and if any one of the
> mapper or reducer got failure only the failed transaction will be rolled
> back Any previously-committed transactions will remain durable in the
> database, leading to a partially-complete export.
>
> Is there any way to commit at last to avoid partially-complete?
>
> Cheers!
> Manoj.
>
>
>
--
José Pablo Alberto Andreotti.
Tel: 54 351 4730292
Móvil: 54351156526363.
MSN: albertoandreotti@gmail.com
Skype: andreottialberto
Re: Reg: when failures on writing to DB from map\reduce
Posted by Alberto Andreotti <al...@gmail.com>.
unsubscribe
On 23 August 2012 12:36, Manoj Babu <ma...@gmail.com> wrote:
>
> Hi All,
>
> In Sqoop:
> When exporting from HDFS to DB, If an export map task fails due to these
> or other reasons, it will cause the export job to fail. The results of a
> failed export are undefined. Each export map task operates in a separate
> transaction. Furthermore, individual map tasks commit their current
> transaction periodically. If a task fails, the current transaction will be
> rolled back. Any previously-committed transactions will remain durable in
> the database, leading to a partially-complete export.
>
> When using DBOutputFormat in custom map\reduce program:
> When emitting from multiple mappers or reducers and if any one of the
> mapper or reducer got failure only the failed transaction will be rolled
> back Any previously-committed transactions will remain durable in the
> database, leading to a partially-complete export.
>
> Is there any way to commit at last to avoid partially-complete?
>
> Cheers!
> Manoj.
>
>
>
--
José Pablo Alberto Andreotti.
Tel: 54 351 4730292
Móvil: 54351156526363.
MSN: albertoandreotti@gmail.com
Skype: andreottialberto
Re: Reg: when failures on writing to DB from map\reduce
Posted by Alberto Andreotti <al...@gmail.com>.
unsubscribe
On 23 August 2012 12:36, Manoj Babu <ma...@gmail.com> wrote:
>
> Hi All,
>
> In Sqoop:
> When exporting from HDFS to DB, If an export map task fails due to these
> or other reasons, it will cause the export job to fail. The results of a
> failed export are undefined. Each export map task operates in a separate
> transaction. Furthermore, individual map tasks commit their current
> transaction periodically. If a task fails, the current transaction will be
> rolled back. Any previously-committed transactions will remain durable in
> the database, leading to a partially-complete export.
>
> When using DBOutputFormat in custom map\reduce program:
> When emitting from multiple mappers or reducers and if any one of the
> mapper or reducer got failure only the failed transaction will be rolled
> back Any previously-committed transactions will remain durable in the
> database, leading to a partially-complete export.
>
> Is there any way to commit at last to avoid partially-complete?
>
> Cheers!
> Manoj.
>
>
>
--
José Pablo Alberto Andreotti.
Tel: 54 351 4730292
Móvil: 54351156526363.
MSN: albertoandreotti@gmail.com
Skype: andreottialberto