You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Masatake Iwasaki <iw...@oss.nttdata.co.jp> on 2014/10/25 04:09:49 UTC
Review Request 27187: SQOOP-1622: Copying from staging table should
be in single transaction for pg_bulkload connector
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27187/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-1622
https://issues.apache.org/jira/browse/SQOOP-1622
Repository: sqoop-trunk
Description
-------
Each reduce tasks should do copying from staging table in single transaction.
Diffs
-----
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 32fe077
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportMapper.java 16af84c
src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java fc5fd6d
Diff: https://reviews.apache.org/r/27187/diff/
Testing
-------
PGBulkloadExportReducer#reduce may be called per map task because the map output key is unique ID of map task. This patch fixed the map output key to make reduce tasks do work in single transaction.
Thanks,
Masatake Iwasaki
Re: Review Request 27187: SQOOP-1622: Copying from staging table
should be in single transaction for pg_bulkload connector
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27187/#review61558
-----------------------------------------------------------
Ship it!
https://warosu.org/data/lit/img/0051/88/1406367367200.gif
- Abraham Elmahrek
On Nov. 3, 2014, 6:08 p.m., Masatake Iwasaki wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27187/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2014, 6:08 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1622
> https://issues.apache.org/jira/browse/SQOOP-1622
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> PGBulkloadExportReducer#reduce may be called per map task because the map output key is unique ID of map task. This patch fixed the map output key to make reduce tasks do work in single transaction.
> This patch include trivial fixes in test code.
> Crutial part this patch is just moving the call conn.commit() in PGBulkloadExportReducer.
>
>
> Diffs
> -----
>
> src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 32fe077
> src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportMapper.java 16af84c
> src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportReducer.java 3dc05a7
> src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java fc5fd6d
>
> Diff: https://reviews.apache.org/r/27187/diff/
>
>
> Testing
> -------
>
> PGBulkloadManagerManualTest passed.
>
> I removed PGBulkloadManagerManualTest#multiFileTestWithGenericArgs because TestExport#multiFileTest can handle generic args as far as args are given in the form as not "-Dvar=val" but "-D" and "var=val".
>
>
> Thanks,
>
> Masatake Iwasaki
>
>
Re: Review Request 27187: SQOOP-1622: Copying from staging table
should be in single transaction for pg_bulkload connector
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27187/
-----------------------------------------------------------
(Updated Nov. 3, 2014, 6:08 p.m.)
Review request for Sqoop.
Bugs: SQOOP-1622
https://issues.apache.org/jira/browse/SQOOP-1622
Repository: sqoop-trunk
Description (updated)
-------
PGBulkloadExportReducer#reduce may be called per map task because the map output key is unique ID of map task. This patch fixed the map output key to make reduce tasks do work in single transaction.
This patch include trivial fixes in test code.
Crutial part this patch is just moving the call conn.commit() in PGBulkloadExportReducer.
Diffs
-----
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 32fe077
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportMapper.java 16af84c
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportReducer.java 3dc05a7
src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java fc5fd6d
Diff: https://reviews.apache.org/r/27187/diff/
Testing (updated)
-------
PGBulkloadManagerManualTest passed.
I removed PGBulkloadManagerManualTest#multiFileTestWithGenericArgs because TestExport#multiFileTest can handle generic args as far as args are given in the form as not "-Dvar=val" but "-D" and "var=val".
Thanks,
Masatake Iwasaki
Re: Review Request 27187: SQOOP-1622: Copying from staging table
should be in single transaction for pg_bulkload connector
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27187/
-----------------------------------------------------------
(Updated Oct. 26, 2014, 9:09 p.m.)
Review request for Sqoop.
Changes
-------
I updated the patch to fix in more straight forward way.
Bugs: SQOOP-1622
https://issues.apache.org/jira/browse/SQOOP-1622
Repository: sqoop-trunk
Description
-------
PGBulkloadExportReducer#reduce may be called per map task because the map output key is unique ID of map task. This patch fixed the map output key to make reduce tasks do work in single transaction.
This patch include trivial fixes in test code.
Diffs (updated)
-----
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 32fe077
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportMapper.java 16af84c
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportReducer.java 3dc05a7
src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java fc5fd6d
Diff: https://reviews.apache.org/r/27187/diff/
Testing
-------
PGBulkloadManagerManualTest passed.
Thanks,
Masatake Iwasaki
Re: Review Request 27187: SQOOP-1622: Copying from staging table
should be in single transaction for pg_bulkload connector
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27187/
-----------------------------------------------------------
(Updated Oct. 26, 2014, 9:08 p.m.)
Review request for Sqoop.
Bugs: SQOOP-1622
https://issues.apache.org/jira/browse/SQOOP-1622
Repository: sqoop-trunk
Description (updated)
-------
PGBulkloadExportReducer#reduce may be called per map task because the map output key is unique ID of map task. This patch fixed the map output key to make reduce tasks do work in single transaction.
This patch include trivial fixes in test code.
Diffs
-----
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 32fe077
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportMapper.java 16af84c
src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java fc5fd6d
Diff: https://reviews.apache.org/r/27187/diff/
Testing (updated)
-------
PGBulkloadManagerManualTest passed.
Thanks,
Masatake Iwasaki
Re: Review Request 27187: SQOOP-1622: Copying from staging table
should be in single transaction for pg_bulkload connector
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27187/
-----------------------------------------------------------
(Updated Oct. 25, 2014, 4:42 p.m.)
Review request for Sqoop.
Bugs: SQOOP-1622
https://issues.apache.org/jira/browse/SQOOP-1622
Repository: sqoop-trunk
Description
-------
Each reduce tasks should do copying from staging table in single transaction.
Diffs
-----
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportJob.java 32fe077
src/java/org/apache/sqoop/mapreduce/postgresql/PGBulkloadExportMapper.java 16af84c
src/test/com/cloudera/sqoop/manager/PGBulkloadManagerManualTest.java fc5fd6d
Diff: https://reviews.apache.org/r/27187/diff/
Testing (updated)
-------
PGBulkloadExportReducer#reduce may be called per map task because the map output key is unique ID of map task. This patch fixed the map output key to make reduce tasks do work in single transaction.
This patch include trivial fixes in test code.
Thanks,
Masatake Iwasaki