You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Bilung Lee (JIRA)" <ji...@apache.org> on 2011/08/12 01:46:28 UTC
[jira] [Created] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Basic export hangs when target database does not support INSERT syntax with multiple rows of values
---------------------------------------------------------------------------------------------------
Key: SQOOP-314
URL: https://issues.apache.org/jira/browse/SQOOP-314
Project: Sqoop
Issue Type: Bug
Reporter: Bilung Lee
Priority: Minor
Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
The configuration variable "sqoop.connection.factories" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087931#comment-13087931 ]
jiraposter@reviews.apache.org commented on SQOOP-314:
-----------------------------------------------------
bq. On 2011-08-19 18:02:13, Arvind Prabhakar wrote:
bq. > +1
bq. >
bq. > Can you please rebase the patch and attach it to the Jira? Thanks!
Yes, just did.
- Bilung
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/#review1553
-----------------------------------------------------------
On 2011-08-19 00:49:12, Bilung Lee wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1585/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 00:49:12)
bq.
bq.
bq. Review request for Sqoop, Arvind Prabhakar and jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Basic export job will hang when the target database does not support insert syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
bq.
bq. This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
bq.
bq. The fix is to introduce a new "--batch" option to execute underlying statements in batch mode instead of a single multirow insert statement.
bq.
bq.
bq. This addresses bug SQOOP-314.
bq. https://issues.apache.org/jira/browse/SQOOP-314
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6
bq. src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1
bq. src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091
bq. src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c
bq. src/docs/man/sqoop-export.txt 6090ea1
bq. src/docs/user/export.txt 4f87886
bq. src/java/com/cloudera/sqoop/SqoopOptions.java d760d39
bq. src/java/com/cloudera/sqoop/manager/OracleManager.java 6a55312
bq. src/java/com/cloudera/sqoop/manager/SQLServerManager.java e1ce2af
bq. src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41
bq. src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION
bq. src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37
bq.
bq. Diff: https://reviews.apache.org/r/1585/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Bilung
bq.
bq.
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
> Attachments: SQOOP-314.diff
>
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "Arvind Prabhakar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar updated SQOOP-314:
-----------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
Status: Resolved (was: Patch Available)
Patch committed. Thanks Bilung!
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
> Fix For: 1.4.0
>
> Attachments: SQOOP-314.diff
>
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087976#comment-13087976 ]
Hudson commented on SQOOP-314:
------------------------------
Integrated in Sqoop-jdk-1.6 #14 (See [https://builds.apache.org/job/Sqoop-jdk-1.6/14/])
SQOOP-314. Support for batch insert.
(Bilung Lee via Arvind Prabhakar)
arvind : http://svn.apache.org/viewvc/?view=rev&rev=1159773
Files :
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/SQLServerManager.java
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/SqoopOptions.java
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/manager/OracleManager.java
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java
* /incubator/sqoop/trunk/src/docs/user/export.txt
* /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
* /incubator/sqoop/trunk/src/docs/man/sqoop-export.txt
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java
* /incubator/sqoop/trunk/src/java/com/cloudera/sqoop/tool/ExportTool.java
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
> Fix For: 1.4.0
>
> Attachments: SQOOP-314.diff
>
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "Bilung Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bilung Lee updated SQOOP-314:
-----------------------------
Status: Patch Available (was: Open)
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
> Attachments: SQOOP-314.diff
>
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087433#comment-13087433 ]
jiraposter@reviews.apache.org commented on SQOOP-314:
-----------------------------------------------------
bq. On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
bq. > Changes look good Bilung. One high-level suggestion is to modify OracleManager and SQLServerManager to use the new ExportBatchOutputFormat. The classes for OracleExportOutputFormat and SQLSErverExportOutputFormat can then be deprecated.
Done in the new patch. Thanks.
bq. On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
bq. > src/java/com/cloudera/sqoop/SqoopOptions.java, line 1026
bq. > <https://reviews.apache.org/r/1585/diff/1/?file=33455#file33455line1026>
bq. >
bq. > This will trigger a checkstyle warning. Should change the argument name to mode or something else.
Done in the new patch. Thanks.
bq. On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
bq. > src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java, line 41
bq. > <https://reviews.apache.org/r/1585/diff/1/?file=33457#file33457line41>
bq. >
bq. > Longer than 80.
Done in the new patch. Thanks.
bq. On 2011-08-18 22:35:36, Arvind Prabhakar wrote:
bq. > src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java, line 179
bq. > <https://reviews.apache.org/r/1585/diff/1/?file=33456#file33456line179>
bq. >
bq. > Thread.stop() is unsafe operation that can lead to corruption. Would it help if the join call was part of the try block instead of in finally? In that case, throwing the IOException would automatically get rid of the thread.
Done in the new patch. Thanks.
- Bilung
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/#review1531
-----------------------------------------------------------
On 2011-08-19 00:49:12, Bilung Lee wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1585/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 00:49:12)
bq.
bq.
bq. Review request for Sqoop, Arvind Prabhakar and jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Basic export job will hang when the target database does not support insert syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
bq.
bq. This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
bq.
bq. The fix is to introduce a new "--batch" option to execute underlying statements in batch mode instead of a single multirow insert statement.
bq.
bq.
bq. This addresses bug SQOOP-314.
bq. https://issues.apache.org/jira/browse/SQOOP-314
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6
bq. src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1
bq. src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091
bq. src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c
bq. src/docs/man/sqoop-export.txt 6090ea1
bq. src/docs/user/export.txt 4f87886
bq. src/java/com/cloudera/sqoop/SqoopOptions.java d760d39
bq. src/java/com/cloudera/sqoop/manager/OracleManager.java 6a55312
bq. src/java/com/cloudera/sqoop/manager/SQLServerManager.java e1ce2af
bq. src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41
bq. src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION
bq. src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37
bq.
bq. Diff: https://reviews.apache.org/r/1585/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Bilung
bq.
bq.
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "Bilung Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bilung Lee updated SQOOP-314:
-----------------------------
Assignee: Bilung Lee
Priority: Major (was: Minor)
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087143#comment-13087143 ]
jiraposter@reviews.apache.org commented on SQOOP-314:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/
-----------------------------------------------------------
(Updated 2011-08-18 17:46:40.735309)
Review request for Sqoop, Arvind Prabhakar and jmhsieh.
Summary
-------
Basic export job will hang when the target database does not support insert syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
The fix is to introduce a new "--batch" option to execute underlying statements in batch mode instead of a single multirow insert statement.
This addresses bug SQOOP-314.
https://issues.apache.org/jira/browse/SQOOP-314
Diffs
-----
src/docs/man/sqoop-export.txt 6090ea1
src/docs/user/export.txt 4f87886
src/java/com/cloudera/sqoop/SqoopOptions.java d760d39
src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41
src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION
src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37
src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1
src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091
src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c
Diff: https://reviews.apache.org/r/1585/diff
Testing
-------
Thanks,
Bilung
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087357#comment-13087357 ]
jiraposter@reviews.apache.org commented on SQOOP-314:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/#review1531
-----------------------------------------------------------
Changes look good Bilung. One high-level suggestion is to modify OracleManager and SQLServerManager to use the new ExportBatchOutputFormat. The classes for OracleExportOutputFormat and SQLSErverExportOutputFormat can then be deprecated.
src/java/com/cloudera/sqoop/SqoopOptions.java
<https://reviews.apache.org/r/1585/#comment3483>
This will trigger a checkstyle warning. Should change the argument name to mode or something else.
src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java
<https://reviews.apache.org/r/1585/#comment3486>
Thread.stop() is unsafe operation that can lead to corruption. Would it help if the join call was part of the try block instead of in finally? In that case, throwing the IOException would automatically get rid of the thread.
src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java
<https://reviews.apache.org/r/1585/#comment3484>
Longer than 80.
- Arvind
On 2011-08-18 17:46:40, Bilung Lee wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1585/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-18 17:46:40)
bq.
bq.
bq. Review request for Sqoop, Arvind Prabhakar and jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Basic export job will hang when the target database does not support insert syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
bq.
bq. This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
bq.
bq. The fix is to introduce a new "--batch" option to execute underlying statements in batch mode instead of a single multirow insert statement.
bq.
bq.
bq. This addresses bug SQOOP-314.
bq. https://issues.apache.org/jira/browse/SQOOP-314
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/docs/man/sqoop-export.txt 6090ea1
bq. src/docs/user/export.txt 4f87886
bq. src/java/com/cloudera/sqoop/SqoopOptions.java d760d39
bq. src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41
bq. src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION
bq. src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37
bq. src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6
bq. src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1
bq. src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091
bq. src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c
bq.
bq. Diff: https://reviews.apache.org/r/1585/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Bilung
bq.
bq.
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087840#comment-13087840 ]
jiraposter@reviews.apache.org commented on SQOOP-314:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/#review1553
-----------------------------------------------------------
Ship it!
+1
Can you please rebase the patch and attach it to the Jira? Thanks!
- Arvind
On 2011-08-19 00:49:12, Bilung Lee wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1585/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-19 00:49:12)
bq.
bq.
bq. Review request for Sqoop, Arvind Prabhakar and jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Basic export job will hang when the target database does not support insert syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
bq.
bq. This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
bq.
bq. The fix is to introduce a new "--batch" option to execute underlying statements in batch mode instead of a single multirow insert statement.
bq.
bq.
bq. This addresses bug SQOOP-314.
bq. https://issues.apache.org/jira/browse/SQOOP-314
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6
bq. src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1
bq. src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091
bq. src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c
bq. src/docs/man/sqoop-export.txt 6090ea1
bq. src/docs/user/export.txt 4f87886
bq. src/java/com/cloudera/sqoop/SqoopOptions.java d760d39
bq. src/java/com/cloudera/sqoop/manager/OracleManager.java 6a55312
bq. src/java/com/cloudera/sqoop/manager/SQLServerManager.java e1ce2af
bq. src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41
bq. src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION
bq. src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37
bq.
bq. Diff: https://reviews.apache.org/r/1585/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Bilung
bq.
bq.
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "Bilung Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bilung Lee updated SQOOP-314:
-----------------------------
Attachment: SQOOP-314.diff
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
> Attachments: SQOOP-314.diff
>
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087434#comment-13087434 ]
jiraposter@reviews.apache.org commented on SQOOP-314:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1585/
-----------------------------------------------------------
(Updated 2011-08-19 00:49:12.862727)
Review request for Sqoop, Arvind Prabhakar and jmhsieh.
Changes
-------
New patch that incorporates comments.
Summary
-------
Basic export job will hang when the target database does not support insert syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
The fix is to introduce a new "--batch" option to execute underlying statements in batch mode instead of a single multirow insert statement.
This addresses bug SQOOP-314.
https://issues.apache.org/jira/browse/SQOOP-314
Diffs (updated)
-----
src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java d2a6cf6
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java 8f629f1
src/java/com/cloudera/sqoop/tool/ExportTool.java b4b1091
src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java 8687b0c
src/docs/man/sqoop-export.txt 6090ea1
src/docs/user/export.txt 4f87886
src/java/com/cloudera/sqoop/SqoopOptions.java d760d39
src/java/com/cloudera/sqoop/manager/OracleManager.java 6a55312
src/java/com/cloudera/sqoop/manager/SQLServerManager.java e1ce2af
src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java 193cf41
src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java PRE-CREATION
src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java 9799e37
Diff: https://reviews.apache.org/r/1585/diff
Testing
-------
Thanks,
Bilung
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Assignee: Bilung Lee
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-314) Basic export hangs when target
database does not support INSERT syntax with multiple rows of values
Posted by "Bilung Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bilung Lee updated SQOOP-314:
-----------------------------
Description:
Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
was:
Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
The configuration variable "sqoop.connection.factories" can be set to 1 as a workaround for this problem.
> Basic export hangs when target database does not support INSERT syntax with multiple rows of values
> ---------------------------------------------------------------------------------------------------
>
> Key: SQOOP-314
> URL: https://issues.apache.org/jira/browse/SQOOP-314
> Project: Sqoop
> Issue Type: Bug
> Reporter: Bilung Lee
> Priority: Minor
>
> Basic export job will hang when the target database does not support INSERT syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)
> This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.
> The configuration variable "sqoop.export.records.per.statement" can be set to 1 as a workaround for this problem.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira