You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2009/08/15 01:11:15 UTC
[jira] Created: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Make DBRecordReader execute queries lazily
------------------------------------------
Key: MAPREDUCE-875
URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Attachments: MAPREDUCE-875.patch
DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated MAPREDUCE-875:
------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I have committed this. Thanks Aaron!
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-875.2.patch, MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-875:
------------------------------------
Attachment: MAPREDUCE-875.patch
Trivial patch that makes the query execute only when nextKeyValue() is actually run for the first time. This is inside the control of the MapRunnable.
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-875) Make DBRecordReader execute
queries lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744816#action_12744816 ]
Aaron Kimball commented on MAPREDUCE-875:
-----------------------------------------
The failing Sqoop tests claim that it's failing because it can't find Avro. Not sure why this is happening -- Sqoop doesn't make use of Avro anywhere. Recycling patch status in case this was transient. If not, do I have to put some more random libraries in ivy.xml?
According to {{git-blame}}, this was added to the root ivy.xml earlier that day:
{code}
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 276) <dependency org="org.apache.hadoop"
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 277) name="avro"
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 278) rev="1.0.0"
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 279) conf="common->default"/>
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 280) <dependency org="org.codehaus.jackso
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 281) name="jackson-mapper-asl"
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 282) rev="1.0.1"
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 283) conf="common->default"/>
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 284) <dependency org="com.thoughtworks.pa
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 285) name="paranamer"
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 286) rev="1.5"
9e58f6fc (Sharad Agarwal 2009-08-14 05:10:40 +0000 287) conf="common->default"/>
{code}
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-875) Make DBRecordReader execute
queries lazily
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747263#action_12747263 ]
Enis Soztutar commented on MAPREDUCE-875:
-----------------------------------------
+1
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.2.patch, MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-875) Make DBRecordReader execute
queries lazily
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747314#action_12747314 ]
Hadoop QA commented on MAPREDUCE-875:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12416939/MAPREDUCE-875.2.patch
against trunk revision 807165.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/513/console
This message is automatically generated.
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.2.patch, MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-875:
------------------------------------
Attachment: MAPREDUCE-875.2.patch
Attaching new patch after resync'ing with trunk. Just realized that avro was already added to sqoop's ivy.xml
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.2.patch, MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-875:
------------------------------------
Status: Open (was: Patch Available)
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-875:
------------------------------------
Status: Patch Available (was: Open)
Recycling patch again.. seems to have been dropped from the queue.
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.2.patch, MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-875) Make DBRecordReader execute
queries lazily
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743529#action_12743529 ]
Hadoop QA commented on MAPREDUCE-875:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12416623/MAPREDUCE-875.patch
against trunk revision 804284.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/477/console
This message is automatically generated.
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-875:
------------------------------------
Status: Patch Available (was: Open)
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-875:
------------------------------------
Status: Open (was: Patch Available)
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.2.patch, MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-875) Make DBRecordReader execute queries
lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-875:
------------------------------------
Status: Patch Available (was: Open)
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-875) Make DBRecordReader execute
queries lazily
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747511#action_12747511 ]
Aaron Kimball commented on MAPREDUCE-875:
-----------------------------------------
No new tests because it's a very small code change; existing Sqoop unit tests cover this and pass.
Core test failure is unrelated.
> Make DBRecordReader execute queries lazily
> ------------------------------------------
>
> Key: MAPREDUCE-875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-875
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-875.2.patch, MAPREDUCE-875.patch
>
>
> DBInputFormat's DBRecordReader executes the user's SQL query in the constructor. If the query is long-running, this can cause task timeout. The user is unable to spawn a background thread (e.g., in a MapRunnable) to inform Hadoop of on-going progress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.