You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org> on 2011/06/22 16:01:48 UTC
[jira] [Created] (MAPREDUCE-2614) Allow append arbitrary text at
the end of generated query in DBOutputFormat class
Allow append arbitrary text at the end of generated query in DBOutputFormat class
---------------------------------------------------------------------------------
Key: MAPREDUCE-2614
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2614
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Jarek Jarcec Cecho
Priority: Minor
It would be wonderful if DBOutputFormat class allow addition of arbitrary text at the end of generated query. This feature can be useful for example in case of MySQL database to specify "ON DUPLICATE KEY UPDATE ..." part of the query.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2614) Allow append arbitrary text at
the end of generated query in DBOutputFormat class
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated MAPREDUCE-2614:
------------------------------------------
Attachment: MAPREDUCE-2614.patch
> Allow append arbitrary text at the end of generated query in DBOutputFormat class
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2614
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2614
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Attachments: MAPREDUCE-2614.patch
>
>
> It would be wonderful if DBOutputFormat class allow addition of arbitrary text at the end of generated query. This feature can be useful for example in case of MySQL database to specify "ON DUPLICATE KEY UPDATE ..." part of the query.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2614) Allow append arbitrary text at
the end of generated query in DBOutputFormat class
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053931#comment-13053931 ]
Josh Wills commented on MAPREDUCE-2614:
---------------------------------------
I agree that the ability to use ON DUPLICATE KEY UPDATE syntax for outputs that write to MySQL would be useful-- as would the ability to specify INSERT IGNORE and turn speculative reduce execution back on if you knew that your output table had unique keys. But I also imagine that Oracle users would like to be able to use INSERT WHEN syntax, or use Oracle's TO_DATE function on some inserts. So I think that the right way to solve this problem is to decouple the query building logic from DBRecordWriter- create a generic default impl that does the same thing as the current code, and then create MySQL/Oracle/etc. specific implementations of the query building interface.
> Allow append arbitrary text at the end of generated query in DBOutputFormat class
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2614
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2614
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Attachments: MAPREDUCE-2614.patch
>
>
> It would be wonderful if DBOutputFormat class allow addition of arbitrary text at the end of generated query. This feature can be useful for example in case of MySQL database to specify "ON DUPLICATE KEY UPDATE ..." part of the query.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2614) Allow append arbitrary text at
the end of generated query in DBOutputFormat class
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated MAPREDUCE-2614:
------------------------------------------
Release Note: Add new configuration property for DBConfiguration and it's propagation to DBOutputFormat
Status: Patch Available (was: Open)
I've add new configuration property to DBConfiguration called OUTPUT_QUERY_APPEND_PROPERTY. If this property is not empty than it's content is appended at the end of query generation in class DBOutputFormat (constructQuery method). I've also fixed both test around this two classes.
> Allow append arbitrary text at the end of generated query in DBOutputFormat class
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2614
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2614
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
>
> It would be wonderful if DBOutputFormat class allow addition of arbitrary text at the end of generated query. This feature can be useful for example in case of MySQL database to specify "ON DUPLICATE KEY UPDATE ..." part of the query.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2614) Allow append arbitrary text at
the end of generated query in DBOutputFormat class
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054246#comment-13054246 ]
Jarek Jarcec Cecho commented on MAPREDUCE-2614:
-----------------------------------------------
Hi Josh,
thanks for supporting my idea. I was actually planning to add the ability for INSERT IGNORE and INSERT DELAYED as well, right after I get patch for this issue working.
Your idea about getting the functionality out and making several classes specific to databases seems reasonable to me, so I'll change my patch and try to upload it once more.
Jarcec
> Allow append arbitrary text at the end of generated query in DBOutputFormat class
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2614
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2614
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Attachments: MAPREDUCE-2614.patch
>
>
> It would be wonderful if DBOutputFormat class allow addition of arbitrary text at the end of generated query. This feature can be useful for example in case of MySQL database to specify "ON DUPLICATE KEY UPDATE ..." part of the query.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2614) Allow append arbitrary text at
the end of generated query in DBOutputFormat class
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated MAPREDUCE-2614:
------------------------------------------
Status: Open (was: Patch Available)
> Allow append arbitrary text at the end of generated query in DBOutputFormat class
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2614
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2614
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Attachments: MAPREDUCE-2614.patch
>
>
> It would be wonderful if DBOutputFormat class allow addition of arbitrary text at the end of generated query. This feature can be useful for example in case of MySQL database to specify "ON DUPLICATE KEY UPDATE ..." part of the query.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2614) Allow append arbitrary text at
the end of generated query in DBOutputFormat class
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053346#comment-13053346 ]
Hadoop QA commented on MAPREDUCE-2614:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483446/MAPREDUCE-2614.patch
against trunk revision 1138301.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
org.apache.hadoop.fs.TestFileSystem
org.apache.hadoop.mapreduce.lib.db.TestDBJob
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/411//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/411//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/411//console
This message is automatically generated.
> Allow append arbitrary text at the end of generated query in DBOutputFormat class
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2614
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2614
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Attachments: MAPREDUCE-2614.patch
>
>
> It would be wonderful if DBOutputFormat class allow addition of arbitrary text at the end of generated query. This feature can be useful for example in case of MySQL database to specify "ON DUPLICATE KEY UPDATE ..." part of the query.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira