You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Omer Trajman (JIRA)" <ji...@apache.org> on 2009/10/13 05:07:31 UTC
[jira] Created: (MAPREDUCE-1097) Use deploy_design function in
Vertica 3.5
Use deploy_design function in Vertica 3.5
-----------------------------------------
Key: MAPREDUCE-1097
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.21.0
Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
Reporter: Omer Trajman
Assignee: Omer Trajman
Priority: Minor
Vertica 3.5 includes a deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Hadoop Flags: [Incompatible change]
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Status: Patch Available (was: Open)
Thought about this some more - since 0.21 hasn't been released and this is the first version that has this contrib seems like its ok to not handle the case of a persisted Writable with a double instead of a decimal. Changed to incompatible to make explicit.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781258#action_12781258 ]
Hadoop QA commented on MAPREDUCE-1097:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12425767/MAPREDUCE-1097.patch
against trunk revision 882790.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/259/console
This message is automatically generated.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Fix Version/s: 0.21.0
Status: Patch Available (was: Open)
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Attachment: MAPREDUCE-1097.patch
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Attachment: MAPREDUCE-1097-3.patch
Updated patch against latest changes.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1097:
-------------------------------------
Component/s: contrib/vertica
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/vertica
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786413#action_12786413 ]
Omer Trajman commented on MAPREDUCE-1097:
-----------------------------------------
Test failure was in gridmix. Any suggestions? Should I just resubmit?
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787431#action_12787431 ]
Hadoop QA commented on MAPREDUCE-1097:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427297/MAPREDUCE-1097-2.patch
against trunk revision 888269.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/173/console
This message is automatically generated.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828472#action_12828472 ]
Hudson commented on MAPREDUCE-1097:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/])
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/vertica
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787176#action_12787176 ]
Aaron Kimball commented on MAPREDUCE-1097:
------------------------------------------
Omer,
The gridmix failure is a known bug elsewhere in Hadoop; that's unrelated to this patch.
As for the rest of the code you posted...
VerticaOutputFormat:
* make a constant {{VERSION_3_5 = 305}} in VerticaUtil?
* "pool for refresh complete" -> "poll..."
* You select table_name, projection_name, and status from vt_projection_refresh, but you never seem to read projection_name from the ResultSet. Is this intentional?
* Also, I think you should call ResultSet.close() before calling the next stmt.executeQuery()
* Related, at the end of the function, I think you may want to call stmt.close() to ensure that all those resources are freed before continuing on.
* What's an "ahm" ? (Add to source comment?)
VerticaRecord:
* line 378: no need to cast to BigDecimal before calling toString
* It looks like REAL, DECIMAL, and NUMERIC used to be treated as doubles in Java, and are now treated as BigDecimal. Is this correct? If so, what happens to existing data (e.g., serialized into SequenceFiles) that contains data of this type? Is this an incompatible change?
* It also looks as though you are changing how nulls are handled. Is this backwards compatible with existing VerticaRecord deployments?
* line 575: unnecessary cast to BigDecimal.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786344#action_12786344 ]
Hadoop QA commented on MAPREDUCE-1097:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12425767/MAPREDUCE-1097.patch
against trunk revision 887135.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/293/console
This message is automatically generated.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788248#action_12788248 ]
Aaron Kimball commented on MAPREDUCE-1097:
------------------------------------------
VerticaRecord does implement Writable, so it's conceivable that users may have stored VerticaRecord instances in SequenceFiles on HDFS; regardless of whether you intended this to be an internal-only class, users do funny things.. but hopefully not too many will have done so. I don't think it's a big problem, especially since this is a fairly young contrib module. Some volatility should probably be expected here.
At the very least, though, you should mark this issue as an "incompatible change" (see the "Edit this issue" link on the left).
Thanks for making the other changes. I'm +0.75 on this patch -- would appreciate it if someone else could weigh in on the issue of whether they think persistent VerticaRecord incompatibility is problematic or if they've got a creative solution. (Some day we'll all just use Avro and schema evolution will take care of all of this... but until then..)
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Status: Open (was: Patch Available)
fixes per Aaron's comments.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Status: Patch Available (was: Open)
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787273#action_12787273 ]
Omer Trajman commented on MAPREDUCE-1097:
-----------------------------------------
Thanks Aaron - I'll tackle these and post a new patch.
On VerticaRecord - it's not supposed to be used as a persistent format so I didn't put in any handling for these changes. Not sure if there's any way to indicate that.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Attachment: MAPREDUCE-1097-2.patch
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Status: Open (was: Patch Available)
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798445#action_12798445 ]
Hadoop QA commented on MAPREDUCE-1097:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429841/MAPREDUCE-1097-3.patch
against trunk revision 897118.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/371/console
This message is automatically generated.
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Release Note: Adds support for Vertica 3.5 truncate table, deploy_design and numeric types.
Status: Patch Available (was: Open)
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Description:
Vertica 3.5 includes three changes that the formatters should handle:
1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
2) truncate table instead of recreating the table
3) numeric, decimal, money, number types (all the same path)
was:Vertica 3.5 includes a deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
Summary: Changes/fixes to support Vertica 3.5 (was: Use deploy_design function in Vertica 3.5)
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804583#action_12804583 ]
Hudson commented on MAPREDUCE-1097:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #216 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/216/])
. Add support for Vertica 3.5 to its contrib module. Contributed by Omer Trajman
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/vertica
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Omer Trajman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omer Trajman updated MAPREDUCE-1097:
------------------------------------
Status: Open (was: Patch Available)
wrong target
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Attachments: MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1097) Changes/fixes to support Vertica
3.5
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1097:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change])
Status: Resolved (was: Patch Available)
bq. Thought about this some more - since 0.21 hasn't been released and this is the first version that has this contrib seems like its ok to not handle the case of a persisted Writable with a double instead of a decimal. Changed to incompatible to make explicit.
Since you're the sole contributor to this module, I view this as your decision. FWIW, as it hasn't yet been released, an incompatible change is exactly the risk early adopters take, in my opinion.
I committed this. Thanks, Omer!
> Changes/fixes to support Vertica 3.5
> ------------------------------------
>
> Key: MAPREDUCE-1097
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1097
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/vertica
> Affects Versions: 0.21.0
> Environment: Hadoop 0.21.0 pre-release and Vertica 3.5
> Reporter: Omer Trajman
> Assignee: Omer Trajman
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1097-2.patch, MAPREDUCE-1097-3.patch, MAPREDUCE-1097.patch
>
>
> Vertica 3.5 includes three changes that the formatters should handle:
> 1) deploy_design function that handles much of the logic in the optimize method. This improvement uses deploy_design if the server version supports it instead of orchestrating in the formatter function.
> 2) truncate table instead of recreating the table
> 3) numeric, decimal, money, number types (all the same path)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.