You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2009/12/09 20:24:18 UTC
[jira] Created: (HIVE-976) test outputs should compare the
file/directory for sampling
test outputs should compare the file/directory for sampling
------------------------------------------------------------
Key: HIVE-976
URL: https://issues.apache.org/jira/browse/HIVE-976
Project: Hadoop Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Namit Jain
Assignee: Paul Yang
Fix For: 0.5.0
Currently, all lines starting with file: are ignored.
It means that
file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
and
file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
are same - that is not good because it will hide some of the optimizations of sampling.
This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi reassigned HIVE-976:
-------------------------------
Assignee: John Sichi (was: Ning Zhang)
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-976.
-----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Committed. Thanks John
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
> Attachments: HIVE-976.2.patch, HIVE-976.patch
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789088#action_12789088 ]
Paul Yang commented on HIVE-976:
--------------------------------
I'm looking at sample7.q.out, which has the lines that are quoted in the description. As I understand it, the test should check lines of the form 'file:/[directory]/[file]' and verify that the file part (or the last token) is the same as those in the reference test files. But aren't there lines in the test output that begin with 'file:/' that should be ignored completely?
Anyway, assuming that we can look at just the last token of lines that begin with 'file:/', it doesn't seem like there is a way to use only diff to handle this case. The man page for diff does not describe any useful options. I have two ideas
1. Instead of using diff directly, write a script that uses diff + some additional logic to detect this condition. Probably will be a little slower.
2. in mapredWork, add an additional function that would display the last token of the paths. Sort of an ugly idea, but would work with diff in the current form.
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Paul Yang
> Fix For: 0.5.0
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-976:
----------------------------
Attachment: HIVE-976.patch
Code changes together with log updates, for review by Namit. I had to add ORDER BY to a number of existing .q files due to non-determinism in the expected results.
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
> Attachments: HIVE-976.patch
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-976:
----------------------------
Attachment: HIVE-976.2.patch
Here's the updated patch which addresses Namit's comments (1) and (3).
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
> Attachments: HIVE-976.2.patch, HIVE-976.patch
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang reassigned HIVE-976:
------------------------------
Assignee: Ning Zhang (was: Paul Yang)
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Fix For: 0.5.0
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-976 started by John Sichi.
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788273#action_12788273 ]
Zheng Shao commented on HIVE-976:
---------------------------------
I am seeing sample6.q and sample7.q test failures due to this.
When running them by themselves, they are working correctly. But when running all tests together, they are failure because somehow they took the whole srcbucket as input while table sample should really take only srcbucket0.txt.
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Paul Yang
> Fix For: 0.5.0
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789132#action_12789132 ]
Namit Jain commented on HIVE-976:
---------------------------------
I would go with 2. - tests take a long time anyway, slowing them down further will be difficult.
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: Paul Yang
> Fix For: 0.5.0
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795550#action_12795550 ]
Namit Jain commented on HIVE-976:
---------------------------------
+1
looks good - will commit if the tests pass
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
> Attachments: HIVE-976.2.patch, HIVE-976.patch
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794975#action_12794975 ]
John Sichi commented on HIVE-976:
---------------------------------
I have implemented 2 and am currently running through tests; will submit a patch with updated logs once they pass.
I added a new derived attribute (normalExplain=false) "base file name" on partitionDesc, and since the whole point is to avoid filtering this information out, I think there will be a lot of log updates required as part of the patch (unless there's a way to make it selective for only tests that need it).
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795460#action_12795460 ]
Namit Jain commented on HIVE-976:
---------------------------------
1. Make the new field in partitionDesc transient.
2. Are you getting diff in input40.q - if not, remove the order by - it will change the query plan and I think the test was added to run without a map-reduce job.
3. sample6.q - change the explain plan statement also along with the query in all cases.
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
> Attachments: HIVE-976.patch
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795504#action_12795504 ]
John Sichi commented on HIVE-976:
---------------------------------
Regarding input40.q: yes, I get a diff because the results from this query come out in a different order:
select * from tmp_insert_test_p where ds = '2009-08-01';
A big chunk of rows moves in the output. Diff-based testing doesn't really work if it's not possible to add order by uniformly, does it?
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
> Attachments: HIVE-976.patch
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-976) test outputs should compare the
file/directory for sampling
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795507#action_12795507 ]
Namit Jain commented on HIVE-976:
---------------------------------
I agree - feel free to add order by in input40.q
> test outputs should compare the file/directory for sampling
> ------------------------------------------------------------
>
> Key: HIVE-976
> URL: https://issues.apache.org/jira/browse/HIVE-976
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Testing Infrastructure
> Reporter: Namit Jain
> Assignee: John Sichi
> Fix For: 0.5.0
>
> Attachments: HIVE-976.patch
>
>
> Currently, all lines starting with file: are ignored.
> It means that
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket/srcbucket0.txt [s]
> and
> file:/Users/heyongqiang/Documents/workspace/Hive-Test/build/ql/test/data/warehouse/srcbucket [s]
> are same - that is not good because it will hide some of the optimizations of sampling.
> This should be changed to compare the last token.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.