You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2008/02/14 21:31:07 UTC
[jira] Created: (HADOOP-2834) Iterator for MapFileOutputFormat
Iterator for MapFileOutputFormat
--------------------------------
Key: HADOOP-2834
URL: https://issues.apache.org/jira/browse/HADOOP-2834
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Affects Versions: 0.17.0
Reporter: Andrzej Bialecki
Fix For: 0.17.0
MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by Andrzej Bialecki <ab...@getopt.org>.
arkady borkovsky wrote:
> Is this (iterate over keys from all parts) part of the contract?
> Or is it a hack?
AFAIK, it's not a part of the contract for MapFileOutputFormat - the
OutputFormat is the contract.
Other static methods in this class are also not a part of the contract,
although they are useful higher-level abstractions for casual users of
this format. That's also the intention of this proposed change.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: [jira] Created: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by arkady borkovsky <ar...@yahoo-inc.com>.
Is this (iterate over keys from all parts) part of the contract?
Or is it a hack?
--ab
On Feb 14, 2008, at 12:31 PM, Andrzej Bialecki (JIRA) wrote:
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/
> HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
>
> MapFileOutputFormat produces output data that is sorted locally in
> each part-NNNNN file - however, there is no easy way to iterate
> over keys from all parts in a globally ascending order.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569168#action_12569168 ]
Owen O'Malley commented on HADOOP-2834:
---------------------------------------
Wouldn't it be easier to just use SequenceFile.Sorter.merge between the part-* files? It gives you a RawKeyValueIterator, but it would be easy to wrap it and make an object iterator.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2834:
----------------------------------
Status: Open (was: Patch Available)
I'm sorry, but you've got a few accidental white space diffs.
I also think it would be a very good idea if your Reader.next validated that the types of the key and value that were passed in matched the types in the sequence file. Otherwise, the user can accidentally cross the streams and get random data corruption and run time errors.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: map-file-v2.patch, map-file-v3.patch, map-file-v4.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Attachment: map-file.patch
This patch adds a method:
public Iterator<Entry<WritableComparable, Writable>> getIterator(Path path, Configuration conf)
which you can use to iterate over key/value pairs in a globally ascending key order.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573121#action_12573121 ]
Hadoop QA commented on HADOOP-2834:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12375644/map-file-v3.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1853/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1853/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1853/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1853/console
This message is automatically generated.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569102#action_12569102 ]
Hadoop QA commented on HADOOP-2834:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12375624/map-file.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1801/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1801/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1801/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1801/console
This message is automatically generated.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Attachment: (was: map-file.patch)
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Status: Open (was: Patch Available)
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Status: Patch Available (was: Open)
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: map-file-v2.patch, map-file-v3.patch, map-file-v4.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569124#action_12569124 ]
Andrzej Bialecki commented on HADOOP-2834:
-------------------------------------------
We want to be sure that we are dealing with a bunch of MapFile-s, which means the parts should be directories. A common mistake will be to attempt to process a SequenceFileOutputFormat data. Is there any less expensive method to check this?
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576204#action_12576204 ]
Enis Soztutar commented on HADOOP-2834:
---------------------------------------
We do not offer an Iterator for MapFiles, but use MapFile.Reader#next(). Wouldn't it be better if we (a) add Iterator to MapFile.Reader and apply this patch or (b) change this patch to define MapFileOutputFormat.Reader instead of Iterators, so that reading from MapFile and MapFileOutputFormat is consistent.
one more minor issue : I think we should change the generics to :
{code}
private static final class IteratorEntry<K extends WritableComparable, V extends Writable> implements Entry<K, V> {
...
}
private static final class MapFileOutputFormatIterator<K extends WritableComparable, V extends Writable> implements Iterator<Entry<K, V>> {
...
}
public static<K extends WritableComparable, V extends Writable> Iterator<Entry<K, V>>
getIterator(Path dir, Configuration conf) throws IOException {
...
}
{code}
so that we can use :
{code}
Iterator<Entry<Text, Text>> x = MapFileOutputFormat.getIterator(path, conf);
{code}
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577244#action_12577244 ]
Andrzej Bialecki commented on HADOOP-2834:
-------------------------------------------
The implementation in the latest patch doesn't create new Entries, it reuses a single instance. Though I agree that considering your doubt, and the already existing MapFile.Reader API, it would be more intuitive to create a MapFileOutputFormat.Reader, as Enis suggested.
I'll prepare an updated patch.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587796#action_12587796 ]
Hadoop QA commented on HADOOP-2834:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379878/map-file-v4.patch
against trunk revision 645773.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 4 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac -1. The applied patch generated 500 javac compiler warnings (more than the trunk's current 498 warnings).
release audit -1. The applied patch generated 203 release audit warnings (more than the trunk's current 202 warnings).
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2200/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2200/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2200/artifact/trunk/build/test/checkstyle-errors.html
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2200/artifact/trunk/current/releaseAuditDiffWarnings.txt
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2200/console
This message is automatically generated.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: map-file-v2.patch, map-file-v3.patch, map-file-v4.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577242#action_12577242 ]
Doug Cutting commented on HADOOP-2834:
--------------------------------------
Would this return a new Entry per key/value pair? Would these each contain new key/value instances? The existing API encourages object reuse, which can provide substantial performance improvements. Do we want to encourage developers to develop suboptimal code?
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580422#action_12580422 ]
Andrzej Bialecki commented on HADOOP-2834:
-------------------------------------------
I'm working on an updated patch, and I just realized that we can only iterate over entries (using RawKeyValueIterator), and we can't support some other methods in the Reader abstraction, methods that are available in other readers such as seek ... We can only implement next(), get(), reset() and close(). If people feel that this still falls under the Reader rather than Iterator I'll complete the patch that implements it.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Status: Patch Available (was: Open)
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Attachment: map-file-v2.patch
Avoid creating streams.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569235#action_12569235 ]
Andrzej Bialecki commented on HADOOP-2834:
-------------------------------------------
I believe that's exactly what the patch does ;) The "easy" part that you refer to amounts to the bulk of the patch.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569119#action_12569119 ]
Doug Cutting commented on HADOOP-2834:
--------------------------------------
One more nit: the isFile() call in MapFilePathFilter is expensive. Is it really required? Isn't checking that the name is "part-" sufficient?
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Attachment: map-file-v3.patch
Updated patch - I removed the PathFilter, it wasn't worth the hassle.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580787#action_12580787 ]
Enis Soztutar commented on HADOOP-2834:
---------------------------------------
I think we can keep the iterator functionality but, name the class Reader. The Reader might not have seek, etc, but iterating the elements *reusing* Writables will be more intuitive in :
{code}
MapFileOutputFormat.Reader reader;
K key;
V value;
...
while(reader.next(key, value)) {
//use key, value
}
{code}
than :
{code}
Iterator<Entry<K, V>> it = MapFileOutputFormat.getIterator(path, conf);
while(it.hasNext()) {
Entry<K, V> entry = it.next();
// entry.getKey(); entry.getValue():
}
{code}
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-2834:
------------------------------------
Fix Version/s: (was: 0.17.0)
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Attachment: map-file-v4.patch
Patch changed to implement a Reader. JUnit test case added.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Attachments: map-file-v2.patch, map-file-v3.patch, map-file-v4.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-2834:
--------------------------------
Assignee: Andrzej Bialecki
Status: Open (was: Patch Available)
Cancelling the patch since the updated patch hasn't been submitted yet.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569126#action_12569126 ]
Doug Cutting commented on HADOOP-2834:
--------------------------------------
> Is there any less expensive method to check this?
Yes. When looping over the results of listStatus(), check whether each is a directory or not.
We should probably add a StatusFilter interface, since that's more general...
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-2834:
--------------------------------------
Status: Patch Available (was: Open)
Check this with Hudson.
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file-v2.patch, map-file-v3.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2834) Iterator for MapFileOutputFormat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12569092#action_12569092 ]
Doug Cutting commented on HADOOP-2834:
--------------------------------------
Instead of creating a new ByteArrayInputStream and DataInputStream per key and value, why not re-use a single DataInputBuffer, cached as a field in the iterator?
> Iterator for MapFileOutputFormat
> --------------------------------
>
> Key: HADOOP-2834
> URL: https://issues.apache.org/jira/browse/HADOOP-2834
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Andrzej Bialecki
> Fix For: 0.17.0
>
> Attachments: map-file.patch
>
>
> MapFileOutputFormat produces output data that is sorted locally in each part-NNNNN file - however, there is no easy way to iterate over keys from all parts in a globally ascending order.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.