You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by Josh Wills <jw...@cloudera.com> on 2012/12/02 23:14:15 UTC
Review Request: Update the io.* package to support all InputFormats that
ship with hadoop-client.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8308/
-----------------------------------------------------------
Review request for crunch.
Description
-------
We should add support for the NLineInputFormat and the KeyValueTextInputFormat so that all of the InputFormats in hadoop's mapredice.lib.input package are supported in Crunch.
This addresses bug CRUNCH-119.
https://issues.apache.org/jira/browse/CRUNCH-119
Diffs
-----
crunch/src/it/java/org/apache/crunch/io/CompositePathIterableIT.java 796b821
crunch/src/it/java/org/apache/crunch/io/NLineInputIT.java PRE-CREATION
crunch/src/it/java/org/apache/crunch/io/TextFileTableIT.java PRE-CREATION
crunch/src/main/java/org/apache/crunch/io/ReadableSource.java 73a13a3
crunch/src/main/java/org/apache/crunch/io/avro/AvroFileReaderFactory.java 6f21dd2
crunch/src/main/java/org/apache/crunch/io/avro/AvroFileSource.java 2226556
crunch/src/main/java/org/apache/crunch/io/impl/AutoClosingIterator.java d58f290
crunch/src/main/java/org/apache/crunch/io/impl/FileTableSourceImpl.java f6e8f1d
crunch/src/main/java/org/apache/crunch/io/seq/SeqFileReaderFactory.java ad1b81b
crunch/src/main/java/org/apache/crunch/io/seq/SeqFileSource.java e8f3dcf
crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableReaderFactory.java 20c749a
crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableSource.java 56ed985
crunch/src/main/java/org/apache/crunch/io/text/LineParser.java PRE-CREATION
crunch/src/main/java/org/apache/crunch/io/text/NLineFileSource.java PRE-CREATION
crunch/src/main/java/org/apache/crunch/io/text/TextFileReaderFactory.java a0c48e0
crunch/src/main/java/org/apache/crunch/io/text/TextFileSource.java ee51c04
crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSource.java PRE-CREATION
crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSourceTarget.java PRE-CREATION
crunch/src/main/java/org/apache/crunch/io/text/TextFileTarget.java c7e06d3
crunch/src/test/java/org/apache/crunch/io/avro/AvroFileReaderFactoryTest.java 66863ba
Diff: https://reviews.apache.org/r/8308/diff/
Testing
-------
Unit and integration on the new sources.
Thanks,
Josh Wills
Re: Review Request: Update the io.* package to support all InputFormats that
ship with hadoop-client.
Posted by Matthias Friedrich <ma...@mafr.de>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8308/#review13975
-----------------------------------------------------------
Ship it!
Looks good! Some nice to have suggestions below, but +1 for commit in any case.
crunch/src/main/java/org/apache/crunch/io/text/LineParser.java
<https://reviews.apache.org/r/8308/#comment29895>
What about something like Splitter.on(sep).limit(1).split(input)?
crunch/src/main/java/org/apache/crunch/io/text/NLineFileSource.java
<https://reviews.apache.org/r/8308/#comment29896>
I think the linesPerTask parameter is interesting enough to warrant some javadoc.
- Matthias Friedrich
On Dec. 2, 2012, 10:14 p.m., Josh Wills wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8308/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2012, 10:14 p.m.)
>
>
> Review request for crunch.
>
>
> Description
> -------
>
> We should add support for the NLineInputFormat and the KeyValueTextInputFormat so that all of the InputFormats in hadoop's mapredice.lib.input package are supported in Crunch.
>
>
> This addresses bug CRUNCH-119.
> https://issues.apache.org/jira/browse/CRUNCH-119
>
>
> Diffs
> -----
>
> crunch/src/it/java/org/apache/crunch/io/CompositePathIterableIT.java 796b821
> crunch/src/it/java/org/apache/crunch/io/NLineInputIT.java PRE-CREATION
> crunch/src/it/java/org/apache/crunch/io/TextFileTableIT.java PRE-CREATION
> crunch/src/main/java/org/apache/crunch/io/ReadableSource.java 73a13a3
> crunch/src/main/java/org/apache/crunch/io/avro/AvroFileReaderFactory.java 6f21dd2
> crunch/src/main/java/org/apache/crunch/io/avro/AvroFileSource.java 2226556
> crunch/src/main/java/org/apache/crunch/io/impl/AutoClosingIterator.java d58f290
> crunch/src/main/java/org/apache/crunch/io/impl/FileTableSourceImpl.java f6e8f1d
> crunch/src/main/java/org/apache/crunch/io/seq/SeqFileReaderFactory.java ad1b81b
> crunch/src/main/java/org/apache/crunch/io/seq/SeqFileSource.java e8f3dcf
> crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableReaderFactory.java 20c749a
> crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableSource.java 56ed985
> crunch/src/main/java/org/apache/crunch/io/text/LineParser.java PRE-CREATION
> crunch/src/main/java/org/apache/crunch/io/text/NLineFileSource.java PRE-CREATION
> crunch/src/main/java/org/apache/crunch/io/text/TextFileReaderFactory.java a0c48e0
> crunch/src/main/java/org/apache/crunch/io/text/TextFileSource.java ee51c04
> crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSource.java PRE-CREATION
> crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSourceTarget.java PRE-CREATION
> crunch/src/main/java/org/apache/crunch/io/text/TextFileTarget.java c7e06d3
> crunch/src/test/java/org/apache/crunch/io/avro/AvroFileReaderFactoryTest.java 66863ba
>
> Diff: https://reviews.apache.org/r/8308/diff/
>
>
> Testing
> -------
>
> Unit and integration on the new sources.
>
>
> Thanks,
>
> Josh Wills
>
>