You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "robert Cook (JIRA)" <ji...@apache.org> on 2010/01/28 22:13:34 UTC
[jira] Updated: (HADOOP-6513) SequenceFile.Sorter design issue and
class-check bug
[ https://issues.apache.org/jira/browse/HADOOP-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
robert Cook updated HADOOP-6513:
--------------------------------
Attachment: TestSequenceFile.java
Here's an update to TestSequenceFile with a new Unit test as you requested.
the test suceeds; it should fail.
PS Re 6512
By your argument, there should be no synchronized prefixes in the SequenceFile user I/O methods,
but there are.
I respectfully disagree with your conclusion.
> SequenceFile.Sorter design issue and class-check bug
> -----------------------------------------------------
>
> Key: HADOOP-6513
> URL: https://issues.apache.org/jira/browse/HADOOP-6513
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: hadoop 20.1, java 1.6.0_17,fedora
> Reporter: robert Cook
> Attachments: TestSequenceFile.java
>
>
> SequenceFile.Writer takes key/value classes as creation arguments and checks for validity on every append.
> Reader does not take class arguments on creation because they are derived from the input file.
> Sorter takes key/value classes as creation arguments?? no point. should be derived from input.
> In any case, SortPass does not compare Sorter key/value classes with input file classes.
> No error is given for the following:
> private static void writeTest4(FileSystem fs, int count, int seed, Path file,
> SequenceFile.CompressionType compressionType, CompressionCodec codec, Configuration conf)
> throws IOException {
> fs.delete(file, true);
> LOG.info("creating " + count + " records with " + compressionType +
> " compression");
> SequenceFile.Writer writer =
> SequenceFile.createWriter(fs, conf, file,
> StringWritable.class, FloatWritable.class, compressionType, codec);
> FloatWritable x=new FloatWritable();
> StringWritable y=new StringWritable();
> for (int i = count-1; i >= 0; i--) {
> x.set(i); y.set(""+i);
> writer.append(y, x);
> }
> writer.close();
> }
> private static void sortTest(FileSystem fs, int count, int megabytes,
> int factor, boolean fast, Path file, Configuration conf)
> throws IOException {
> fs.delete(new Path(file+".sorted"), true);
> SequenceFile.Sorter sorter = newSorter(fs, fast, megabytes, factor, conf);
> LOG.debug("sorting " + count + " records");
> sorter.sort(file, file.suffix(".sorted"));
> LOG.info("done sorting " + count + " debug");
> }
>
> private static SequenceFile.Sorter newSorter(FileSystem fs,
> boolean fast,
> int megabytes, int factor, Configuration conf) {
> SequenceFile.Sorter sorter =
> fast
> ? new SequenceFile.Sorter(fs, new IntWritable.Comparator(),
> FloatWritable.class, IntWritable.class, conf)
> : new SequenceFile.Sorter(fs, FloatWritable.class, IntWritable.class, conf);
> sorter.setMemory(megabytes * 1024*1024);
> sorter.setFactor(factor);
> return sorter;
> }
> ---------------------Note String/Float does not match Float/Int
> Macintosh-2:datanode bobcook$ od -c file
> 0000000 S E Q 006 016 S t r i n g W r i t a
> 0000020 b l e \r F l o a t W r i t a b l
> 0000040 e \0 \0 \0 \0 \0 \0 203 ` n E J z 272 d 352
> 0000060 w 177 373 \n 364 M 276 \0 \0 \0 \n \0 \0 \0 006 \0
> 0000100 \0 \0 001 \0 4 @ 200 \0 \0 \0 \0 \0 \n \0 \0 \0
> 0000120 006 \0 \0 \0 001 \0 3 @ @ \0 \0 \0 \0 \0 \n \0
> 0000140 \0 \0 006 \0 \0 \0 001 \0 2 @ \0 \0 \0 \0 \0 \0
> 0000160 \n \0 \0 \0 006 \0 \0 \0 001 \0 1 ? 200 \0 \0 \0
> 0000200 \0 \0 \n \0 \0 \0 006 \0 \0 \0 001 \0 0 \0 \0 \0
> *
> 0000220
> Macintosh-2:datanode bobcook$ od -c file.sorted
> 0000000 S E Q 006 \r F l o a t W r i t a b
> 0000020 l e \v I n t W r i t a b l e \0 \0
> 0000040 \0 \0 \0 \0 6 364 343 \r 256 h U 222 365 T 7 l
> 0000060 357 i ~ } \0 \0 \0 \n \0 \0 \0 006 \0 \0 \0 001
> 0000100 \0 4 @ 200 \0 \0 \0 \0 \0 \n \0 \0 \0 006 \0 \0
> 0000120 \0 001 \0 3 @ @ \0 \0 \0 \0 \0 \n \0 \0 \0 006
> 0000140 \0 \0 \0 001 \0 2 @ \0 \0 \0 \0 \0 \0 \n \0 \0
> 0000160 \0 006 \0 \0 \0 001 \0 1 ? 200 \0 \0 \0 \0 \0 \n
> 0000200 \0 \0 \0 006 \0 \0 \0 001 \0 0 \0 \0 \0 \0
> 0000216
> NOTE OUTPUT FILE IS TOTALLY TOASTED, but no error was generated!
> PS: Your evaluation of my previous bug reports was enlightening.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.