You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by xeon Mailinglist <xe...@gmail.com> on 2016/01/06 16:38:22 UTC
How can I create a SequenceFiles with `org.apache.hadoop.io.Text`?
Hi,
This is a snippet of a Gridmix code available in Hadop MapReduce V1, but I
have the following question.
They set `org.apache.hadoop.mapred.SequenceFileInputFormat` and
`org.apache.hadoop.mapred.SequenceFileOutputFormat` as the `inFormat` and
`outFormat` respectively, and it also has `org.apache.hadoop.io.Text` as
`outKey` and `outValue`. For me, this seems that this example accepts Text
files as sequence files. How can I create a SequenceFiles with
`org.apache.hadoop.io.Text`?
```
WEBDATASCAN("webdataScan") {
public void addJob(int numReducers, boolean mapoutputCompressed,
boolean outputCompressed, Size size, JobControl gridmix) {
final String prop = String.format("webdataScan.%sJobs.inputFiles",
size);
final String indir = getInputDirsFor(prop,
size.defaultPath(VARCOMPSEQ));
final String outdir = addTSSuffix("perf-out/webdata-scan-out-dir-" +
size);
StringBuffer sb = new StringBuffer();
sb.append("-keepmap 0.2 ");
sb.append("-keepred 5 ");
sb.append("-inFormat org.apache.hadoop.mapred.SequenceFileInputFormat
");
sb.append("-outFormat
org.apache.hadoop.mapred.SequenceFileOutputFormat ");
sb.append("-outKey org.apache.hadoop.io.Text ");
sb.append("-outValue org.apache.hadoop.io.Text ");
sb.append("-indir ").append(indir).append(" ");
sb.append("-outdir ").append(outdir).append(" ");
sb.append("-r ").append(numReducers);
String[] args = sb.toString().split(" ");
clearDir(outdir);
try {
JobConf jobconf = GenericMRLoadJobCreator.createJob(
args, mapoutputCompressed, outputCompressed);
jobconf.setJobName("GridmixWebdatascan." + size);
Job job = new Job(jobconf);
gridmix.addJob(job);
} catch (Exception ex) {
System.out.println(ex.getStackTrace());
}
}
}
```