You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by Corbin Hoenes <co...@tynt.com> on 2010/11/11 00:31:36 UTC
ChukwaStorer
I'm using the chukwa-pig7 contrib in my pig scripts and noticed it trys to compress the chukwa output when I do a store.
@Override
public void setStoreLocation(String location, Job job) throws IOException {
FileOutputFormat.setOutputPath(job, new Path(location));
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, DefaultCodec.class);
job.setOutputKeyClass(ChukwaRecordKey.class);
job.setOutputValueClass(ChukwaRecord.class);
}
What is the best way to make this configurable? I was hoping I could do something like this:
@Override
public void setStoreLocation(String location, Job job) throws IOException {
FileOutputFormat.setOutputPath(job, new Path(location));
FileOutputFormat.setCompressOutput(job, this.compress);
if(this.compress)
FileOutputFormat.setOutputCompressorClass(job, DefaultCodec.class);
job.setOutputKeyClass(ChukwaRecordKey.class);
job.setOutputValueClass(ChukwaRecord.class);
}
How to make the codec class pluggable/configurable as well--for example if one wanted to use LZO?
Re: ChukwaStorer
Posted by Bill Graham <bi...@gmail.com>.
I would think adding another constructor that takes these options in
the signature would be the best approach.
On Wed, Nov 10, 2010 at 6:31 PM, Corbin Hoenes <co...@tynt.com> wrote:
> I'm using the chukwa-pig7 contrib in my pig scripts and noticed it trys to compress the chukwa output when I do a store.
>
>
> @Override
> public void setStoreLocation(String location, Job job) throws IOException {
> FileOutputFormat.setOutputPath(job, new Path(location));
> FileOutputFormat.setCompressOutput(job, true);
> FileOutputFormat.setOutputCompressorClass(job, DefaultCodec.class);
> job.setOutputKeyClass(ChukwaRecordKey.class);
> job.setOutputValueClass(ChukwaRecord.class);
> }
>
> What is the best way to make this configurable? I was hoping I could do something like this:
>
> @Override
> public void setStoreLocation(String location, Job job) throws IOException {
> FileOutputFormat.setOutputPath(job, new Path(location));
> FileOutputFormat.setCompressOutput(job, this.compress);
> if(this.compress)
> FileOutputFormat.setOutputCompressorClass(job, DefaultCodec.class);
> job.setOutputKeyClass(ChukwaRecordKey.class);
> job.setOutputValueClass(ChukwaRecord.class);
> }
>
> How to make the codec class pluggable/configurable as well--for example if one wanted to use LZO?
>
>