You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@sqoop.apache.org by Lars Francke <la...@gmail.com> on 2012/01/18 16:16:39 UTC

Should compression with Avro import work?

Hi,

I've tried using sqoop import with Avro and it works beautifully. But
I've tried it with compression which according to the documentation
should work. It didn't for me so before I go any further: Is this a
problem in the documentation?

I briefly looked at the AvroOutputFormat and it doesn't do any
compression as far as I can tell. The job.xml is certainly correct in
that it mentions mapred.output.compress=true.

Removing the --as-avrodatafile option gives me a compressed text file
so I'm pretty sure I've got the command line right.

Cheers,
Lars

Re: Should compression with Avro import work?

Posted by Lars Francke <la...@gmail.com>.

Hey Tom,

thanks for the answer, that saves me from searching any further.

I'll open an issue.

Cheers,
Lars

Re: Should compression with Avro import work?

Posted by Tom White <to...@cloudera.com>.

Hi Lars,

I don't think compression is wired up for Avro imports - would you
mind filing a JIRA for it?

Thanks,
Tom

On Wed, Jan 18, 2012 at 7:16 AM, Lars Francke <la...@gmail.com> wrote:
> Hi,
>
> I've tried using sqoop import with Avro and it works beautifully. But
> I've tried it with compression which according to the documentation
> should work. It didn't for me so before I go any further: Is this a
> problem in the documentation?
>
> I briefly looked at the AvroOutputFormat and it doesn't do any
> compression as far as I can tell. The job.xml is certainly correct in
> that it mentions mapred.output.compress=true.
>
> Removing the --as-avrodatafile option gives me a compressed text file
> so I'm pretty sure I've got the command line right.
>
> Cheers,
> Lars