You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@avro.apache.org by Nishanth S <ch...@gmail.com> on 2015/09/11 00:14:09 UTC

LazyOutputFormat with AvroMultipleOutputs

Hello All,
We are using avro 1.7.7 and the  map reduce program we have writes  to
multiple avro output files.I  am using lazy out fomat in the driver but I
could still see empty avro files getting created if there is no data.We
have a hive table that points to this location and when  these avro files
are  empty hive queries fail on read.Is there a way to avoid writing empty
avro files when no data is  present or is there is a way  to set a property
in hive so  that these files are ignored?.Thanks for your help.

Thanks,
Nishanth

Re: LazyOutputFormat with AvroMultipleOutputs

Posted by Nishanth S <ch...@gmail.com>.

Hello guys,
We are using avro 1.7.7 and the  map reduce program we have writes  to
multiple avro output files.I  am using lazy out fomat in the driver but I
could still see empty avro files getting created if there is no data.We
have a hive table that points to this location and when  these avro files
are  empty hive queries fail on read.Is there a way to avoid writing empty
avro files when no data is  present or is there is a way  to set a property
in hive so  that these files are ignored?.Thanks for your help.

Thanks,
Nishanth

On Thu, Sep 10, 2015 at 4:14 PM, Nishanth S <ch...@gmail.com> wrote:

> Hello All,
> We are using avro 1.7.7 and the  map reduce program we have writes  to
> multiple avro output files.I  am using lazy out fomat in the driver but I
> could still see empty avro files getting created if there is no data.We
> have a hive table that points to this location and when  these avro files
> are  empty hive queries fail on read.Is there a way to avoid writing empty
> avro files when no data is  present or is there is a way  to set a property
> in hive so  that these files are ignored?.Thanks for your help.
>
> Thanks,
> Nishanth
>