You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Nathan Bamford <na...@redpoint.net> on 2015/02/13 22:50:18 UTC

writing to partitions with HCatWriter

Hi all,

  I'm using HCatWriter in a java program to write records to a partitioned Hive table. It works great, but I notice it leaves behind the _SCRATCH directories it uses for staging (before HCatWriter.commit is called).?

  When it's all said and done, the partitioned records are in the appropriate directory (e.g. state=CO), and the _SCRATCH directories are empty.

  I tried running a load of the same records/partition values via the CLI, and after the mapreduce job has finished, the _SCRATCH directories are cleaned up. Only the finished partition dirs remain.

  Is there something I'm  missing with HCatWriter?


Thanks,


Nathan

Re: writing to partitions with HCatWriter

Posted by Alan Gates <al...@gmail.com>.

This sounds like a bug in the HCatWriter.  You should file a JIRA so we 
can track it.

Alan.

> Nathan Bamford <ma...@redpoint.net>
> February 13, 2015 at 13:50
>
> Hi all,
>
>   I'm using HCatWriter in a java program to write records to a 
> partitioned Hive table. It works great, but I notice it leaves behind 
> the _SCRATCH directories it uses for staging (before HCatWriter.commit 
> is called).
>
>   When it's all said and done, the partitioned records are in the 
> appropriate directory (e.g. state=CO), and the _SCRATCH directories 
> are empty.
>
>   I tried running a load of the same records/partition values via the 
> CLI, and after the mapreduce job has finished, the _SCRATCH 
> directories are cleaned up. Only the finished partition dirs remain.
>
>   Is there something I'm  missing with HCatWriter?
>
>
> Thanks,
>
>
> Nathan
>
>