You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Gabriel Eisbruch <ga...@gmail.com> on 2011/11/02 21:42:42 UTC

Hive Loader perfectly to use together with flume

Hello every body,
    I have developed a hive loder that enable us to load hive partitions
base on the file name into a directoy It give us the posibility of get hive
date partition or other partitions kind using flume (while wait for the
version 9.5).

The project: https://github.com/geisbruch/HiveLoader

Thanks,
Gabriel.

Re: Hive Loader perfectly to use together with flume

Posted by Gabriel Eisbruch <ga...@gmail.com>.

Hi Migjie,
    In hive you doesn't need move the data to another site however if ypu
use hive reading a simple folder, it's imposible run querys with using only
a little part of data (for ejample if you have server logs, read only one
day or one month)  so you can use "partition" to generate "buckets". The
problem it's that flume doesn't have a "trigger" or similar to create new
partitions in hive when a file is finished (this problem is solved by my
little program).

Gabriel.



2011/11/3 Mingjie Lai <mj...@gmail.com>

> Sounds interesting.
>
> My stupid question, do you mean loading data from hive to somewhere else?
> (I don't know much detail about hive)
>
>
>
> On 11/02/2011 01:42 PM, Gabriel Eisbruch wrote:
>
>> Hello every body,
>>     I have developed a hive loder that enable us to load hive
>> partitions base on the file name into a directoy It give us the
>> posibility of get hive date partition or other partitions kind using
>> flume (while wait for the version 9.5).
>>
>> The project: https://github.com/geisbruch/**HiveLoader<https://github.com/geisbruch/HiveLoader>
>>
>> Thanks,
>> Gabriel.
>>
>>

Re: Hive Loader perfectly to use together with flume

Posted by Mingjie Lai <mj...@gmail.com>.

Sounds interesting.

My stupid question, do you mean loading data from hive to somewhere 
else? (I don't know much detail about hive)


On 11/02/2011 01:42 PM, Gabriel Eisbruch wrote:
> Hello every body,
>      I have developed a hive loder that enable us to load hive
> partitions base on the file name into a directoy It give us the
> posibility of get hive date partition or other partitions kind using
> flume (while wait for the version 9.5).
>
> The project: https://github.com/geisbruch/HiveLoader
>
> Thanks,
> Gabriel.
>