You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Adam O'Donnell <ad...@immunet.com> on 2010/02/12 07:23:45 UTC
Distributing additional files for reduce scripts
Guys:
How do you go about distributing additional files that may be needed
by your reduce scripts? For example, I need to distribute a GeoIP
database with my reduce script to do some lookups.
Thanks!
Adam
--
Adam J. O'Donnell, Ph.D.
Immunet Corporation
Cell: +1 (267) 251-0070
Re: Distributing additional files for reduce scripts
Posted by Adam O'Donnell <ad...@immunet.com>.
So when I use the add file command in amazon's hive distribution, it
appears that the files get rewritten to have the s3:// url as part of
the file name, so that:
add file s3://bucket/module becomes s3_bucket_module.
Is there any way to get the add command to refer to the file as its
name without the extensions?
On Fri, Feb 12, 2010 at 1:06 PM, Adam O'Donnell <ad...@immunet.com> wrote:
> I see, said the blind man. Thanks!
>
> On Thu, Feb 11, 2010 at 10:30 PM, Zheng Shao <zs...@gmail.com> wrote:
>> add file myfile.txt;
>>
>> You can find some examples in *.q files in the distribution.
>>
>> Zheng
>>
>> On Thu, Feb 11, 2010 at 10:23 PM, Adam O'Donnell <ad...@immunet.com> wrote:
>>> Guys:
>>>
>>> How do you go about distributing additional files that may be needed
>>> by your reduce scripts? For example, I need to distribute a GeoIP
>>> database with my reduce script to do some lookups.
>>>
>>> Thanks!
>>>
>>> Adam
>>>
>>> --
>>> Adam J. O'Donnell, Ph.D.
>>> Immunet Corporation
>>> Cell: +1 (267) 251-0070
>>>
>>
>>
>>
>> --
>> Yours,
>> Zheng
>>
>
>
>
> --
> Adam J. O'Donnell, Ph.D.
> Immunet Corporation
> Cell: +1 (267) 251-0070
>
--
Adam J. O'Donnell, Ph.D.
Immunet Corporation
Cell: +1 (267) 251-0070
Re: Distributing additional files for reduce scripts
Posted by Adam O'Donnell <ad...@immunet.com>.
I see, said the blind man. Thanks!
On Thu, Feb 11, 2010 at 10:30 PM, Zheng Shao <zs...@gmail.com> wrote:
> add file myfile.txt;
>
> You can find some examples in *.q files in the distribution.
>
> Zheng
>
> On Thu, Feb 11, 2010 at 10:23 PM, Adam O'Donnell <ad...@immunet.com> wrote:
>> Guys:
>>
>> How do you go about distributing additional files that may be needed
>> by your reduce scripts? For example, I need to distribute a GeoIP
>> database with my reduce script to do some lookups.
>>
>> Thanks!
>>
>> Adam
>>
>> --
>> Adam J. O'Donnell, Ph.D.
>> Immunet Corporation
>> Cell: +1 (267) 251-0070
>>
>
>
>
> --
> Yours,
> Zheng
>
--
Adam J. O'Donnell, Ph.D.
Immunet Corporation
Cell: +1 (267) 251-0070
Re: Distributing additional files for reduce scripts
Posted by Zheng Shao <zs...@gmail.com>.
add file myfile.txt;
You can find some examples in *.q files in the distribution.
Zheng
On Thu, Feb 11, 2010 at 10:23 PM, Adam O'Donnell <ad...@immunet.com> wrote:
> Guys:
>
> How do you go about distributing additional files that may be needed
> by your reduce scripts? For example, I need to distribute a GeoIP
> database with my reduce script to do some lookups.
>
> Thanks!
>
> Adam
>
> --
> Adam J. O'Donnell, Ph.D.
> Immunet Corporation
> Cell: +1 (267) 251-0070
>
--
Yours,
Zheng