You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ryan LeCompte <le...@gmail.com> on 2008/12/08 23:19:18 UTC
Streaming data into Hadoop
Hello all,
I normally upload files into hadoop via bin/hadoop fs -put file dest.
However, is there a way to somehow stream data into Hadoop?
For example, I'd love to do something like this:
zcat xxx >> HADOOP_HDFS_DESTINATION
This would save me a ton of time since I don't have to first unpack
the .tgz file and upload the raw file into HDFS.
Is this possible with Hadoop 0.19?
Thanks,
Ryan
Re: Streaming data into Hadoop
Posted by Ryan LeCompte <le...@gmail.com>.
Even better! I'll try this out tomorrow.
Thanks,
Ryan
On Dec 9, 2008, at 10:36 PM, "Aaron Kimball" <aa...@cloudera.com> wrote:
> Note also that "cat foo | bin/hadoop fs -put - some/hdfs/path" will
> use
> stdin.
> - Aaron
>
> On Mon, Dec 8, 2008 at 5:56 PM, Ryan LeCompte <le...@gmail.com>
> wrote:
>
>> Just what I need -- thanks!
>>
>> On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard <al...@cloudera.com>
>> wrote:
>>> This should answer your questions:
>>>
>>> <http://wiki.apache.org/hadoop/MountableHDFS>
>>>
>>> Alex
>>>
>>> On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <le...@gmail.com>
>> wrote:
>>>
>>>> Hello all,
>>>>
>>>> I normally upload files into hadoop via bin/hadoop fs -put file
>>>> dest.
>>>>
>>>> However, is there a way to somehow stream data into Hadoop?
>>>>
>>>> For example, I'd love to do something like this:
>>>>
>>>> zcat xxx >> HADOOP_HDFS_DESTINATION
>>>>
>>>> This would save me a ton of time since I don't have to first unpack
>>>> the .tgz file and upload the raw file into HDFS.
>>>>
>>>> Is this possible with Hadoop 0.19?
>>>>
>>>> Thanks,
>>>> Ryan
>>>>
>>>
>>
Re: Streaming data into Hadoop
Posted by Aaron Kimball <aa...@cloudera.com>.
Note also that "cat foo | bin/hadoop fs -put - some/hdfs/path" will use
stdin.
- Aaron
On Mon, Dec 8, 2008 at 5:56 PM, Ryan LeCompte <le...@gmail.com> wrote:
> Just what I need -- thanks!
>
> On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard <al...@cloudera.com>
> wrote:
> > This should answer your questions:
> >
> > <http://wiki.apache.org/hadoop/MountableHDFS>
> >
> > Alex
> >
> > On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <le...@gmail.com>
> wrote:
> >
> >> Hello all,
> >>
> >> I normally upload files into hadoop via bin/hadoop fs -put file dest.
> >>
> >> However, is there a way to somehow stream data into Hadoop?
> >>
> >> For example, I'd love to do something like this:
> >>
> >> zcat xxx >> HADOOP_HDFS_DESTINATION
> >>
> >> This would save me a ton of time since I don't have to first unpack
> >> the .tgz file and upload the raw file into HDFS.
> >>
> >> Is this possible with Hadoop 0.19?
> >>
> >> Thanks,
> >> Ryan
> >>
> >
>
Re: Streaming data into Hadoop
Posted by Ryan LeCompte <le...@gmail.com>.
Just what I need -- thanks!
On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard <al...@cloudera.com> wrote:
> This should answer your questions:
>
> <http://wiki.apache.org/hadoop/MountableHDFS>
>
> Alex
>
> On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <le...@gmail.com> wrote:
>
>> Hello all,
>>
>> I normally upload files into hadoop via bin/hadoop fs -put file dest.
>>
>> However, is there a way to somehow stream data into Hadoop?
>>
>> For example, I'd love to do something like this:
>>
>> zcat xxx >> HADOOP_HDFS_DESTINATION
>>
>> This would save me a ton of time since I don't have to first unpack
>> the .tgz file and upload the raw file into HDFS.
>>
>> Is this possible with Hadoop 0.19?
>>
>> Thanks,
>> Ryan
>>
>
Re: Streaming data into Hadoop
Posted by Alex Loddengaard <al...@cloudera.com>.
This should answer your questions:
<http://wiki.apache.org/hadoop/MountableHDFS>
Alex
On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <le...@gmail.com> wrote:
> Hello all,
>
> I normally upload files into hadoop via bin/hadoop fs -put file dest.
>
> However, is there a way to somehow stream data into Hadoop?
>
> For example, I'd love to do something like this:
>
> zcat xxx >> HADOOP_HDFS_DESTINATION
>
> This would save me a ton of time since I don't have to first unpack
> the .tgz file and upload the raw file into HDFS.
>
> Is this possible with Hadoop 0.19?
>
> Thanks,
> Ryan
>