You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Vivi Lang <sq...@gmail.com> on 2012/11/09 00:15:18 UTC
Create a data block in a certain data node
Hi all,
If I want have get a inputstream in a datanode (and this should contains
exactly 64MB data which I want to replace the content of an existing data
block to store that content), can anyone tell me which is the easiest way
to do this? Moreover, I would like to write this stream in the local
datanode.
Thanks,
Re: Create a data block in a certain data node
Posted by Vivi Lang <sq...@gmail.com>.
Yes, I know the which block is corrupted and want to rewrite the content of
the block. (asuming the replication factor of the related file is 1)
On Fri, Nov 9, 2012 at 12:33 PM, Steve Loughran <st...@hortonworks.com>wrote:
> On 9 November 2012 17:37, Vivi Lang <sq...@gmail.com> wrote:
>
> > thxs for the replying,
> > what i want to do is, for example, I have a file which already in hdfs,
> > this file have lots of blocks, but I know some parts( and the related
> > offset ) of this file is corrupted and I want to rewrite only a few
> blocks
> > (most likely only one block) instead of all, I am wondering is there any
> > convenient way to do this replacement
> >
> > Thanks again!
> >
> >
> Is the situation here that you have corrupted HDFS blocks?
>
Re: Create a data block in a certain data node
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 November 2012 17:37, Vivi Lang <sq...@gmail.com> wrote:
> thxs for the replying,
> what i want to do is, for example, I have a file which already in hdfs,
> this file have lots of blocks, but I know some parts( and the related
> offset ) of this file is corrupted and I want to rewrite only a few blocks
> (most likely only one block) instead of all, I am wondering is there any
> convenient way to do this replacement
>
> Thanks again!
>
>
Is the situation here that you have corrupted HDFS blocks?
Re: Create a data block in a certain data node
Posted by Vivi Lang <sq...@gmail.com>.
thxs for the replying,
what i want to do is, for example, I have a file which already in hdfs,
this file have lots of blocks, but I know some parts( and the related
offset ) of this file is corrupted and I want to rewrite only a few blocks
(most likely only one block) instead of all, I am wondering is there any
convenient way to do this replacement
Thanks again!
On Friday, November 9, 2012, Harsh J wrote:
> Hi Vivi,
>
> I don't think HDFS offers any such API to do this today. What is your
> use-case for needing this (replacing a given existing block with
> another, arbitrary block?) though?
>
> On Fri, Nov 9, 2012 at 7:19 AM, Vivi Lang <sqlxweiwei@gmail.com<javascript:;>>
> wrote:
> > Sorry, I found the previous one is full of typo and hard to understand. I
> > will clarify my question:
> >
> > If I have got a inputstream in a datanode (and this stream contains
> exactly
> > 64MB's data which I want to store it as a data block of the HDFS in that
> > datanode. More specifically, I want to use this block to replace another
> > exsiting block of one file that already stored in the HDFS), can anyone
> > tell me which is the easiest way to do this?
> >
> > Thanks,
> >
> >
> > On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <sqlxweiwei@gmail.com<javascript:;>>
> wrote:
> >
> >> Hi all,
> >>
> >> If I want have get a inputstream in a datanode (and this should
> contains
> >> exactly 64MB data which I want to replace the content of an existing
> data
> >> block to store that content), can anyone tell me which is the easiest
> way
> >> to do this? Moreover, I would like to write this stream in the local
> >> datanode.
> >>
> >> Thanks,
> >>
>
>
>
> --
> Harsh J
>
Re: Create a data block in a certain data node
Posted by Steve Loughran <st...@hortonworks.com>.
On 9 November 2012 13:29, Binglin Chang <de...@gmail.com> wrote:
> If you just need some experiment setup,
> just set dfs.replication to 1, and using dfsclient on the same datenode to
> write data, the block is guaranteed to be the same datanode.
>
There's no guarantees: if there is no space on the local DN it will be
moved off-server.
Re: Create a data block in a certain data node
Posted by Binglin Chang <de...@gmail.com>.
If you just need some experiment setup,
just set dfs.replication to 1, and using dfsclient on the same datenode to
write data, the block is guaranteed to be the same datanode.
On Fri, Nov 9, 2012 at 7:13 PM, Harsh J <ha...@cloudera.com> wrote:
> Hi Vivi,
>
> I don't think HDFS offers any such API to do this today. What is your
> use-case for needing this (replacing a given existing block with
> another, arbitrary block?) though?
>
> On Fri, Nov 9, 2012 at 7:19 AM, Vivi Lang <sq...@gmail.com> wrote:
> > Sorry, I found the previous one is full of typo and hard to understand. I
> > will clarify my question:
> >
> > If I have got a inputstream in a datanode (and this stream contains
> exactly
> > 64MB's data which I want to store it as a data block of the HDFS in that
> > datanode. More specifically, I want to use this block to replace another
> > exsiting block of one file that already stored in the HDFS), can anyone
> > tell me which is the easiest way to do this?
> >
> > Thanks,
> >
> >
> > On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <sq...@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> If I want have get a inputstream in a datanode (and this should
> contains
> >> exactly 64MB data which I want to replace the content of an existing
> data
> >> block to store that content), can anyone tell me which is the easiest
> way
> >> to do this? Moreover, I would like to write this stream in the local
> >> datanode.
> >>
> >> Thanks,
> >>
>
>
>
> --
> Harsh J
>
Re: Create a data block in a certain data node
Posted by Harsh J <ha...@cloudera.com>.
Hi Vivi,
I don't think HDFS offers any such API to do this today. What is your
use-case for needing this (replacing a given existing block with
another, arbitrary block?) though?
On Fri, Nov 9, 2012 at 7:19 AM, Vivi Lang <sq...@gmail.com> wrote:
> Sorry, I found the previous one is full of typo and hard to understand. I
> will clarify my question:
>
> If I have got a inputstream in a datanode (and this stream contains exactly
> 64MB's data which I want to store it as a data block of the HDFS in that
> datanode. More specifically, I want to use this block to replace another
> exsiting block of one file that already stored in the HDFS), can anyone
> tell me which is the easiest way to do this?
>
> Thanks,
>
>
> On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <sq...@gmail.com> wrote:
>
>> Hi all,
>>
>> If I want have get a inputstream in a datanode (and this should contains
>> exactly 64MB data which I want to replace the content of an existing data
>> block to store that content), can anyone tell me which is the easiest way
>> to do this? Moreover, I would like to write this stream in the local
>> datanode.
>>
>> Thanks,
>>
--
Harsh J
Re: Create a data block in a certain data node
Posted by Vivi Lang <sq...@gmail.com>.
Sorry, I found the previous one is full of typo and hard to understand. I
will clarify my question:
If I have got a inputstream in a datanode (and this stream contains exactly
64MB's data which I want to store it as a data block of the HDFS in that
datanode. More specifically, I want to use this block to replace another
exsiting block of one file that already stored in the HDFS), can anyone
tell me which is the easiest way to do this?
Thanks,
On Thu, Nov 8, 2012 at 5:15 PM, Vivi Lang <sq...@gmail.com> wrote:
> Hi all,
>
> If I want have get a inputstream in a datanode (and this should contains
> exactly 64MB data which I want to replace the content of an existing data
> block to store that content), can anyone tell me which is the easiest way
> to do this? Moreover, I would like to write this stream in the local
> datanode.
>
> Thanks,
>