You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Shalish VJ <sh...@yahoo.com> on 2013/07/12 18:50:22 UTC
Hadoop property precedence
Hi,
Suppose block size set in configuration file at client side is 64MB,
block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
Please advice, If the client is writing a file to hdfs,which property would be executed.
Thanks,
Shalish.
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi Please help me on this. Its urgent.
________________________________
From: Shalish VJ <sh...@yahoo.com>
To: hadoop-mailerlist <us...@hadoop.apache.org>
Sent: Friday, July 12, 2013 10:20 PM
Subject: Hadoop property precedence
Hi,
Suppose block size set in configuration file at client side is 64MB,
block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
Please advice, If the client is writing a file to hdfs,which property would be executed.
Thanks,
Shalish.
Re: Hadoop property precedence
Posted by Shekhar Sharma <sh...@gmail.com>.
Check out how does the writing happens on HDFS...
When client issues the command "hadoop fs -put local_source
hdfs_destination", client contact the namenode that he wants to write and
then NameNode creates a blockID and ask three datanodes ( if replication on
the client side is set to 3) to host the replica. THis information is sent
to client and then client starts writing the data on to the data nodes by
forming pipe line..And then client write that much amount of data onto a
block which he has set.
Regards,
Som Shekhar Sharma
+91-8197243810
On Sun, Jul 14, 2013 at 4:24 PM, Harsh J <ha...@cloudera.com> wrote:
> Replication, block size, etc. are all per-file and pure client
> supplied properties. They either take their default from the client
> config, or directly from an API argument override.
>
> On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> > What Shumin told is correct,hadoop configurations has been over written
> > through client application.
> >
> > We have faced similar type of issue,Where default replication factor was
> > mentioned 2 in hadoop configuration.But when when ever the client
> > application writes a files,it was having 3 copies in hadoop
> cluster.Later on
> > checking client application it's default replica factor has 3.
> >
> >
> > On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> >>
> >> I Think the client side configuration will take effect.
> >>
> >> Shumin
> >>
> >> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>>
> >>> Suppose block size set in configuration file at client side is
> 64MB,
> >>> block size set in configuration file at name node side is 128MB and
> block
> >>> size set in configuration file at datanode side is something else.
> >>> Please advice, If the client is writing a file to hdfs,which property
> >>> would be executed.
> >>>
> >>> Thanks,
> >>> Shalish.
> >
> >
> >
> >
> > --
> > Regards,
> > Varun Kumar.P
>
>
>
> --
> Harsh J
>
Re: Hadoop property precedence
Posted by Shekhar Sharma <sh...@gmail.com>.
Check out how does the writing happens on HDFS...
When client issues the command "hadoop fs -put local_source
hdfs_destination", client contact the namenode that he wants to write and
then NameNode creates a blockID and ask three datanodes ( if replication on
the client side is set to 3) to host the replica. THis information is sent
to client and then client starts writing the data on to the data nodes by
forming pipe line..And then client write that much amount of data onto a
block which he has set.
Regards,
Som Shekhar Sharma
+91-8197243810
On Sun, Jul 14, 2013 at 4:24 PM, Harsh J <ha...@cloudera.com> wrote:
> Replication, block size, etc. are all per-file and pure client
> supplied properties. They either take their default from the client
> config, or directly from an API argument override.
>
> On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> > What Shumin told is correct,hadoop configurations has been over written
> > through client application.
> >
> > We have faced similar type of issue,Where default replication factor was
> > mentioned 2 in hadoop configuration.But when when ever the client
> > application writes a files,it was having 3 copies in hadoop
> cluster.Later on
> > checking client application it's default replica factor has 3.
> >
> >
> > On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> >>
> >> I Think the client side configuration will take effect.
> >>
> >> Shumin
> >>
> >> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>>
> >>> Suppose block size set in configuration file at client side is
> 64MB,
> >>> block size set in configuration file at name node side is 128MB and
> block
> >>> size set in configuration file at datanode side is something else.
> >>> Please advice, If the client is writing a file to hdfs,which property
> >>> would be executed.
> >>>
> >>> Thanks,
> >>> Shalish.
> >
> >
> >
> >
> > --
> > Regards,
> > Varun Kumar.P
>
>
>
> --
> Harsh J
>
Re: Hadoop property precedence
Posted by Shekhar Sharma <sh...@gmail.com>.
Check out how does the writing happens on HDFS...
When client issues the command "hadoop fs -put local_source
hdfs_destination", client contact the namenode that he wants to write and
then NameNode creates a blockID and ask three datanodes ( if replication on
the client side is set to 3) to host the replica. THis information is sent
to client and then client starts writing the data on to the data nodes by
forming pipe line..And then client write that much amount of data onto a
block which he has set.
Regards,
Som Shekhar Sharma
+91-8197243810
On Sun, Jul 14, 2013 at 4:24 PM, Harsh J <ha...@cloudera.com> wrote:
> Replication, block size, etc. are all per-file and pure client
> supplied properties. They either take their default from the client
> config, or directly from an API argument override.
>
> On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> > What Shumin told is correct,hadoop configurations has been over written
> > through client application.
> >
> > We have faced similar type of issue,Where default replication factor was
> > mentioned 2 in hadoop configuration.But when when ever the client
> > application writes a files,it was having 3 copies in hadoop
> cluster.Later on
> > checking client application it's default replica factor has 3.
> >
> >
> > On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> >>
> >> I Think the client side configuration will take effect.
> >>
> >> Shumin
> >>
> >> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>>
> >>> Suppose block size set in configuration file at client side is
> 64MB,
> >>> block size set in configuration file at name node side is 128MB and
> block
> >>> size set in configuration file at datanode side is something else.
> >>> Please advice, If the client is writing a file to hdfs,which property
> >>> would be executed.
> >>>
> >>> Thanks,
> >>> Shalish.
> >
> >
> >
> >
> > --
> > Regards,
> > Varun Kumar.P
>
>
>
> --
> Harsh J
>
Re: Hadoop property precedence
Posted by Shekhar Sharma <sh...@gmail.com>.
Check out how does the writing happens on HDFS...
When client issues the command "hadoop fs -put local_source
hdfs_destination", client contact the namenode that he wants to write and
then NameNode creates a blockID and ask three datanodes ( if replication on
the client side is set to 3) to host the replica. THis information is sent
to client and then client starts writing the data on to the data nodes by
forming pipe line..And then client write that much amount of data onto a
block which he has set.
Regards,
Som Shekhar Sharma
+91-8197243810
On Sun, Jul 14, 2013 at 4:24 PM, Harsh J <ha...@cloudera.com> wrote:
> Replication, block size, etc. are all per-file and pure client
> supplied properties. They either take their default from the client
> config, or directly from an API argument override.
>
> On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> > What Shumin told is correct,hadoop configurations has been over written
> > through client application.
> >
> > We have faced similar type of issue,Where default replication factor was
> > mentioned 2 in hadoop configuration.But when when ever the client
> > application writes a files,it was having 3 copies in hadoop
> cluster.Later on
> > checking client application it's default replica factor has 3.
> >
> >
> > On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> >>
> >> I Think the client side configuration will take effect.
> >>
> >> Shumin
> >>
> >> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>>
> >>> Suppose block size set in configuration file at client side is
> 64MB,
> >>> block size set in configuration file at name node side is 128MB and
> block
> >>> size set in configuration file at datanode side is something else.
> >>> Please advice, If the client is writing a file to hdfs,which property
> >>> would be executed.
> >>>
> >>> Thanks,
> >>> Shalish.
> >
> >
> >
> >
> > --
> > Regards,
> > Varun Kumar.P
>
>
>
> --
> Harsh J
>
Re: Hadoop property precedence
Posted by Harsh J <ha...@cloudera.com>.
Replication, block size, etc. are all per-file and pure client
supplied properties. They either take their default from the client
config, or directly from an API argument override.
On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> What Shumin told is correct,hadoop configurations has been over written
> through client application.
>
> We have faced similar type of issue,Where default replication factor was
> mentioned 2 in hadoop configuration.But when when ever the client
> application writes a files,it was having 3 copies in hadoop cluster.Later on
> checking client application it's default replica factor has 3.
>
>
> On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
>>
>> I Think the client side configuration will take effect.
>>
>> Shumin
>>
>> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>>>
>>> Hi,
>>>
>>>
>>> Suppose block size set in configuration file at client side is 64MB,
>>> block size set in configuration file at name node side is 128MB and block
>>> size set in configuration file at datanode side is something else.
>>> Please advice, If the client is writing a file to hdfs,which property
>>> would be executed.
>>>
>>> Thanks,
>>> Shalish.
>
>
>
>
> --
> Regards,
> Varun Kumar.P
--
Harsh J
Re: Hadoop property precedence
Posted by Harsh J <ha...@cloudera.com>.
Replication, block size, etc. are all per-file and pure client
supplied properties. They either take their default from the client
config, or directly from an API argument override.
On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> What Shumin told is correct,hadoop configurations has been over written
> through client application.
>
> We have faced similar type of issue,Where default replication factor was
> mentioned 2 in hadoop configuration.But when when ever the client
> application writes a files,it was having 3 copies in hadoop cluster.Later on
> checking client application it's default replica factor has 3.
>
>
> On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
>>
>> I Think the client side configuration will take effect.
>>
>> Shumin
>>
>> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>>>
>>> Hi,
>>>
>>>
>>> Suppose block size set in configuration file at client side is 64MB,
>>> block size set in configuration file at name node side is 128MB and block
>>> size set in configuration file at datanode side is something else.
>>> Please advice, If the client is writing a file to hdfs,which property
>>> would be executed.
>>>
>>> Thanks,
>>> Shalish.
>
>
>
>
> --
> Regards,
> Varun Kumar.P
--
Harsh J
Re: Hadoop property precedence
Posted by Harsh J <ha...@cloudera.com>.
Replication, block size, etc. are all per-file and pure client
supplied properties. They either take their default from the client
config, or directly from an API argument override.
On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> What Shumin told is correct,hadoop configurations has been over written
> through client application.
>
> We have faced similar type of issue,Where default replication factor was
> mentioned 2 in hadoop configuration.But when when ever the client
> application writes a files,it was having 3 copies in hadoop cluster.Later on
> checking client application it's default replica factor has 3.
>
>
> On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
>>
>> I Think the client side configuration will take effect.
>>
>> Shumin
>>
>> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>>>
>>> Hi,
>>>
>>>
>>> Suppose block size set in configuration file at client side is 64MB,
>>> block size set in configuration file at name node side is 128MB and block
>>> size set in configuration file at datanode side is something else.
>>> Please advice, If the client is writing a file to hdfs,which property
>>> would be executed.
>>>
>>> Thanks,
>>> Shalish.
>
>
>
>
> --
> Regards,
> Varun Kumar.P
--
Harsh J
Re: Hadoop property precedence
Posted by Harsh J <ha...@cloudera.com>.
Replication, block size, etc. are all per-file and pure client
supplied properties. They either take their default from the client
config, or directly from an API argument override.
On Sun, Jul 14, 2013 at 4:14 PM, varun kumar <va...@gmail.com> wrote:
> What Shumin told is correct,hadoop configurations has been over written
> through client application.
>
> We have faced similar type of issue,Where default replication factor was
> mentioned 2 in hadoop configuration.But when when ever the client
> application writes a files,it was having 3 copies in hadoop cluster.Later on
> checking client application it's default replica factor has 3.
>
>
> On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
>>
>> I Think the client side configuration will take effect.
>>
>> Shumin
>>
>> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>>>
>>> Hi,
>>>
>>>
>>> Suppose block size set in configuration file at client side is 64MB,
>>> block size set in configuration file at name node side is 128MB and block
>>> size set in configuration file at datanode side is something else.
>>> Please advice, If the client is writing a file to hdfs,which property
>>> would be executed.
>>>
>>> Thanks,
>>> Shalish.
>
>
>
>
> --
> Regards,
> Varun Kumar.P
--
Harsh J
Re: Hadoop property precedence
Posted by varun kumar <va...@gmail.com>.
What Shumin told is correct,hadoop configurations has been over written
through client application.
We have faced similar type of issue,Where default replication factor was
mentioned 2 in hadoop configuration.But when when ever the client
application writes a files,it was having 3 copies in hadoop cluster.Later
on checking client application it's default replica factor has 3.
On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> I Think the client side configuration will take effect.
>
> Shumin
> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
--
Regards,
Varun Kumar.P
Re: Hadoop property precedence
Posted by varun kumar <va...@gmail.com>.
What Shumin told is correct,hadoop configurations has been over written
through client application.
We have faced similar type of issue,Where default replication factor was
mentioned 2 in hadoop configuration.But when when ever the client
application writes a files,it was having 3 copies in hadoop cluster.Later
on checking client application it's default replica factor has 3.
On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> I Think the client side configuration will take effect.
>
> Shumin
> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
--
Regards,
Varun Kumar.P
Re: Hadoop property precedence
Posted by varun kumar <va...@gmail.com>.
What Shumin told is correct,hadoop configurations has been over written
through client application.
We have faced similar type of issue,Where default replication factor was
mentioned 2 in hadoop configuration.But when when ever the client
application writes a files,it was having 3 copies in hadoop cluster.Later
on checking client application it's default replica factor has 3.
On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> I Think the client side configuration will take effect.
>
> Shumin
> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
--
Regards,
Varun Kumar.P
Re: Hadoop property precedence
Posted by varun kumar <va...@gmail.com>.
What Shumin told is correct,hadoop configurations has been over written
through client application.
We have faced similar type of issue,Where default replication factor was
mentioned 2 in hadoop configuration.But when when ever the client
application writes a files,it was having 3 copies in hadoop cluster.Later
on checking client application it's default replica factor has 3.
On Sun, Jul 14, 2013 at 4:51 AM, Shumin Guo <gs...@gmail.com> wrote:
> I Think the client side configuration will take effect.
>
> Shumin
> On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
--
Regards,
Varun Kumar.P
Re: Hadoop property precedence
Posted by Shumin Guo <gs...@gmail.com>.
I Think the client side configuration will take effect.
Shumin
On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Shumin Guo <gs...@gmail.com>.
I Think the client side configuration will take effect.
Shumin
On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Subroto <ss...@datameer.com>.
Hi Shalish,
The client side conf will take precedence. Further you can use the FileSystem API which can set the block size:
create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
Cheers,
Subroto Sanyal
On Jul 13, 2013, at 9:10 PM, Shalish VJ wrote:
> Hi,
>
> Ru sure?
> or have u ever tried it out.
> Pls advice.
>
> From: Azuryy Yu <az...@gmail.com>
> To: user@hadoop.apache.org
> Sent: Saturday, July 13, 2013 3:19 PM
> Subject: Re: Hadoop property precedence
>
> the conf that client running on will take effect.
> On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property would be executed.
>
> Thanks,
> Shalish.
>
>
>
Re: Hadoop property precedence
Posted by Subroto <ss...@datameer.com>.
Hi Shalish,
The client side conf will take precedence. Further you can use the FileSystem API which can set the block size:
create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
Cheers,
Subroto Sanyal
On Jul 13, 2013, at 9:10 PM, Shalish VJ wrote:
> Hi,
>
> Ru sure?
> or have u ever tried it out.
> Pls advice.
>
> From: Azuryy Yu <az...@gmail.com>
> To: user@hadoop.apache.org
> Sent: Saturday, July 13, 2013 3:19 PM
> Subject: Re: Hadoop property precedence
>
> the conf that client running on will take effect.
> On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property would be executed.
>
> Thanks,
> Shalish.
>
>
>
Re: Hadoop property precedence
Posted by Subroto <ss...@datameer.com>.
Hi Shalish,
The client side conf will take precedence. Further you can use the FileSystem API which can set the block size:
create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
Cheers,
Subroto Sanyal
On Jul 13, 2013, at 9:10 PM, Shalish VJ wrote:
> Hi,
>
> Ru sure?
> or have u ever tried it out.
> Pls advice.
>
> From: Azuryy Yu <az...@gmail.com>
> To: user@hadoop.apache.org
> Sent: Saturday, July 13, 2013 3:19 PM
> Subject: Re: Hadoop property precedence
>
> the conf that client running on will take effect.
> On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property would be executed.
>
> Thanks,
> Shalish.
>
>
>
Re: Hadoop property precedence
Posted by Subroto <ss...@datameer.com>.
Hi Shalish,
The client side conf will take precedence. Further you can use the FileSystem API which can set the block size:
create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
Cheers,
Subroto Sanyal
On Jul 13, 2013, at 9:10 PM, Shalish VJ wrote:
> Hi,
>
> Ru sure?
> or have u ever tried it out.
> Pls advice.
>
> From: Azuryy Yu <az...@gmail.com>
> To: user@hadoop.apache.org
> Sent: Saturday, July 13, 2013 3:19 PM
> Subject: Re: Hadoop property precedence
>
> the conf that client running on will take effect.
> On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property would be executed.
>
> Thanks,
> Shalish.
>
>
>
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi,
Ru sure?
or have u ever tried it out.
Pls advice.
________________________________
From: Azuryy Yu <az...@gmail.com>
To: user@hadoop.apache.org
Sent: Saturday, July 13, 2013 3:19 PM
Subject: Re: Hadoop property precedence
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
Shalish,
>
>The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
>Thanks,
>Kiran
>
>
>
>On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>>block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
>>Please advice, If the client is writing a file to hdfs,which property would be executed.
>>
>>Thanks,
>>Shalish.
>
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi,
Ru sure?
or have u ever tried it out.
Pls advice.
________________________________
From: Azuryy Yu <az...@gmail.com>
To: user@hadoop.apache.org
Sent: Saturday, July 13, 2013 3:19 PM
Subject: Re: Hadoop property precedence
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
Shalish,
>
>The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
>Thanks,
>Kiran
>
>
>
>On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>>block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
>>Please advice, If the client is writing a file to hdfs,which property would be executed.
>>
>>Thanks,
>>Shalish.
>
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi,
Ru sure?
or have u ever tried it out.
Pls advice.
________________________________
From: Azuryy Yu <az...@gmail.com>
To: user@hadoop.apache.org
Sent: Saturday, July 13, 2013 3:19 PM
Subject: Re: Hadoop property precedence
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
Shalish,
>
>The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
>Thanks,
>Kiran
>
>
>
>On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>>block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
>>Please advice, If the client is writing a file to hdfs,which property would be executed.
>>
>>Thanks,
>>Shalish.
>
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi,
Ru sure?
or have u ever tried it out.
Pls advice.
________________________________
From: Azuryy Yu <az...@gmail.com>
To: user@hadoop.apache.org
Sent: Saturday, July 13, 2013 3:19 PM
Subject: Re: Hadoop property precedence
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
Shalish,
>
>The default block size is 64MB which is good at the client end. Make sure the same at your end also in conf. You can increase the size of each block to 128MB or greater than that only thing you can see the processing will be fast but at end there may be chances of losing data.
>
>Thanks,
>Kiran
>
>
>
>On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>>block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
>>Please advice, If the client is writing a file to hdfs,which property would be executed.
>>
>>Thanks,
>>Shalish.
>
Re: Hadoop property precedence
Posted by Azuryy Yu <az...@gmail.com>.
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure
> the same at your end also in conf. You can increase the size of each block
> to 128MB or greater than that only thing you can see the processing will be
> fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
>
Re: Hadoop property precedence
Posted by Azuryy Yu <az...@gmail.com>.
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure
> the same at your end also in conf. You can increase the size of each block
> to 128MB or greater than that only thing you can see the processing will be
> fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
>
Re: Hadoop property precedence
Posted by Azuryy Yu <az...@gmail.com>.
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure
> the same at your end also in conf. You can increase the size of each block
> to 128MB or greater than that only thing you can see the processing will be
> fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
>
Re: Hadoop property precedence
Posted by Azuryy Yu <az...@gmail.com>.
the conf that client running on will take effect.
On Jul 13, 2013 4:42 PM, "Kiran Dangeti" <ki...@gmail.com> wrote:
> Shalish,
>
> The default block size is 64MB which is good at the client end. Make sure
> the same at your end also in conf. You can increase the size of each block
> to 128MB or greater than that only thing you can see the processing will be
> fast but at end there may be chances of losing data.
>
> Thanks,
> Kiran
>
>
> On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
>
>> Hi,
>>
>>
>> Suppose block size set in configuration file at client side is 64MB,
>> block size set in configuration file at name node side is 128MB and block
>> size set in configuration file at datanode side is something else.
>> Please advice, If the client is writing a file to hdfs,which property
>> would be executed.
>>
>> Thanks,
>> Shalish.
>>
>
>
Re: Hadoop property precedence
Posted by Kiran Dangeti <ki...@gmail.com>.
Shalish,
The default block size is 64MB which is good at the client end. Make sure
the same at your end also in conf. You can increase the size of each block
to 128MB or greater than that only thing you can see the processing will be
fast but at end there may be chances of losing data.
Thanks,
Kiran
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi Please help me on this. Its urgent.
________________________________
From: Shalish VJ <sh...@yahoo.com>
To: hadoop-mailerlist <us...@hadoop.apache.org>
Sent: Friday, July 12, 2013 10:20 PM
Subject: Hadoop property precedence
Hi,
Suppose block size set in configuration file at client side is 64MB,
block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
Please advice, If the client is writing a file to hdfs,which property would be executed.
Thanks,
Shalish.
Re: Hadoop property precedence
Posted by Raj K Singh <ra...@gmail.com>.
I think it will take 128MB of file size which is given in namenode config
file but nt sure not even try it.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Shumin Guo <gs...@gmail.com>.
I Think the client side configuration will take effect.
Shumin
On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Kiran Dangeti <ki...@gmail.com>.
Shalish,
The default block size is 64MB which is good at the client end. Make sure
the same at your end also in conf. You can increase the size of each block
to 128MB or greater than that only thing you can see the processing will be
fast but at end there may be chances of losing data.
Thanks,
Kiran
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Kiran Dangeti <ki...@gmail.com>.
Shalish,
The default block size is 64MB which is good at the client end. Make sure
the same at your end also in conf. You can increase the size of each block
to 128MB or greater than that only thing you can see the processing will be
fast but at end there may be chances of losing data.
Thanks,
Kiran
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi Please help me on this. Its urgent.
________________________________
From: Shalish VJ <sh...@yahoo.com>
To: hadoop-mailerlist <us...@hadoop.apache.org>
Sent: Friday, July 12, 2013 10:20 PM
Subject: Hadoop property precedence
Hi,
Suppose block size set in configuration file at client side is 64MB,
block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
Please advice, If the client is writing a file to hdfs,which property would be executed.
Thanks,
Shalish.
Re: Hadoop property precedence
Posted by Kiran Dangeti <ki...@gmail.com>.
Shalish,
The default block size is 64MB which is good at the client end. Make sure
the same at your end also in conf. You can increase the size of each block
to 128MB or greater than that only thing you can see the processing will be
fast but at end there may be chances of losing data.
Thanks,
Kiran
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Shalish VJ <sh...@yahoo.com>.
Hi Please help me on this. Its urgent.
________________________________
From: Shalish VJ <sh...@yahoo.com>
To: hadoop-mailerlist <us...@hadoop.apache.org>
Sent: Friday, July 12, 2013 10:20 PM
Subject: Hadoop property precedence
Hi,
Suppose block size set in configuration file at client side is 64MB,
block size set in configuration file at name node side is 128MB and block size set in configuration file at datanode side is something else.
Please advice, If the client is writing a file to hdfs,which property would be executed.
Thanks,
Shalish.
Re: Hadoop property precedence
Posted by Raj K Singh <ra...@gmail.com>.
I think it will take 128MB of file size which is given in namenode config
file but nt sure not even try it.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Raj K Singh <ra...@gmail.com>.
I think it will take 128MB of file size which is given in namenode config
file but nt sure not even try it.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Shumin Guo <gs...@gmail.com>.
I Think the client side configuration will take effect.
Shumin
On Jul 12, 2013 11:50 AM, "Shalish VJ" <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>
Re: Hadoop property precedence
Posted by Raj K Singh <ra...@gmail.com>.
I think it will take 128MB of file size which is given in namenode config
file but nt sure not even try it.
::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile Tel: +91 (0)9899821370
On Fri, Jul 12, 2013 at 10:20 PM, Shalish VJ <sh...@yahoo.com> wrote:
> Hi,
>
>
> Suppose block size set in configuration file at client side is 64MB,
> block size set in configuration file at name node side is 128MB and block
> size set in configuration file at datanode side is something else.
> Please advice, If the client is writing a file to hdfs,which property
> would be executed.
>
> Thanks,
> Shalish.
>