You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Steve Loughran <st...@hortonworks.com> on 2016/09/23 08:52:28 UTC
Fwd: HADOOP-13636 and io.bytes.per.checksum
I got silence from HDFS dev here, so I'm raising it on common dev.
Why is HDFS tagging as deprecated " io.bytes.per.checksum ", given its an option being set in core-default, and used by other filesystems?
>INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1182)) - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
I don't see why it should be deprecated. If it isn't what HDFS likes, then the code there could be smarter: look for the dfs value, and if not found then grab the io.bytes one —warning the user.
I will volunteer to write this code if I get a promise that someone agrees with the premise and is willing to help nurture it in.
Begin forwarded message:
From: Steve Loughran <st...@hortonworks.com>>
Subject: HADOOP-13636 and io.bytes.per.checksum
Date: 21 September 2016 at 17:12:00 BST
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
I'm getting told off for using the deprecated option: io.bytes.per.checksum
https://issues.apache.org/jira/browse/HADOOP-13636
Except: I'm not. FileSystem.getServerDefaults() is, which is used by Trash to work out where to delete things.
It strikes me that the system is inconsitent: HdfsConfiguration is deprecating a property that everything else is happy to use; I see it in four places in production, and various tests, plus core-default.xml
Is it really deprecated? If so, are there any volunteers to remove it from the codebase, while pulling up the default value into core-default?
otherwise: how about the complaint is turned off?
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 24, 2016, at 5:11 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 24, 2016, at 4:24 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>>
>> On 23 Sep 2016, at 18:55, Andrew Wang <an...@cloudera.com>> wrote:
>>
>> Have you git blamed to dig up the original JIRA conversation? I think that deprecation predates many of us, so you might not get much historical perspective from the mailing list.
>>
>> its so old that it's lost in the history of the "show git history for selection" feature in IDEA which does exactly that.
>>
>> I'm happy to lend a +1 though, since like you said, it doesn't seem like that config key is going anywhere.
>>
>
> dfs.bytes-per-checksum was added as part of HADOOP-6419 / 00de6db0e3976cf0108010ecbd6e73a5ceab7b1f. It looks like the internal-to-Yahoo! commit that was part of that one accidentally included a lot more than what was documented. So if there is any ASF-level documentation of the change, it's going to be in a JIRA patch somewhere, likely in the 0.20 timeline.
>
OK neat. It was added then, but not actually used until HDFS-631.... which you'll find in the old hadoop-hdfs tree as part of 0.21. So it was trapped in the project split... which is why it's mainly stuff outside of HDFS that is still using it. I have a hunch that HDFS-2000 was basically misguided(?): it was deprecated for HDFS's internal usage but needed in common for non-HDFS file systems to set. So it's not REALLY deprecated in the classic sense.
Definitely need better documentation here. :/
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 24, 2016, at 5:11 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 24, 2016, at 4:24 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>>
>> On 23 Sep 2016, at 18:55, Andrew Wang <an...@cloudera.com>> wrote:
>>
>> Have you git blamed to dig up the original JIRA conversation? I think that deprecation predates many of us, so you might not get much historical perspective from the mailing list.
>>
>> its so old that it's lost in the history of the "show git history for selection" feature in IDEA which does exactly that.
>>
>> I'm happy to lend a +1 though, since like you said, it doesn't seem like that config key is going anywhere.
>>
>
> dfs.bytes-per-checksum was added as part of HADOOP-6419 / 00de6db0e3976cf0108010ecbd6e73a5ceab7b1f. It looks like the internal-to-Yahoo! commit that was part of that one accidentally included a lot more than what was documented. So if there is any ASF-level documentation of the change, it's going to be in a JIRA patch somewhere, likely in the 0.20 timeline.
>
OK neat. It was added then, but not actually used until HDFS-631.... which you'll find in the old hadoop-hdfs tree as part of 0.21. So it was trapped in the project split... which is why it's mainly stuff outside of HDFS that is still using it. I have a hunch that HDFS-2000 was basically misguided(?): it was deprecated for HDFS's internal usage but needed in common for non-HDFS file systems to set. So it's not REALLY deprecated in the classic sense.
Definitely need better documentation here. :/
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 24, 2016, at 4:24 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
>
> On 23 Sep 2016, at 18:55, Andrew Wang <an...@cloudera.com>> wrote:
>
> Have you git blamed to dig up the original JIRA conversation? I think that deprecation predates many of us, so you might not get much historical perspective from the mailing list.
>
> its so old that it's lost in the history of the "show git history for selection" feature in IDEA which does exactly that.
>
> I'm happy to lend a +1 though, since like you said, it doesn't seem like that config key is going anywhere.
>
dfs.bytes-per-checksum was added as part of HADOOP-6419 / 00de6db0e3976cf0108010ecbd6e73a5ceab7b1f. It looks like the internal-to-Yahoo! commit that was part of that one accidentally included a lot more than what was documented. So if there is any ASF-level documentation of the change, it's going to be in a JIRA patch somewhere, likely in the 0.20 timeline.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 24, 2016, at 4:24 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
>
> On 23 Sep 2016, at 18:55, Andrew Wang <an...@cloudera.com>> wrote:
>
> Have you git blamed to dig up the original JIRA conversation? I think that deprecation predates many of us, so you might not get much historical perspective from the mailing list.
>
> its so old that it's lost in the history of the "show git history for selection" feature in IDEA which does exactly that.
>
> I'm happy to lend a +1 though, since like you said, it doesn't seem like that config key is going anywhere.
>
dfs.bytes-per-checksum was added as part of HADOOP-6419 / 00de6db0e3976cf0108010ecbd6e73a5ceab7b1f. It looks like the internal-to-Yahoo! commit that was part of that one accidentally included a lot more than what was documented. So if there is any ASF-level documentation of the change, it's going to be in a JIRA patch somewhere, likely in the 0.20 timeline.
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Steve Loughran <st...@hortonworks.com>.
On 23 Sep 2016, at 18:55, Andrew Wang <an...@cloudera.com>> wrote:
Have you git blamed to dig up the original JIRA conversation? I think that deprecation predates many of us, so you might not get much historical perspective from the mailing list.
its so old that it's lost in the history of the "show git history for selection" feature in IDEA which does exactly that.
I'm happy to lend a +1 though, since like you said, it doesn't seem like that config key is going anywhere.
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Steve Loughran <st...@hortonworks.com>.
On 23 Sep 2016, at 18:55, Andrew Wang <an...@cloudera.com>> wrote:
Have you git blamed to dig up the original JIRA conversation? I think that deprecation predates many of us, so you might not get much historical perspective from the mailing list.
its so old that it's lost in the history of the "show git history for selection" feature in IDEA which does exactly that.
I'm happy to lend a +1 though, since like you said, it doesn't seem like that config key is going anywhere.
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Andrew Wang <an...@cloudera.com>.
Have you git blamed to dig up the original JIRA conversation? I think that
deprecation predates many of us, so you might not get much historical
perspective from the mailing list.
I'm happy to lend a +1 though, since like you said, it doesn't seem like
that config key is going anywhere.
On Fri, Sep 23, 2016 at 1:52 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> I got silence from HDFS dev here, so I'm raising it on common dev.
>
> Why is HDFS tagging as deprecated " io.bytes.per.checksum ", given its an
> option being set in core-default, and used by other filesystems?
>
>
> >INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1182))
> - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
>
> I don't see why it should be deprecated. If it isn't what HDFS likes, then
> the code there could be smarter: look for the dfs value, and if not found
> then grab the io.bytes one —warning the user.
>
> I will volunteer to write this code if I get a promise that someone agrees
> with the premise and is willing to help nurture it in.
>
> Begin forwarded message:
>
> From: Steve Loughran <stevel@hortonworks.com<mailto:stevel@hortonworks.com
> >>
> Subject: HADOOP-13636 and io.bytes.per.checksum
> Date: 21 September 2016 at 17:12:00 BST
> To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>
>
> I'm getting told off for using the deprecated option: io.bytes.per.checksum
>
> https://issues.apache.org/jira/browse/HADOOP-13636
>
> Except: I'm not. FileSystem.getServerDefaults() is, which is used by Trash
> to work out where to delete things.
>
> It strikes me that the system is inconsitent: HdfsConfiguration is
> deprecating a property that everything else is happy to use; I see it in
> four places in production, and various tests, plus core-default.xml
>
> Is it really deprecated? If so, are there any volunteers to remove it from
> the codebase, while pulling up the default value into core-default?
>
> otherwise: how about the complaint is turned off?
>
>
Re: HADOOP-13636 and io.bytes.per.checksum
Posted by Andrew Wang <an...@cloudera.com>.
Have you git blamed to dig up the original JIRA conversation? I think that
deprecation predates many of us, so you might not get much historical
perspective from the mailing list.
I'm happy to lend a +1 though, since like you said, it doesn't seem like
that config key is going anywhere.
On Fri, Sep 23, 2016 at 1:52 AM, Steve Loughran <st...@hortonworks.com>
wrote:
> I got silence from HDFS dev here, so I'm raising it on common dev.
>
> Why is HDFS tagging as deprecated " io.bytes.per.checksum ", given its an
> option being set in core-default, and used by other filesystems?
>
>
> >INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1182))
> - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
>
> I don't see why it should be deprecated. If it isn't what HDFS likes, then
> the code there could be smarter: look for the dfs value, and if not found
> then grab the io.bytes one —warning the user.
>
> I will volunteer to write this code if I get a promise that someone agrees
> with the premise and is willing to help nurture it in.
>
> Begin forwarded message:
>
> From: Steve Loughran <stevel@hortonworks.com<mailto:stevel@hortonworks.com
> >>
> Subject: HADOOP-13636 and io.bytes.per.checksum
> Date: 21 September 2016 at 17:12:00 BST
> To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <
> hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>>
>
> I'm getting told off for using the deprecated option: io.bytes.per.checksum
>
> https://issues.apache.org/jira/browse/HADOOP-13636
>
> Except: I'm not. FileSystem.getServerDefaults() is, which is used by Trash
> to work out where to delete things.
>
> It strikes me that the system is inconsitent: HdfsConfiguration is
> deprecating a property that everything else is happy to use; I see it in
> four places in production, and various tests, plus core-default.xml
>
> Is it really deprecated? If so, are there any volunteers to remove it from
> the codebase, while pulling up the default value into core-default?
>
> otherwise: how about the complaint is turned off?
>
>