You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Haohui Mai <hm...@hortonworks.com> on 2014/12/09 06:57:55 UTC
Upgrading findbugs
Hi,
The recent changes on moving to Java 7 triggers a bug in findbug (
http://sourceforge.net/p/findbugs/bugs/918), which causes all pre-commit
runs (e.g., HADOOP-11287) to fail.
The current version of findbugs (1.3.9) used by Hadoop is released in 2009.
Given that:
(1) The current bug that we hit are fixed by a later version of findbug.
(2) A newer findbug (3.0.0) is required to analyze Hadoop that is compiled
against Java 8.
(3) Newer findbugs are capable of catching more bugs. :-)
Is it a good time to consider upgrading findbugs, which gives us better
tools on ensuring the quality of the code case?
I ran findbugs 3.0.0 against trunk today. It reported 111 warnings for
hadoop-common, 44 for HDFS and 40+ for YARN. Many of them are possible
NPEs, resource leaks, and ignored exception which are indeed bugs and are
worthwhile to address.
However, one issue that needs to be considered is that how to deal with the
additional warnings reported by the newer findbugs without breaking the
Jenkins pre-commit runs.
Personally I can see three possible routes if we decide to upgrade findbugs:
(1) Fix all warnings before upgrading to newer findbugs.
(2) Add all new warnings to the exclude list and fix them slowly.
(3) Update test-patch.sh to make sure that new code won't introduce any new
findbugs warnings.
I proposed upgrading to findbugs 2.0.2 and fixing new warnings in
HADOOP-10476, which could be dated backed to April, 2014. I volunteer to
accelerate the effort if it is required.
Thoughts?
Regards,
Haohui
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Zhijie Shen <zs...@hortonworks.com>.
Findbugs warnings are cleaned up at YARN side. Thanks for the contribution
from Li Lu and Varun Saxena!
On Thu, Dec 18, 2014 at 2:44 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> So far we made great progress on fixing findbugs warnings. We're free of
> findbugs warnings in hdfs, nfs, and a couple other sub projects. There are
> two findbugs warnings left in hadoop-common. I saw there are some
> progresses on the YARN side as well.
>
> Thanks very much for the contributors (Brandon Li, Li Lu, and many others)
> that have worked on this. We have a cleaner code base now, and the newer
> findbugs can help us to catch more issues during pre-commits. :-)
>
> I plan to finish the remaining work on in 2.7 timeframe. Thanks again for
> contribution.
>
> Thanks,
> Haohui
>
>
> On Tue, Dec 9, 2014 at 2:16 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
> >
> > +1 to upgrade.
> >
> > regarding the newly surfacing issues, I'd recommend we look at them and
> see
> > which are critical problems and fix them.
> >
> > One of the conclusions I got from the building is that there are a lot of
> > javac and javadoc warnings that everyone ignores. Sitting down to fix
> them
> > is time consuming and doesn't directly fix anything or add new features
> > —but it keeps the code cleaner, which is something we want to encourage
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Zhijie Shen <zs...@hortonworks.com>.
Findbugs warnings are cleaned up at YARN side. Thanks for the contribution
from Li Lu and Varun Saxena!
On Thu, Dec 18, 2014 at 2:44 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> So far we made great progress on fixing findbugs warnings. We're free of
> findbugs warnings in hdfs, nfs, and a couple other sub projects. There are
> two findbugs warnings left in hadoop-common. I saw there are some
> progresses on the YARN side as well.
>
> Thanks very much for the contributors (Brandon Li, Li Lu, and many others)
> that have worked on this. We have a cleaner code base now, and the newer
> findbugs can help us to catch more issues during pre-commits. :-)
>
> I plan to finish the remaining work on in 2.7 timeframe. Thanks again for
> contribution.
>
> Thanks,
> Haohui
>
>
> On Tue, Dec 9, 2014 at 2:16 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
> >
> > +1 to upgrade.
> >
> > regarding the newly surfacing issues, I'd recommend we look at them and
> see
> > which are critical problems and fix them.
> >
> > One of the conclusions I got from the building is that there are a lot of
> > javac and javadoc warnings that everyone ignores. Sitting down to fix
> them
> > is time consuming and doesn't directly fix anything or add new features
> > —but it keeps the code cleaner, which is something we want to encourage
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Zhijie Shen <zs...@hortonworks.com>.
Findbugs warnings are cleaned up at YARN side. Thanks for the contribution
from Li Lu and Varun Saxena!
On Thu, Dec 18, 2014 at 2:44 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> So far we made great progress on fixing findbugs warnings. We're free of
> findbugs warnings in hdfs, nfs, and a couple other sub projects. There are
> two findbugs warnings left in hadoop-common. I saw there are some
> progresses on the YARN side as well.
>
> Thanks very much for the contributors (Brandon Li, Li Lu, and many others)
> that have worked on this. We have a cleaner code base now, and the newer
> findbugs can help us to catch more issues during pre-commits. :-)
>
> I plan to finish the remaining work on in 2.7 timeframe. Thanks again for
> contribution.
>
> Thanks,
> Haohui
>
>
> On Tue, Dec 9, 2014 at 2:16 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
> >
> > +1 to upgrade.
> >
> > regarding the newly surfacing issues, I'd recommend we look at them and
> see
> > which are critical problems and fix them.
> >
> > One of the conclusions I got from the building is that there are a lot of
> > javac and javadoc warnings that everyone ignores. Sitting down to fix
> them
> > is time consuming and doesn't directly fix anything or add new features
> > —but it keeps the code cleaner, which is something we want to encourage
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Zhijie Shen <zs...@hortonworks.com>.
Findbugs warnings are cleaned up at YARN side. Thanks for the contribution
from Li Lu and Varun Saxena!
On Thu, Dec 18, 2014 at 2:44 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> So far we made great progress on fixing findbugs warnings. We're free of
> findbugs warnings in hdfs, nfs, and a couple other sub projects. There are
> two findbugs warnings left in hadoop-common. I saw there are some
> progresses on the YARN side as well.
>
> Thanks very much for the contributors (Brandon Li, Li Lu, and many others)
> that have worked on this. We have a cleaner code base now, and the newer
> findbugs can help us to catch more issues during pre-commits. :-)
>
> I plan to finish the remaining work on in 2.7 timeframe. Thanks again for
> contribution.
>
> Thanks,
> Haohui
>
>
> On Tue, Dec 9, 2014 at 2:16 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
> >
> > +1 to upgrade.
> >
> > regarding the newly surfacing issues, I'd recommend we look at them and
> see
> > which are critical problems and fix them.
> >
> > One of the conclusions I got from the building is that there are a lot of
> > javac and javadoc warnings that everyone ignores. Sitting down to fix
> them
> > is time consuming and doesn't directly fix anything or add new features
> > —but it keeps the code cleaner, which is something we want to encourage
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Haohui Mai <hm...@hortonworks.com>.
So far we made great progress on fixing findbugs warnings. We're free of
findbugs warnings in hdfs, nfs, and a couple other sub projects. There are
two findbugs warnings left in hadoop-common. I saw there are some
progresses on the YARN side as well.
Thanks very much for the contributors (Brandon Li, Li Lu, and many others)
that have worked on this. We have a cleaner code base now, and the newer
findbugs can help us to catch more issues during pre-commits. :-)
I plan to finish the remaining work on in 2.7 timeframe. Thanks again for
contribution.
Thanks,
Haohui
On Tue, Dec 9, 2014 at 2:16 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> +1 to upgrade.
>
> regarding the newly surfacing issues, I'd recommend we look at them and see
> which are critical problems and fix them.
>
> One of the conclusions I got from the building is that there are a lot of
> javac and javadoc warnings that everyone ignores. Sitting down to fix them
> is time consuming and doesn't directly fix anything or add new features
> —but it keeps the code cleaner, which is something we want to encourage
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Haohui Mai <hm...@hortonworks.com>.
So far we made great progress on fixing findbugs warnings. We're free of
findbugs warnings in hdfs, nfs, and a couple other sub projects. There are
two findbugs warnings left in hadoop-common. I saw there are some
progresses on the YARN side as well.
Thanks very much for the contributors (Brandon Li, Li Lu, and many others)
that have worked on this. We have a cleaner code base now, and the newer
findbugs can help us to catch more issues during pre-commits. :-)
I plan to finish the remaining work on in 2.7 timeframe. Thanks again for
contribution.
Thanks,
Haohui
On Tue, Dec 9, 2014 at 2:16 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> +1 to upgrade.
>
> regarding the newly surfacing issues, I'd recommend we look at them and see
> which are critical problems and fix them.
>
> One of the conclusions I got from the building is that there are a lot of
> javac and javadoc warnings that everyone ignores. Sitting down to fix them
> is time consuming and doesn't directly fix anything or add new features
> —but it keeps the code cleaner, which is something we want to encourage
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Steve Loughran <st...@hortonworks.com>.
+1 to upgrade.
regarding the newly surfacing issues, I'd recommend we look at them and see
which are critical problems and fix them.
One of the conclusions I got from the building is that there are a lot of
javac and javadoc warnings that everyone ignores. Sitting down to fix them
is time consuming and doesn't directly fix anything or add new features
—but it keeps the code cleaner, which is something we want to encourage
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Karthik Kambatla <ka...@cloudera.com>.
Thanks for initiating this, Haohui. +1 to upgrading findbugs version.
Inline.
On Mon, Dec 8, 2014 at 9:57 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> Hi,
>
> The recent changes on moving to Java 7 triggers a bug in findbug (
> http://sourceforge.net/p/findbugs/bugs/918), which causes all pre-commit
> runs (e.g., HADOOP-11287) to fail.
>
> The current version of findbugs (1.3.9) used by Hadoop is released in 2009.
> Given that:
>
> (1) The current bug that we hit are fixed by a later version of findbug.
> (2) A newer findbug (3.0.0) is required to analyze Hadoop that is compiled
> against Java 8.
> (3) Newer findbugs are capable of catching more bugs. :-)
>
> Is it a good time to consider upgrading findbugs, which gives us better
> tools on ensuring the quality of the code case?
>
> I ran findbugs 3.0.0 against trunk today. It reported 111 warnings for
> hadoop-common, 44 for HDFS and 40+ for YARN. Many of them are possible
> NPEs, resource leaks, and ignored exception which are indeed bugs and are
> worthwhile to address.
>
> However, one issue that needs to be considered is that how to deal with the
> additional warnings reported by the newer findbugs without breaking the
> Jenkins pre-commit runs.
>
> Personally I can see three possible routes if we decide to upgrade
> findbugs:
>
> (1) Fix all warnings before upgrading to newer findbugs.
>
This might take a while. We might want to use the newer findbugs sooner?
> (2) Add all new warnings to the exclude list and fix them slowly.
>
I have my doubts on how soon we fix these warnings unless we make the
associated JIRAs (assuming we have one per exclude) blockers for the next
release. A findbugs "Fix It" day would be ideal to get this done.
> (3) Update test-patch.sh to make sure that new code won't introduce any new
> findbugs warnings.
>
Seems the best, especially if test-patch.sh shows the warnings, but doesn't
-1 unless there are new findbugs warnings. This way, the contributor can
choose to fix "related" warnings at the least.
>
> I proposed upgrading to findbugs 2.0.2 and fixing new warnings in
> HADOOP-10476, which could be dated backed to April, 2014. I volunteer to
> accelerate the effort if it is required.
> Thoughts?
>
> Regards,
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es
Re: Upgrading findbugs
Posted by Steve Loughran <st...@hortonworks.com>.
+1 to upgrade.
regarding the newly surfacing issues, I'd recommend we look at them and see
which are critical problems and fix them.
One of the conclusions I got from the building is that there are a lot of
javac and javadoc warnings that everyone ignores. Sitting down to fix them
is time consuming and doesn't directly fix anything or add new features
—but it keeps the code cleaner, which is something we want to encourage
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Steve Loughran <st...@hortonworks.com>.
+1 to upgrade.
regarding the newly surfacing issues, I'd recommend we look at them and see
which are critical problems and fix them.
One of the conclusions I got from the building is that there are a lot of
javac and javadoc warnings that everyone ignores. Sitting down to fix them
is time consuming and doesn't directly fix anything or add new features
—but it keeps the code cleaner, which is something we want to encourage
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Steve Loughran <st...@hortonworks.com>.
+1 to upgrade.
regarding the newly surfacing issues, I'd recommend we look at them and see
which are critical problems and fix them.
One of the conclusions I got from the building is that there are a lot of
javac and javadoc warnings that everyone ignores. Sitting down to fix them
is time consuming and doesn't directly fix anything or add new features
—but it keeps the code cleaner, which is something we want to encourage
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Upgrading findbugs
Posted by Karthik Kambatla <ka...@cloudera.com>.
Thanks for initiating this, Haohui. +1 to upgrading findbugs version.
Inline.
On Mon, Dec 8, 2014 at 9:57 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> Hi,
>
> The recent changes on moving to Java 7 triggers a bug in findbug (
> http://sourceforge.net/p/findbugs/bugs/918), which causes all pre-commit
> runs (e.g., HADOOP-11287) to fail.
>
> The current version of findbugs (1.3.9) used by Hadoop is released in 2009.
> Given that:
>
> (1) The current bug that we hit are fixed by a later version of findbug.
> (2) A newer findbug (3.0.0) is required to analyze Hadoop that is compiled
> against Java 8.
> (3) Newer findbugs are capable of catching more bugs. :-)
>
> Is it a good time to consider upgrading findbugs, which gives us better
> tools on ensuring the quality of the code case?
>
> I ran findbugs 3.0.0 against trunk today. It reported 111 warnings for
> hadoop-common, 44 for HDFS and 40+ for YARN. Many of them are possible
> NPEs, resource leaks, and ignored exception which are indeed bugs and are
> worthwhile to address.
>
> However, one issue that needs to be considered is that how to deal with the
> additional warnings reported by the newer findbugs without breaking the
> Jenkins pre-commit runs.
>
> Personally I can see three possible routes if we decide to upgrade
> findbugs:
>
> (1) Fix all warnings before upgrading to newer findbugs.
>
This might take a while. We might want to use the newer findbugs sooner?
> (2) Add all new warnings to the exclude list and fix them slowly.
>
I have my doubts on how soon we fix these warnings unless we make the
associated JIRAs (assuming we have one per exclude) blockers for the next
release. A findbugs "Fix It" day would be ideal to get this done.
> (3) Update test-patch.sh to make sure that new code won't introduce any new
> findbugs warnings.
>
Seems the best, especially if test-patch.sh shows the warnings, but doesn't
-1 unless there are new findbugs warnings. This way, the contributor can
choose to fix "related" warnings at the least.
>
> I proposed upgrading to findbugs 2.0.2 and fixing new warnings in
> HADOOP-10476, which could be dated backed to April, 2014. I volunteer to
> accelerate the effort if it is required.
> Thoughts?
>
> Regards,
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es
Re: Upgrading findbugs
Posted by Karthik Kambatla <ka...@cloudera.com>.
Thanks for initiating this, Haohui. +1 to upgrading findbugs version.
Inline.
On Mon, Dec 8, 2014 at 9:57 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> Hi,
>
> The recent changes on moving to Java 7 triggers a bug in findbug (
> http://sourceforge.net/p/findbugs/bugs/918), which causes all pre-commit
> runs (e.g., HADOOP-11287) to fail.
>
> The current version of findbugs (1.3.9) used by Hadoop is released in 2009.
> Given that:
>
> (1) The current bug that we hit are fixed by a later version of findbug.
> (2) A newer findbug (3.0.0) is required to analyze Hadoop that is compiled
> against Java 8.
> (3) Newer findbugs are capable of catching more bugs. :-)
>
> Is it a good time to consider upgrading findbugs, which gives us better
> tools on ensuring the quality of the code case?
>
> I ran findbugs 3.0.0 against trunk today. It reported 111 warnings for
> hadoop-common, 44 for HDFS and 40+ for YARN. Many of them are possible
> NPEs, resource leaks, and ignored exception which are indeed bugs and are
> worthwhile to address.
>
> However, one issue that needs to be considered is that how to deal with the
> additional warnings reported by the newer findbugs without breaking the
> Jenkins pre-commit runs.
>
> Personally I can see three possible routes if we decide to upgrade
> findbugs:
>
> (1) Fix all warnings before upgrading to newer findbugs.
>
This might take a while. We might want to use the newer findbugs sooner?
> (2) Add all new warnings to the exclude list and fix them slowly.
>
I have my doubts on how soon we fix these warnings unless we make the
associated JIRAs (assuming we have one per exclude) blockers for the next
release. A findbugs "Fix It" day would be ideal to get this done.
> (3) Update test-patch.sh to make sure that new code won't introduce any new
> findbugs warnings.
>
Seems the best, especially if test-patch.sh shows the warnings, but doesn't
-1 unless there are new findbugs warnings. This way, the contributor can
choose to fix "related" warnings at the least.
>
> I proposed upgrading to findbugs 2.0.2 and fixing new warnings in
> HADOOP-10476, which could be dated backed to April, 2014. I volunteer to
> accelerate the effort if it is required.
> Thoughts?
>
> Regards,
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es
Re: Upgrading findbugs
Posted by Karthik Kambatla <ka...@cloudera.com>.
Thanks for initiating this, Haohui. +1 to upgrading findbugs version.
Inline.
On Mon, Dec 8, 2014 at 9:57 PM, Haohui Mai <hm...@hortonworks.com> wrote:
> Hi,
>
> The recent changes on moving to Java 7 triggers a bug in findbug (
> http://sourceforge.net/p/findbugs/bugs/918), which causes all pre-commit
> runs (e.g., HADOOP-11287) to fail.
>
> The current version of findbugs (1.3.9) used by Hadoop is released in 2009.
> Given that:
>
> (1) The current bug that we hit are fixed by a later version of findbug.
> (2) A newer findbug (3.0.0) is required to analyze Hadoop that is compiled
> against Java 8.
> (3) Newer findbugs are capable of catching more bugs. :-)
>
> Is it a good time to consider upgrading findbugs, which gives us better
> tools on ensuring the quality of the code case?
>
> I ran findbugs 3.0.0 against trunk today. It reported 111 warnings for
> hadoop-common, 44 for HDFS and 40+ for YARN. Many of them are possible
> NPEs, resource leaks, and ignored exception which are indeed bugs and are
> worthwhile to address.
>
> However, one issue that needs to be considered is that how to deal with the
> additional warnings reported by the newer findbugs without breaking the
> Jenkins pre-commit runs.
>
> Personally I can see three possible routes if we decide to upgrade
> findbugs:
>
> (1) Fix all warnings before upgrading to newer findbugs.
>
This might take a while. We might want to use the newer findbugs sooner?
> (2) Add all new warnings to the exclude list and fix them slowly.
>
I have my doubts on how soon we fix these warnings unless we make the
associated JIRAs (assuming we have one per exclude) blockers for the next
release. A findbugs "Fix It" day would be ideal to get this done.
> (3) Update test-patch.sh to make sure that new code won't introduce any new
> findbugs warnings.
>
Seems the best, especially if test-patch.sh shows the warnings, but doesn't
-1 unless there are new findbugs warnings. This way, the contributor can
choose to fix "related" warnings at the least.
>
> I proposed upgrading to findbugs 2.0.2 and fixing new warnings in
> HADOOP-10476, which could be dated backed to April, 2014. I volunteer to
> accelerate the effort if it is required.
> Thoughts?
>
> Regards,
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Karthik Kambatla
Software Engineer, Cloudera Inc.
--------------------------------------------
http://five.sentenc.es