PROPOSAL: Be stricter about creating Bugzilla issue before committing

[Sidney Markowitz <si...@sidney.com>]

My task this weekend has been tracking down mismatches between trunk and
branch 3.4 to make sure that anything that has been committed to trunk and is
or should be targeted for the 3.4.2 release has been committed to the branch.

In doing that I'm encountering a number of commits to trunk that do not refer
back to a Bugzilla issue. Usually it is a simple and obvious patch for an
issue that clearly ought to be done, and probably didn't seem worth the extra
steps of creating an issue in Bugzilla for it. Maybe some of them were a
result of the committer not including the bug number in the commit message - I
would not be able to tell if that's the case without a search in Bugzilla in
which I guess the keywords to look for.

I propose that we make a practice of 1) Create a Bugzilla issue for anything
we commit to source files or to any other files that are part of the build
process; 2) Include the text "bug NNNN" somewhere in the commit message.

The benefits of doing this include 1) Makes it easier to track what has to be
ported from trunk to branch before a release; 2) Makes it possible to look up
the reasons and discussions related to a commit, and to find other commits
that are related to the same issue even when at the time of the first commit
it was not anticipated that there would be anything more to be committed.

We already have in our guidelines to include "bug NNNN" in the commit message

https://wiki.apache.org/spamassassin/UsingBugzilla#Committing_Fixes_For_Bugs

I would like to see if there is consensus on adding to our guidelines that we
always create a Bugzilla issue for something that will be committed into our
source or related files.

It also seems like a good idea to create a CommitterGuidelines page on the
wiki. I had some difficulty finding the guideline for svn commit messages
where it is hiding under UsingBugzilla.

Comments, votes?

 Sidney

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

[Bill Cole <sa...@billmail.scconsult.com>]

On 14 Apr 2017, at 22:40, Sidney Markowitz wrote:

> I propose that we make a practice of 1) Create a Bugzilla issue for 
> anything
> we commit to source files or to any other files that are part of the 
> build
> process; 2) Include the text "bug NNNN" somewhere in the commit 
> message.
>
>
> Comments, votes?

+1

Re: rules/*.cf in syncing 3.4 and trunk

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 4/15/2017 4:11 AM, Sidney Markowitz wrote:
> Are any of these relevant for syncing 3.4 with trunk, or are only the files in
> trunk used for rule updates? These are the files in the rules directory that
> have differences between trunk and the 3.4 branch:
>
> 20_dnsbl_tests.cf
> 20_drugs.cf
> 20_freemail.cf
> 20_freemail_domains.cf
> 20_imageinfo.cf
> 20_ratware.cf
> 25_uribl.cf
> 50_scores.cf

So are the files in trunk used for rules updates, once we relight the 
rule updates, we need to look at how it's done anyway.  I personally 
think that if we use sa version checks, that 3.4 can be used for rule 
updates and can apply to 3.3 & 3.4.  The same for 4.0 and trunk because 
we will start diverging things due to UTF support.  Likely rules are 
just committed in both places though I consider this the last 3.4 release.

Overall, I *like* to provide rules but the goal I see as a project is to 
provide software and examples of rules as templates for implementation 
more so than a standalone product.

With that said though, 3.4 is now in sync with trunk as we started out.  
Here's what I did with the cf files:

20_dnsbl_tests.cf - synced trunk to 3.4 based on the comments available 
re: de-listing fees.

20_freemail.cf - synced 3.4 to trunk
20_freemail_domains.cf - ""
20_imageinfo.cf - ""
20_ratware.cf - ""
25_uribl.cf - ""
50_scores.cf - ""

70_sandbox.cf and all sandbox/ rulesets ignoring because they are well, 
sandbox rules :-)
20_drugs.cf I did not sync as I believe that's related to idn changes 
not in 3.4

Regards,

KAM

rules/*.cf in syncing 3.4 and trunk

[Sidney Markowitz <si...@sidney.com>]

Are any of these relevant for syncing 3.4 with trunk, or are only the files in
trunk used for rule updates? These are the files in the rules directory that
have differences between trunk and the 3.4 branch:

20_dnsbl_tests.cf
20_drugs.cf
20_freemail.cf
20_freemail_domains.cf
20_imageinfo.cf
20_ratware.cf
25_uribl.cf
50_scores.cf

Re: Syncing 3.4 and trunk

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 4/15/2017 12:02 AM, Sidney Markowitz wrote:
> Kevin, this question is probably for you.
>
> Should I apply thge last four changes to rules/50_scores.cf that were 
> made in
> trunk to the 3.4 branch? Only the first of them mentions a bugzilla 
> issue in
> the commit message, bug 7282, but I think they are all part of it and 
> that it
> makes sense to port.
Yes, I took a look and those should have been committed.  I'm almost 
done on the review.
> Also, I'm porting bug 7192 which looks like it was incorrectly labeled 
> with a
> 3.4.1 version even though it was committed after the 3.4.1 release, 
> and then
> never ported to the branch. The commit for that in trunk created a 
> file in
> your sandbox sandbox/kmcgrail/20_rules_to_sandbox.cf
>
> Do I put that in branch/3.4 also, or skip that file?
I'm sure the 3.4.1 tag was a mistake.

I thought that they would have been captured by auto_promotion. I've  
opened 7412 and see you already copied the file so consider that bug 
closed.

Re: Syncing 3.4 and trunk

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 4/15/2017 12:02 AM, Sidney Markowitz wrote:
> Kevin, this question is probably for you.
>
> Should I apply thge last four changes to rules/50_scores.cf that were made in
> trunk to the 3.4 branch? Only the first of them mentions a bugzilla issue in
> the commit message, bug 7282, but I think they are all part of it and that it
> makes sense to port.
Yes, I took a
> Also, I'm porting bug 7192 which looks like it was incorrectly labeled with a
> 3.4.1 version even though it was committed after the 3.4.1 release, and then
> never ported to the branch. The commit for that in trunk created a file in
> your sandbox sandbox/kmcgrail/20_rules_to_sandbox.cf
>
> Do I put that in branch/3.4 also, or skip that file?
I'm sure the 3.4.1 tag was a mistake.

I thought that they would have been captured by auto_promotion. I've  
opened 7412 and see you already copied the file so consider that bug closed.

Syncing 3.4 and trunk

[Sidney Markowitz <si...@sidney.com>]

Kevin, this question is probably for you.

Should I apply thge last four changes to rules/50_scores.cf that were made in
trunk to the 3.4 branch? Only the first of them mentions a bugzilla issue in
the commit message, bug 7282, but I think they are all part of it and that it
makes sense to port.

Also, I'm porting bug 7192 which looks like it was incorrectly labeled with a
3.4.1 version even though it was committed after the 3.4.1 release, and then
never ported to the branch. The commit for that in trunk created a file in
your sandbox sandbox/kmcgrail/20_rules_to_sandbox.cf

Do I put that in branch/3.4 also, or skip that file?

 Sidney

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

[Sidney Markowitz <si...@sidney.com>]

Kevin Golding wrote on 15/04/17 8:51 PM:
> Apologies if any overlap

This looks good. I can see from the names of the attachments that most if not
all will be ones I have yet to look at, so the timing is good. I'll grab the
attachments and continue from there for the rest of tonight.

Thanks,

 Sidney

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

[Kevin Golding <kp...@caomhin.org>]

On Sat, 15 Apr 2017 04:25:47 +0100, Sidney Markowitz <si...@sidney.com>  
wrote:

> Kevin A. McGrail wrote on 15/04/17 3:06 PM:
>> Plus one here but will note Kevin Golding and I have already been  
>> syncing
>> trunk and 3.4. I can email you the remaining work tomorrow so we can  
>> sync.
>
> I noticed what you have done. So far I'm about to be done with all the  
> non
> *.pm files and there are only 22 *.pm files with differences left to  
> look at.
> If you and Kevin Golding have anything that you are in the middle of  
> that you
> have not yet committed you should let me know because I may be able to  
> get
> through all the rest by tonight (still afternoon here).

Apologies if any overlap, I was trying to handle things piece by piece  
because I didn't know the order they would be looked at and/or if they  
would all be included. (I think I missed a couple out because I recall KAM  
saying at least one had been surpassed by other changes.) I've just been  
checking against https://svn.apache.org/viewvc/spamassassin/branches/3.4/  
this morning and I'm speeding through to try and be helpful before I  
vanish for a little bit.

The bug and rXXXX named patches should be easiest to associate. The ones  
with filenames may be more cryptic. I know I sent explanations originally  
and if you want any details I can look back and forward the mails directly.

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 4/14/2017 11:25 PM, Sidney Markowitz wrote:
> Kevin A. McGrail wrote on 15/04/17 3:06 PM:
>> Plus one here but will note Kevin Golding and I have already been syncing
>> trunk and 3.4. I can email you the remaining work tomorrow so we can sync.
> I noticed what you have done. So far I'm about to be done with all the non
> *.pm files and there are only 22 *.pm files with differences left to look at.
> If you and Kevin Golding have anything that you are in the middle of that you
> have not yet committed you should let me know because I may be able to get
> through all the rest by tonight (still afternoon here).
I just sent you the files I was working on.  I had been testing the 
sa_install changes and some other bugs for Geo::IP so no worries. We'll 
get it in sync.

Regards,
KAM

Re: All ported

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 4/16/2017 7:04 AM, Sidney Markowitz wrote:
> I'm done with the patches. I verified that I included all the patches you sent
> (except the one for Bug 7215 that Kevin G sent, as it looks like we are not
> including that in 3.4.2), and checked again that any files that are different
> in branch and trunk that I did not port only have the unicode changes that we
> are not releasing in 3.4.2.
>
> I did not touch the following rules files even though there are differences
> between what they contain in branches/3.4 and in trunk.
>
> 20_dnsbl_tests.cf
> 20_drugs.cf
> 20_freemail.cf
> 20_freemail_domains.cf
> 20_imageinfo.cf
> 20_ratware.cf
> 25_uribl.cf
>
>    Sidney

Thanks Sidney.  I'm caught up in Easter festivities but will take a look 
as soon as I can!

All ported

[Sidney Markowitz <si...@sidney.com>]

I'm done with the patches. I verified that I included all the patches you sent
(except the one for Bug 7215 that Kevin G sent, as it looks like we are not
including that in 3.4.2), and checked again that any files that are different
in branch and trunk that I did not port only have the unicode changes that we
are not releasing in 3.4.2.

I did not touch the following rules files even though there are differences
between what they contain in branches/3.4 and in trunk.

20_dnsbl_tests.cf
20_drugs.cf
20_freemail.cf
20_freemail_domains.cf
20_imageinfo.cf
20_ratware.cf
25_uribl.cf

  Sidney

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 4/15/2017 4:29 AM, Kevin Golding wrote:
> Playing timezone bingo I'll assume I might be able to save you some 
> work. I'll try to whizz through the patches I've sent KAM and send any 
> I think are outstanding. 

Heh, yes.  I was also doing regression tests on the patch from Bill Cole 
on Ununto, Centos and RHEL.

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

[Kevin Golding <kp...@caomhin.org>]

On Sat, 15 Apr 2017 04:25:47 +0100, Sidney Markowitz <si...@sidney.com>  
wrote:

> Kevin A. McGrail wrote on 15/04/17 3:06 PM:
>> Plus one here but will note Kevin Golding and I have already been  
>> syncing
>> trunk and 3.4. I can email you the remaining work tomorrow so we can  
>> sync.
>
> I noticed what you have done. So far I'm about to be done with all the  
> non
> *.pm files and there are only 22 *.pm files with differences left to  
> look at.
> If you and Kevin Golding have anything that you are in the middle of  
> that you
> have not yet committed you should let me know because I may be able to  
> get
> through all the rest by tonight (still afternoon here).

Playing timezone bingo I'll assume I might be able to save you some work.  
I'll try to whizz through the patches I've sent KAM and send any I think  
are outstanding.

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

[Sidney Markowitz <si...@sidney.com>]

Kevin A. McGrail wrote on 15/04/17 3:06 PM:
> Plus one here but will note Kevin Golding and I have already been syncing
> trunk and 3.4. I can email you the remaining work tomorrow so we can sync.

I noticed what you have done. So far I'm about to be done with all the non
*.pm files and there are only 22 *.pm files with differences left to look at.
If you and Kevin Golding have anything that you are in the middle of that you
have not yet committed you should let me know because I may be able to get
through all the rest by tonight (still afternoon here).

 Sidney

Re: PROPOSAL: Be stricter about creating Bugzilla issue before committing

["Kevin A. McGrail" <ke...@mcgrail.com>]

Plus one here but will note Kevin Golding and I have already been syncing trunk and 3.4.  I can email you the remaining work tomorrow so we can sync.
Regards,
KAM

On April 14, 2017 10:40:41 PM EDT, Sidney Markowitz <si...@sidney.com> wrote:
>My task this weekend has been tracking down mismatches between trunk
>and
>branch 3.4 to make sure that anything that has been committed to trunk
>and is
>or should be targeted for the 3.4.2 release has been committed to the
>branch.
>
>In doing that I'm encountering a number of commits to trunk that do not
>refer
>back to a Bugzilla issue. Usually it is a simple and obvious patch for
>an
>issue that clearly ought to be done, and probably didn't seem worth the
>extra
>steps of creating an issue in Bugzilla for it. Maybe some of them were
>a
>result of the committer not including the bug number in the commit
>message - I
>would not be able to tell if that's the case without a search in
>Bugzilla in
>which I guess the keywords to look for.
>
>I propose that we make a practice of 1) Create a Bugzilla issue for
>anything
>we commit to source files or to any other files that are part of the
>build
>process; 2) Include the text "bug NNNN" somewhere in the commit
>message.
>
>The benefits of doing this include 1) Makes it easier to track what has
>to be
>ported from trunk to branch before a release; 2) Makes it possible to
>look up
>the reasons and discussions related to a commit, and to find other
>commits
>that are related to the same issue even when at the time of the first
>commit
>it was not anticipated that there would be anything more to be
>committed.
>
>We already have in our guidelines to include "bug NNNN" in the commit
>message
>
>https://wiki.apache.org/spamassassin/UsingBugzilla#Committing_Fixes_For_Bugs
>
>I would like to see if there is consensus on adding to our guidelines
>that we
>always create a Bugzilla issue for something that will be committed
>into our
>source or related files.
>
>It also seems like a good idea to create a CommitterGuidelines page on
>the
>wiki. I had some difficulty finding the guideline for svn commit
>messages
>where it is hiding under UsingBugzilla.
>
>Comments, votes?
>
> Sidney