You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@spamassassin.apache.org on 2022/08/16 20:33:10 UTC
[Bug 8028] New: Errors in t/sql_based_welcomelist.t
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
Bug ID: 8028
Summary: Errors in t/sql_based_welcomelist.t
Product: Spamassassin
Version: unspecified
Hardware: PC
OS: FreeBSD
Status: NEW
Severity: normal
Priority: P2
Component: Regression Tests
Assignee: dev@spamassassin.apache.org
Reporter: sidney@sidney.com
Target Milestone: Undefined
I'm seeing two questionable things in results for t/sql_based_welcomelist.t,
one consistent and the other sporadic and not reproducible.
They both can be seen in the following CPAN test report of a test package in
which I added diagnostic output to dump the log file when the error happened.
Since I can't consistently make the error happen that was the only way I had to
view the log file when it did:
http://www.cpantesters.org/cpan/report/e01cc734-1d23-11ed-81f7-fd3d6e8775ea
Notice this line in the log, in the test looking for "scores -2.4":
>Aug 16 02:18:50.658 [73299] dbg: auto-welcomelist:
> sql-based whitelist_test@whitelist.spamassassin.taint.org|144.137
> scores -2.39999999999999991, msgcount 2
The extra 0.00000000000000009 points seem to come from this extra AWL rule that
shows up in the score summary, which does not appear in the log output of
successful tests:
>-1.2 NICE_002 BODY: No description available.
> 0.0 AWL AWL: No description available.
Anyone have an idea what could inconsistently cause the 0 point AWL to add
0.00000000000000009 to the total? As I said, I cannot reproduce this at will.
The other strange thing I see even in successful tests is in the line of the
log output
>Aug 16 02:18:50.658 [73299] dbg: auto-welcomelist: sql-based add_score/insert
> cpansand|whitelist_test@whitelist.spamassassin.taint.org|144.137|1|-1.2:
> SQL error: UNIQUE constraint failed: awl.username, awl.email, awl.signedby,
> awl.ip
The test creates the AWL table itself. Is this SQL error a bug in the test or
in the code, or something that is expected in the test that can be ignored?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
--- Comment #6 from Sidney Markowitz <si...@sidney.com> ---
I confirmed that the test failure can be consistently reproduced by running
with a perl 5.36.0 built with uselongdouble. I assume that the specific perl
version doesn't matter, just uselongdouble.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
--- Comment #8 from Sidney Markowitz <si...@sidney.com> ---
(In reply to Loren Wilton from comment #7)
> It seems possible to me that while Perl is compiled for long doubles, there
> is some piece of code that is working in normal doubles
I think you're on to something there. This is using DBD::SQLite which works
with its own data representations using single floats. In any case, since I can
now reproduce the problem I should be able to thrown tons of dbg lines at it to
track it down.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
--- Comment #4 from Sidney Markowitz <si...@sidney.com> ---
This commit only fixes the SQL error message, not the failing test
trunk % svn ci -m "bug 8028 - SQLite now handles upsert using same syntax as
pgsql, fix an error message" lib/Mail/SpamAssassin/SQLBasedAddrList.pm
Sending lib/Mail/SpamAssassin/SQLBasedAddrList.pm
Transmitting file data .done
Committing transaction...
Committed revision 1903469.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
--- Comment #1 from Loren Wilton <lw...@earthlink.net> ---
Sidney, you are dealing with roundoff error.
The representation of -2.4 in 64 bit IEEE is 0xc003333333333333.
The top 12 bits are the exponent and mantissa sign, and the remainder is the
mantissa value. As you can deduce from the number above, this is a repeating
fraction, and -2.4 can not be represented exactly.
The standard library math routines try to take roundoff error into account, and
round output numbers to some reasonable number of digits. Sometimes they get
confused for various and sundry reasons.
I'd suggest either accepting a result that is within a small epsilon of the
desired value, or doing a manual round to some reasonable number of digits on
the returned result before the compare.
See https://gregstoll.com/~gregstoll/floattohex/
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
--- Comment #3 from Sidney Markowitz <si...@sidney.com> ---
(In reply to Loren Wilton from comment #1)
> Sidney, you are dealing with roundoff error.
It's pretty clear that it is roundoff error, but I don't think it is in the
printing of -2.4. When the test works, the summary does not have the line "0.0
AWL". I suspect that there is some code that says in effect "if a rule produces
score 0 ignore it" and it is seeing AWL getting a score of 0.00000000000000009.
That's where we would want to round to some precision, but I don't know where
that code is.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
--- Comment #5 from Sidney Markowitz <si...@sidney.com> ---
I think I found the problem. The test machine on which the error occurred
reports that the perl it uses was compiled with uselongdouble. Now it should be
a matter of testing with a uselongdouble perl and seeing what breaks.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
--- Comment #2 from Sidney Markowitz <si...@sidney.com> ---
The SQL error message is because in the case of SQLite the code does not
properly handle an upsert. According to https://www.sqlite.org/lang_UPSERT.html
as of SQLite version 3.24.0 it can use the same ON CONFLICT syntax as
PostgreSQL. Since we already require that minimum version of SQLite, the fix is
to change all the if of dsn pg to be pg|SQLite in SQLBasedAddrList.pm
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
Loren Wilton <lw...@earthlink.net> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |lwilton@earthlink.net
--- Comment #7 from Loren Wilton <lw...@earthlink.net> ---
It seems possible to me that while Perl is compiled for long doubles, there is
some piece of code that is working in normal doubles, and that is then getting
converted to a long double. The extra digits on the end of the long double
mantissa are zero, so the rounding code to handle the error in a repeating
fraction presumably doesn't kick in.
The exact representation of 2.4 in a normal IEEE double is
2.39999999999999991 118. (space added for clarity.)
The value you saw was
2.39999999999999991 xxx.
If I subtract the exact value from 2.4, I get
0.00000000000000008 882.
The error you are seeing is
0.00000000000000009 xxx.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
Sidney Markowitz <si...@sidney.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sidney@sidney.com
Target Milestone|Undefined |4.0.0
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8028] Errors in t/sql_based_welcomelist.t
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8028
Sidney Markowitz <si...@sidney.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #9 from Sidney Markowitz <si...@sidney.com> ---
I verified that with DBD::SQLite in perl with uselongdouble if you write the
value -2.4 to a field that is declared either float or double (SQLIte
internally uses single float for both of those) when you fetch the value back
into a perl variable it is -2.39999999999999991. This is not strictly speaking
a bug, it is a loss of precision going between double and single float, and it
is not off by enough to make any difference when running SpamAssassin.
All that is broken is the test itself, which is looking for the exact result
"-2.4" in a line of dbg output, not even in expected normal SpamAssassin
output.
I've fixed it by making the dbg line use %.1f instead of %s so the output is in
a known rounded format, and having the test look for that format in all its
test cases.
trunk % svn ci -m "bug 8028 - Fix tests that failed when run in perl built with
uselongdouble that was not a SpamAssassin bug"
lib/Mail/SpamAssassin/SQLBasedAddrList.pm t/sql_based_w*.t
Sending lib/Mail/SpamAssassin/SQLBasedAddrList.pm
Sending t/sql_based_welcomelist.t
Sending t/sql_based_whitelist.t
Transmitting file data ...done
Committing transaction...
Committed revision 1903510.
--
You are receiving this mail because:
You are the assignee for the bug.