You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2008/03/19 13:45:02 UTC
[Bug 5855] New: trim down our test suite by default
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5855
Summary: trim down our test suite by default
Product: Spamassassin
Version: unspecified
Platform: Other
OS/Version: All
Status: NEW
Severity: enhancement
Priority: P5
Component: Building & Packaging
AssignedTo: dev@spamassassin.apache.org
ReportedBy: jm@jmason.org
our test suite takes bloody ages to run. This is great for us as developers,
since it's extensive and catches bugs. But for users -- esp for users of
CPAN.pm, which runs all packages' test suites before they install -- it's
painful to have to wait for 30 minutes for Mail::SpamAssassin to decide it's
ready to go, on top of the test suite runtime of all our CPAN dependencies.
I suggest we change at least 50% of our test runtime to use a
set-to-"n"-by-default value in t/config. developers can copy t/config.dist to
t/config to get working tests again, and users will be happier.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 5855] trim down our test suite by default
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5855
--- Comment #3 from Michael Parker <pa...@pobox.com> 2008-08-14 08:09:10 PST ---
I'm a little torn on this one.
I agree that our test suite take awhile to run, but some of the things it runs
will show problems on a users system before they install. Now if we skip those
tests and they run make test and all is fine, but say bayesdbm doesn't really
work, you're going to be creating a support nightmare.
Suggest that instead of a total skip, we move very basic tests to the front of
the tests and run a smaller subset, and not the entire test.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 5855] trim down our test suite by default
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5855
--- Comment #1 from Justin Mason <jm...@jmason.org> 2008-08-14 05:18:36 PST ---
timings from our test suite on my box (in seconds, rounded down, with a total
runtime of 20 minutes):
2 t/basic_lint.t
1 t/basic_meta.t
2 t/basic_obj_api.t
13 t/bayesdbm.t
13 t/bayesdbm_flock.t
11 t/bayessdbm.t
11 t/bayessdbm_seen_delete.t
1 t/bayessql.t
1 t/blacklist_autolearn.t
1 t/body_mod.t
0 t/check_implemented.t
0 t/cidrs.t
0 t/config_errs.t
2 t/config_text.t
0 t/cpp_comments_in_spamc.t
0 t/date.t
3 t/db_awl_path.t
11 t/db_based_whitelist.t
10 t/db_based_whitelist_ips.t
3 t/dcc.t
14 t/debug.t
3 t/desc_wrap.t
0 t/dkim.t
54 t/dnsbl.t
1 t/dnsbl_sc_meta.t
2 t/duplicates.t
2 t/get_all_headers.t
1 t/get_headers.t
3 t/gtube.t
8 t/hashcash.t
0 t/html_colors.t
2 t/html_obfu.t
2 t/html_utf8.t
1 t/ifversion.t
0 t/ip_addrs.t
16 t/lang_lint.t
2 t/lang_pl_tests.t
11 t/line_endings.t
2 t/lint_nocreate_prefs.t
2 t/mass_check.t
0 t/memory_cycles.t
1 t/metadata.t
1 t/mimeheader.t
0 t/mimeparse.t
1 t/missing_hb_separator.t
3 t/mkrules.t
1 t/nonspam.t
1 t/plugin.t
1 t/plugin_file.t
2 t/plugin_priorities.t
2 t/prefs_include.t
0 t/priorities.t
0 t/razor2.t
1 t/rcvd_parser.t
0 t/re_base_extraction.t
5 t/recips.t
3 t/recreate.t
8 t/recursion.t
1 t/regexp_valid.t
1 t/relative_scores.t
5 t/report_safe.t
1 t/reportheader.t
2 t/reportheader_8bit.t
0 t/root_spamd.t
0 t/root_spamd_tell.t
0 t/root_spamd_tell_paranoid.t
0 t/root_spamd_tell_x.t
0 t/root_spamd_tell_x_paranoid.t
0 t/root_spamd_u.t
0 t/root_spamd_u_dcc.t
0 t/root_spamd_virtual.t
0 t/root_spamd_x.t
0 t/root_spamd_x_paranoid.t
0 t/root_spamd_x_u.t
2 t/rule_multiple.t
2 t/rule_names.t
1 t/rule_tests.t
2 t/rule_types.t
0 t/saw_ampersand.t
0 t/sha1.t
2 t/shortcircuit.t
2 t/spam.t
3 t/spamc.t
5 t/spamc_B.t
4 t/spamc_E.t
4 t/spamc_H.t
7 t/spamc_c.t
7 t/spamc_c_stdout_closed.t
4 t/spamc_cf.t
5 t/spamc_headers.t
3 t/spamc_l.t
4 t/spamc_optC.t
6 t/spamc_optL.t
19 t/spamc_x_E_R.t
7 t/spamc_x_e.t
3 t/spamc_y.t
0 t/spamc_z.t
4 t/spamd.t
6 t/spamd_allow_user_rules.t
17 t/spamd_client.t
56 t/spamd_hup.t
57 t/spamd_kill_restart.t
57 t/spamd_kill_restart_rr.t
0 t/spamd_ldap.t
5 t/spamd_maxchildren.t
0 t/spamd_maxsize.t
5 t/spamd_parallel.t
4 t/spamd_plugin.t
4 t/spamd_port.t
21 t/spamd_prefork_stress.t
28 t/spamd_prefork_stress_2.t
150 t/spamd_prefork_stress_3.t
80 t/spamd_prefork_stress_4.t
4 t/spamd_protocol_10.t
4 t/spamd_report.t
4 t/spamd_report_ifspam.t
0 t/spamd_sql_prefs.t
0 t/spamd_ssl.t
0 t/spamd_ssl_accept_fail.t
0 t/spamd_ssl_tls.t
0 t/spamd_ssl_v2.t
0 t/spamd_ssl_v23.t
0 t/spamd_ssl_v3.t
4 t/spamd_stop.t
4 t/spamd_symbols.t
4 t/spamd_syslog.t
4 t/spamd_unix.t
4 t/spamd_unix_and_tcp.t
9 t/spamd_user_rules_leak.t
4 t/spamd_utf8.t
138 t/spf.t
0 t/sql_based_whitelist.t
62 t/strip2.t
10 t/strip_no_subject.t
1 t/stripmarkup.t
2 t/tainted_msg.t
3 t/text_bad_ctype.t
28 t/trust_path.t
1 t/uri.t
1 t/uri_html.t
1 t/uri_text.t
24 t/uribl.t
2 t/utf8.t
0 t/util_wrap.t
11 t/whitelist_addrs.t
14 t/whitelist_from.t
3 t/whitelist_subject.t
2 t/whitelist_to.t
0 t/zz_cleanup.t
So I suggest we just remove any test that takes longer than 10 secs in the
above
list from the default-run list. That'll bring it down from 20 minutes to just
over 4. ;)
Those tests are:
13 t/bayesdbm.t
13 t/bayesdbm_flock.t
11 t/bayessdbm.t
11 t/bayessdbm_seen_delete.t
11 t/db_based_whitelist.t
10 t/db_based_whitelist_ips.t
14 t/debug.t
54 t/dnsbl.t
16 t/lang_lint.t
11 t/line_endings.t
19 t/spamc_x_E_R.t
17 t/spamd_client.t
56 t/spamd_hup.t
57 t/spamd_kill_restart.t
57 t/spamd_kill_restart_rr.t
21 t/spamd_prefork_stress.t
28 t/spamd_prefork_stress_2.t
150 t/spamd_prefork_stress_3.t
80 t/spamd_prefork_stress_4.t
138 t/spf.t
62 t/strip2.t
10 t/strip_no_subject.t
28 t/trust_path.t
24 t/uribl.t
11 t/whitelist_addrs.t
14 t/whitelist_from.t
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 5855] trim down our test suite by default
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5855
Justin Mason <jm...@jmason.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #2 from Justin Mason <jm...@jmason.org> 2008-08-14 07:57:37 PST ---
ok, done.
: jm 200...; svn commit -m "bug 4131: speed up default 'make test' from 20
minutes, to just over 4, by not running the really slow tests by default.
developers should edit t/config and set run_long_tests=y" t build/hudson
Sending build/hudson/run_build
Sending t/bayesdbm.t
Sending t/bayesdbm_flock.t
Sending t/bayessdbm.t
Sending t/bayessdbm_seen_delete.t
Sending t/config.dist
Sending t/db_based_whitelist.t
Sending t/db_based_whitelist_ips.t
Sending t/debug.t
Sending t/dnsbl.t
Sending t/lang_lint.t
Sending t/line_endings.t
Sending t/spamc_x_E_R.t
Sending t/spamd_client.t
Sending t/spamd_hup.t
Sending t/spamd_kill_restart.t
Sending t/spamd_kill_restart_rr.t
Sending t/spamd_prefork_stress.t
Sending t/spamd_prefork_stress_2.t
Sending t/spamd_prefork_stress_3.t
Sending t/spamd_prefork_stress_4.t
Sending t/spf.t
Sending t/strip2.t
Sending t/strip_no_subject.t
Sending t/trust_path.t
Sending t/uribl.t
Sending t/whitelist_addrs.t
Sending t/whitelist_from.t
Transmitting file data ............................
Committed revision 685932.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 5855] trim down our test suite by default
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5855
--- Comment #4 from Justin Mason <jm...@jmason.org> 2008-08-14 08:53:04 PST ---
(In reply to comment #3)
> I'm a little torn on this one.
>
> I agree that our test suite take awhile to run, but some of the things it runs
> will show problems on a users system before they install. Now if we skip those
> tests and they run make test and all is fine, but say bayesdbm doesn't really
> work, you're going to be creating a support nightmare.
I'd be fine with "rescuing" a few of those back into the on-by-default test
set; maybe these?
t/bayesdbm.t
t/bayessdbm.t
t/db_based_whitelist.t
t/trust_path.t
t/whitelist_from.t
they seem to be a good selection of feature coverage. WDYT?
> Suggest that instead of a total skip, we move very basic tests to the front of
> the tests and run a smaller subset, and not the entire test.
well, it's easier to either run or not-run an entire .t script. It's also hard
to see if a test was partially run from the "overview" view, ie the one that
looks like this:
t/foo............................ok
t/bar............................ok
t/baz............................ok
If "t/bar" exited early after testing half of the features, it's not easy to
tell that.
What we could do, is split long-running scripts into 2 scripts, with the first
one being the "quick" bit ("t/bar_quick") and the second being the "slow"
remainder ("t/bar_full"). I'd be fine with that.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 5855] trim down our test suite by default
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5855
Justin Mason <jm...@jmason.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P5 |P3
Target Milestone|Undefined |3.3.0
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.