You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2013/09/04 22:38:24 UTC
[Bug 6971] New: Add support for LMDB as a backend for Bayes DB
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
Bug ID: 6971
Summary: Add support for LMDB as a backend for Bayes DB
Product: Spamassassin
Version: SVN Trunk (Latest Devel Version)
Hardware: PC
OS: Windows 7
Status: NEW
Severity: normal
Priority: P2
Component: spamassassin
Assignee: dev@spamassassin.apache.org
Reporter: quanah@zimbra.com
Now that there is a perl module for LMDB, it would be nice to see support for
it implemented so folks who want a local disk-based DB can use it instead of
the hostilely licensed Oracle BDB software.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
--- Comment #8 from Quanah Gibson-Mount <qu...@zimbra.com> ---
20130927
Cleanup: no more LMDB "database full" errors. Postfix now
requires LMDB >= 0.9.8 which supports on-the-fly database
resizing. When a database becomes full, its size limit is
automatically doubled, and other processes automatically
pick up the new database size limit. Files: util/dict.h,
util/dict_open.c, util/dict_alloc.c, util/dict_lmdb.c,
postmap/postmap.c, postalias/postalias.c, proto/LMDB_README.html,
proto/postconf.proto.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
Quanah Gibson-Mount <qu...@zimbra.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |quanah@zimbra.com
--- Comment #2 from Quanah Gibson-Mount <qu...@zimbra.com> ---
SDBM seems to have an interesting limitation:
BUGS AND WARNINGS
There are a number of limits on the size of the data that you can store in the
SDBM file. The most important is that the length of a key, plus the length of
its associated value, may not exceed 1008 bytes.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
--- Comment #1 from AXB <ax...@gmail.com> ---
SDBM support is already included (and performs much faster that BDB)
Doesn't it's license cover those needs?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
Karsten Bräckelmann <gu...@rudersport.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P2 |P3
Component|spamassassin |Libraries
Hardware|PC |All
OS|Windows 7 |All
Severity|normal |enhancement
--- Comment #7 from Karsten Bräckelmann <gu...@rudersport.de> ---
Feature Request (importance enhancement), component Libraries.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
--- Comment #6 from Quanah Gibson-Mount <qu...@zimbra.com> ---
(In reply to Mark Martinec from comment #5)
> Btw, postfix mailer docs claim:
>
> The status of LMDB databases is "not recommended".
Hi Marc,
I'm quite aware of Wietse's issues, all of which have been addressed except his
problem with configuring a maximum size for the database. This is more of a
philosophical complaint than anything else.
Essentially, with LMDB, you *must* specify a maximum size that you would not
allow the database to exceed vs the ability to allow unlimited growth of the
database so that it can grow unchecked and take down entire servers.
Personally, I prefer the ability to limit how large I'm willing to allow a
database to grow, Wietse does not. And quite frankly, if what you want to do
is allow LMDB to act like any other database and potentially fill your server
disk and take down the entire server, you can simple set the maximum size for
LMDB to match the size of your disk.
I've been using back-mdb (the LMDB backend in OpenLDAP for over a year and a
half), with exceptional results:
http://mishikal.wordpress.com/2013/05/16/openldap-a-comparison-of-back-mdb-and-back-hdb-performance/
http://wiki.zimbra.com/wiki/OpenLDAP_MDB_vs_HDB_performance
I would also note plenty of other projects have added support for LMDB or are
adding support for LMDB. I would even note there is a LMDB backend for Redis.
;)
http://symas.com/mdb/#projects
--Quanah
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
--- Comment #3 from AXB <ax...@gmail.com> ---
(In reply to Quanah Gibson-Mount from comment #2)
> SDBM seems to have an interesting limitation:
>
> BUGS AND WARNINGS
>
> There are a number of limits on the size of the data that you can store in
> the SDBM file. The most important is that the length of a key, plus the
> length of its associated value, may not exceed 1008 bytes.
I've used Bayes/SDBM for years and never found this to be an issue for SA use.
Must admit I never had the need/interest to look into the DB.
Would you check it for us?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
--- Comment #9 from Quanah Gibson-Mount <qu...@zimbra.com> ---
The stable Postfix release is called postfix-2.11.x where 2=major
release number, 11=minor release number, x=patchlevel. The stable
release never changes except for patches that address bugs or
emergencies. Patches change the patchlevel and the release date.
Major changes - LMDB database support
-------------------------------------
LMDB is a memory-mapped database that was originally developed as
part of OpenLDAP. The Postfix LMDB driver was originally contributed
by Howard Chu, LMDB's creator.
LMDB can be used for all Postfix lookup tables and caches. It is
the first persistent Postfix database that can be shared among
multiple writers such as postscreen daemons (Postfix already supported
shared non-persistent memcached caches). See lmdb_table(5) and
LMDB_README for further information, including how to access Postfix
LMDB databases with non-Postfix programs.
Postfix currently requires LMDB version 0.9.11 or later. The minimum
version may change over time in the light of deployment experience.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
--- Comment #4 from Mark Martinec <Ma...@ijs.si> ---
> There are a number of limits on the size of the data that you can store in
> the SDBM file. The most important is that the length of a key, plus the
> length of its associated value, may not exceed 1008 bytes.
The (S)DBM/BDB have their issues, but this isn't one of them for
the purpose they serve in SpamAssassin: it is a non-issue for bayes,
its token keys are 5 bytes, token values are up to 13 bytes.
The seen entries are the longest, still less than 60 characters
for keys, and one byte for its value.
(But yes, I was very happy when I ditched a BDB-based bayes database
and switched to SQL, all the issues with auto-expiration, learning
to journal etc were gone. Now with Bayes on Redis it's even more
comfortable and fast. I wouldn't mind if somebody prepares a LMDB
backend, but I'm not volunteering.)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6971] Add support for LMDB as a backend for Bayes DB
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6971
--- Comment #5 from Mark Martinec <Ma...@ijs.si> ---
Btw, postfix mailer docs claim:
The status of LMDB databases is "not recommended".
postfix / HISTORY :
20130822
The status of LMDB databases is "not recommended". Unlike
other Postfix databases, LMDB does not grow beyond a specified
limit even when the file system has room. This show-stopper
bug breaks applications whose requirements grow with load:
postscreen(8), greylisting, tlsmgr(8) and verify(8).
postfix / LMDB_README :
Warning: LMDB databases have a show-stopper bug: they do not grow beyond a
specified limit, and introduce a "database full" hard error condition that
does not exist with any other Postfix database type. This is a problem for
programs whose database grows with system load. Examples are postscreen(8),
greylisting, verify(8) and tlsmgr(8).
You have been warned.
Postfix provides a configuration parameter that controls how large an
OpenLDAP LMDB database may grow.
* lmdb_map_size (default: 16 MBytes per table). This setting controls
how large any OpenLDAP LMDB database may grow. It must be set large
enough to accommodate the largest table that Postfix will use.
Non-obvious recovery with postmap(1)/postalias(1)/tlsmgr(8) from a
corrupted database.
Problem:
You cannot rebuild a corrupted LMDB database simply by running postmap(1)
or postalias(1), or by waiting until the tlsmgr(8) daemon restarts
automatically. This problem does not exist with other Postfix databases.
Background:
The Postfix LMDB database client does not truncate the database file.
Instead it attempts to create a transaction for a "drop" request plus
subsequent "store" requests. That is obviously not possible with a
corrupted database file.
Impact:
Postfix does not process mail until someone fixes the problem.
Recovery:
First delete the ".lmdb" file by hand, then rebuild the file with the
postmap(1) or postalias(1) command, or wait until the tlsmgr(8) daemon
restarts automatically.
Prevention:
Arrange your file systems such that they never run out of free space.
Use ECC memory to detect and correct silent corruption of in-memory
file system data and metadata.
Use a file system such as ZFS to detect and correct silent corruption
of on-disk file system data and metadata.
--
You are receiving this mail because:
You are the assignee for the bug.