You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@spamassassin.apache.org on 2022/12/08 23:03:10 UTC
[Bug 8090] New: 4.0.0 prevents GeoIP2 mmdb files to be updated
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
Bug ID: 8090
Summary: 4.0.0 prevents GeoIP2 mmdb files to be updated
Product: Spamassassin
Version: 4.0.0
Hardware: PC
OS: Windows 10
Status: NEW
Severity: normal
Priority: P2
Component: Plugins
Assignee: dev@spamassassin.apache.org
Reporter: ruud.vd.hout@gmail.com
Target Milestone: Undefined
installed module:
Dec 8 17:58:19.370 [1896] dbg: diag: [...] optional module installed:
MaxMind::DB::Reader, version 1.000014
Dec 8 17:58:19.370 [1896] dbg: diag: [...] optional module not installed:
MaxMind::DB::Reader::XS ('require' failed)
I am using 4.0.0-RC4 on Windows, using MaxMind::DB::Reader to utilize ASN en
RelayCountry lookups, when running spamd as service it keeps both
GeoLite2-City.mmdb and GeoLite2-ASN.mmdb loaded constantly, making it
impossible to update those databases with geoipupdate.exe
(https://github.com/maxmind/geoipupdate)
if i ran geoipupdate -v it returns:
Performing update request to
https://updates.maxmind.com/geoip/databases/GeoLite2-ASN/update?db_md5=d5194b28eaa48826567fa38a75468c76
error retrieving updates: error while getting database for GeoLite2-ASN:
encountered an issue committing database update: error moving database into
place: rename C:\GeoIP\GeoLite2-ASN.mmdb.temporary C:\GeoIP\GeoLite2-ASN.mmdb:
Access is denied.
This Access is denied error is caused by spamd, if i kill spamd before running
geoipupdate it works fine.
I believe maxmind mmdb files can be loaded into memory, could that possibly
prevent the file in use (Access is denied) error?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
--- Comment #5 from RvdH <ru...@gmail.com> ---
OK, so this is a Windows issue only? Doesn't the same error occur on Linux?
I know mmdb files only has to be updated once a week, but recently i checked
and noticed those mmdb had not been updated since i start using 4.0.0 (RC2, RC3
and now RC4) as spamd is constantly running (as service)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
--- Comment #3 from RvdH <ru...@gmail.com> ---
Many plugins? You only have a single extra plugin (ASN.pm) besides the existing
RelayCountry.pm that could use geoip2 databases, not?
I see the has no option to load in memory, that is a pity, the C# DB readers
does have this functionality, you there can use MemoryMapped or Memory, see:
https://github.com/maxmind/MaxMind-DB-Reader-dotnet
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
--- Comment #6 from Henrik Krohns <ap...@hege.li> ---
(In reply to RvdH from comment #5)
> OK, so this is a Windows issue only? Doesn't the same error occur on Linux?
Yes it's Windows issue as demonstrated. The locking stuff is completely
different there. It's not possible to lock a file on Unix so that it can't be
replaced.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
--- Comment #10 from Henrik Krohns <ap...@hege.li> ---
(In reply to Paul from comment #9)
> Keeping the DB file open also affects Linux systems using geoip. When a
> reload using SIGHUP is requested a 2nd path is opened without closing the
> original path. This eventually leads to spamd stopping when it5 hits open
> files limit of 1024
Please provide proof and exact configuration where this happens? Afaik spamd
starts with a completely new process after HUP, it's not even possible to have
any old open files.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
--- Comment #11 from Paul <pa...@enlund.co.uk> ---
(In reply to Henrik Krohns from comment #10)
> (In reply to Paul from comment #9)
> > Keeping the DB file open also affects Linux systems using geoip. When a
> > reload using SIGHUP is requested a 2nd path is opened without closing the
> > original path. This eventually leads to spamd stopping when it5 hits open
> > files limit of 1024
>
> Please provide proof and exact configuration where this happens? Afaik spamd
> starts with a completely new process after HUP, it's not even possible to
> have any old open files.
spamd configured to have 5 children.
root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root
spamd 766010 root 9r REG
8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat
root@kanuka:/home/paule# /etc/init.d/spamassassin reload
Reloading spamassassin configuration (via systemctl): spamassassin.service.
root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root
spamd 767146 root 9r REG
8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat
spamd 767146 root 11r REG
8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat
root@kanuka:/home/paule# /etc/init.d/spamassassin reload
Reloading spamassassin configuration (via systemctl): spamassassin.service.
root@kanuka:/home/paule# lsof | grep spamd | grep GeoIP.dat | grep root
spamd 767863 root 9r REG
8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat
spamd 767863 root 11r REG
8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat
spamd 767863 root 13r REG
8,5 2579835 260907 /usr/share/GeoIP/GeoIP.dat
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
--- Comment #4 from Henrik Krohns <ap...@hege.li> ---
(In reply to RvdH from comment #3)
> Many plugins? You only have a single extra plugin (ASN.pm) besides the
> existing RelayCountry.pm that could use geoip2 databases, not?
There's URILocalBL also.
It would make no sense to just try loading the databases at the moment of use,
as it will still have the same race condition.
There's really no reason to update more than once a week, and one could time a
restart then. So I'm inclined to close this, but I'll leave around for a while
if someone wants to come up with a simple non-intrusive workaround for Windows.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
--- Comment #12 from Henrik Krohns <ap...@hege.li> ---
(In reply to Paul from comment #11)
> spamd configured to have 5 children.
Thanks, I've moved your report to Bug 8127 as it has nothing to do with this´or
GeoIP2.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
RvdH <ru...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |ruud.vd.hout@gmail.com
--- Comment #1 from RvdH <ru...@gmail.com> ---
FYI, 3.4.6 using GeoIP2::Database::Reader, version 2.006002 doesn't have this
issue locking the files in use
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
Sidney Markowitz <si...@sidney.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sidney@sidney.com
--- Comment #7 from Sidney Markowitz <si...@sidney.com> ---
With MaxMind no longer supporting the Perl version, there isn't any chance that
they will give it the same memory-mapped option that the .NET version has.
I agree with Henrik about leaving this open for a while in case anyone comes up
with an idea that would work, but if nobody does for a while I'm inclined to
close it as a WONTFIX with the workaround that you have to make the stop and
restart of spamd part of your weekly script that calls geopipupdate.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
Paul <pa...@enlund.co.uk> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |paul@enlund.co.uk
--- Comment #9 from Paul <pa...@enlund.co.uk> ---
Keeping the DB file open also affects Linux systems using geoip. When a reload
using SIGHUP is requested a 2nd path is opened without closing the original
path. This eventually leads to spamd stopping when it5 hits open files limit of
1024
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
Kevin A. McGrail <km...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kmcgrail@apache.org
--- Comment #8 from Kevin A. McGrail <km...@apache.org> ---
I think documentation would help if it's windows only but yeah, a file locking
issue on a file that updates weekly seems like something to script up locally
Even if we copy to another location, we would have to add code to see if the
file has been changed, etc. to use it.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 8090] 4.0.0 prevents GeoIP2 mmdb files to be updated
Posted by bu...@spamassassin.apache.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=8090
Henrik Krohns <ap...@hege.li> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |apache@hege.li
--- Comment #2 from Henrik Krohns <ap...@hege.li> ---
3.4 only opens database while it's queried, 4.0 keeps the file open for
performance since many plugins may use it.
Normal Windows shenanigans locking an opened file:
C:\Strawberry>perl -e "open(FOO, '<C:/temp/test.txt') and sleep(30)"
...
C:\temp>move test2.txt test.txt
Overwrite C:\temp\test.txt? (Yes/No/All): y
Access is denied.
There is no memory load option from what I see and maxmind code requires a
physical existing file, only thing we might do is copying the database to some
temporary location and use it from there. That would probably get very messy
too with processes coming and going.
--
You are receiving this mail because:
You are the assignee for the bug.