You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2013/07/16 14:25:23 UTC
[Bug 6959] New: Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
Bug ID: 6959
Summary: Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Product: Spamassassin
Version: 3.4 SVN branch
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: P2
Component: Libraries
Assignee: dev@spamassassin.apache.org
Reporter: Mark.Martinec@ijs.si
This was reported by Axb on the mailing list:
| Malformed UTF-8 character (unexpected non-continuation byte 0x6e,
| immediately after start byte 0xf6) in transliteration (tr///)
| at /data/masscheckwork/weekly_mass_check/masses/../lib/Mail/SpamAssassin
| /DnsResolver.pm line 627.
I was able to reproduce this from a command-line spamassassin in
debug mode, after disabling the config option 'dns_options dns0x20'.
The setting of normalize_charset had no effect of the outcome.
Here is a pair of the indicative debug entries:
[52977] warn: Malformed UTF-8 character (unexpected non-cont
inuation byte 0xd6, immediately after start byte 0xf8) in transliteration
(tr///
) at /usr/local/lib/perl5/site_perl/5.18.0/Mail/SpamAssassin/DnsResolver.pm
line
627, <GEN0> line 9.
[52977] dbg: dns: dns reply to 3798/IN/A/www.moe.gov.cn<A3>
<AC><D1><A7><D0><C5><CD><F8><CF><B5><BD><CC><D3><FD><B2><BF><CE><A8><D2><BB><D6>
<B8><B6><A8><D1><A7><C0><FA><C8><CF><D6><A4><B2><E9><D1><AF><CD><F8><A3><AC><CD>
<F8><D6><B7>www.chsi.com.cn: NXDOMAIN
The message is in Chinese, with host names immediately preceded and
followed by text (no whitespace, as is normal with such text).
Looks like the URL parser should be less eager to include Chinese text
in URLs.
Luckily these are just warnings, but the actual question here is
what flagged the decoded text with an UTF-8 flag without checking
that it really is. If the decoding cannot be properly done, the result
should be just plain bytes, not characters.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
--- Comment #7 from Mark Martinec <Ma...@ijs.si> ---
Created attachment 5161
--> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5161&action=edit
Proposed patch
How about this fix (attached and comitted):
trunk:
Bug 6959 - Malformed UTF-8 character in transliteration at DnsResolver.pm
Sending lib/Mail/SpamAssassin/Dns.pm
Sending lib/Mail/SpamAssassin/DnsResolver.pm
Sending lib/Mail/SpamAssassin/Plugin/AskDNS.pm
Sending lib/Mail/SpamAssassin/Util.pm
Committed revision 1507148.
- Util.pm: encode character string as it comes from a Net::DNS::Packet
query section back into bytes before "RFC 1035 zone file format"-decoding
it to avoid producing an invalid character string
- Util.pm: rename fmt_dns_question_entry to decode_dns_question_entry
to better reflect its purpose
- DnsResolver::send(): encode a character string (with an utf8 flag)
as it comes from a SPF plugin to plain bytes in order to avoid
challenging Net::DNS
- DnsResolver.pm: issue an info message if a domain name is
a character string (with an utf8 flag) instead of plain bytes
(seems the source of these was exclusively the SPF plugin,
actually the underlying SPF perl module)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
Kevin A. McGrail <km...@pccc.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kmcgrail@pccc.com
--- Comment #3 from Kevin A. McGrail <km...@pccc.com> ---
Good catch. Do we need to open a bug with Net::DNS, require a specific version
or is there a workaround do you think?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
--- Comment #5 from Mark Martinec <Ma...@ijs.si> ---
Btw, I'll leave to somebody else to decide (and possibly fix) if a query
for an A record of a domain name:
www.moe.gov.cn<A3><AC><D1><A7><D0><C5><CD><F8><CF><B5><BD><CC><D3><FD>
<B2><BF><CE><A8><D2><BB><D6><B8><B6><A8<D1><A7><C0><FA><C8><CF><D6><A4>
<B2><E9><D1><AF><CD><F8><A3><AC><CD><F8><D6><B7>www.chsi.com.cn
(as generated from the attached sample message) is worth making
or worth the trouble of avoiding. As far as a DNS system is concerned
the query is valid, and my intention is to make sure it can be queried.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
Target Milestone|Undefined |3.4.1
--- Comment #8 from Mark Martinec <Ma...@ijs.si> ---
Closing, fixed.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
--- Comment #4 from Mark Martinec <Ma...@ijs.si> ---
I'm partly retracting my statement: we do get back from Net::DNS
in a question section an utf8 flagged string (despite us supplying
plain bytes). The resulting string is not actually invalid, it is
encoded as ascii in RFC 1035 zone format encoding, so it is
quite unnecessarily flagged as utf8. An invalid character string
is formed when we try to decode the RFC 1035 zone format encoding
while keeping the utf8 flag unchanged.
The cleanest fix would be for Net::DNS not to gratuitously turn
on the utf8 flag on RFC 1035 zone format encoded strings.
A fix/workaround on our side is to decode a returned character
string to bytes before doing the RFC 1035 zone format decoding
on them. ...
... (not to mention a wish/ramble that Net::DNS should start offering
an 8-bit clean API instead of requiring an application to decode/encode
domain names in a format intended for zone file editing. After all,
Perl as well as the DNS system are both 8-bit clean and allow arbitrary
bytes (including nulls) in labels and results)
The solution sounds like a simple call to Encoder::encode, although
that routine (and Encode::is_utf8 and Encode::_utf8_off) have a long
history of misbehaving on various versions of Perl, especially
when dealing with tainted strings.
Working on it...
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
--- Comment #6 from Kevin A. McGrail <km...@pccc.com> ---
(In reply to Mark Martinec from comment #5)
> Btw, I'll leave to somebody else to decide (and possibly fix) if a query
> for an A record of a domain name:
>
> www.moe.gov.cn<A3><AC><D1><A7><D0><C5><CD><F8><CF><B5><BD><CC><D3><FD>
> <B2><BF><CE><A8><D2><BB><D6><B8><B6><A8<D1><A7><C0><FA><C8><CF><D6><A4>
> <B2><E9><D1><AF><CD><F8><A3><AC><CD><F8><D6><B7>www.chsi.com.cn
>
> (as generated from the attached sample message) is worth making
> or worth the trouble of avoiding. As far as a DNS system is concerned
> the query is valid, and my intention is to make sure it can be queried.
As a lazy English-only speaker, I'm at a loss for comment because I've never
seen a valid link like that. Dear me.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
--- Comment #2 from Mark Martinec <Ma...@ijs.si> ---
I see, a bug/misfeature in Net::DNS : we give a properly "RFC 1035 zone
format" -encoded string of bytes (no utf-8 flag) to Net::DNS::Packet->new,
and it gratuitously flags it with an utf-8 flag in the packet->question
section, without checking that the given string of octets really
represents an UTF-8 encoded string, and without any reason to do so,
as DNS is 8-bit clean and works with octet strings / has no notion of
character sets and encodings.
When we later pull that packet->question string back and start processing
it, it blows in our face, as it is flagged as a proper character string,
but it isn't.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6959] Malformed UTF-8 character - in transliteration at
DnsResolver.pm line 627
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6959
--- Comment #1 from Mark Martinec <Ma...@ijs.si> ---
Created attachment 5159
--> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5159&action=edit
A sample message triggering this problem
--
You are receiving this mail because:
You are the assignee for the bug.