You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2011/08/15 17:53:16 UTC

[Bug 6649] New: sa-compile fails with re2c: error: line 207, column 2: unterminated string constant (missing ")

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

             Bug #: 6649
           Summary: sa-compile fails with re2c: error: line 207, column 2:
                    unterminated string constant (missing ")
           Product: Spamassassin
           Version: SVN Trunk (Latest Devel Version)
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Rules
        AssignedTo: dev@spamassassin.apache.org
        ReportedBy: Darxus@ChaosReigns.com
    Classification: Unclassified


Created attachment 4953
  --> https://issues.apache.org/SpamAssassin/attachment.cgi?id=4953
Problematic scanner2.re

This is being discussed on the users list today with the subject:  Subject:
Latest sa-update crashing sa-compile?

# sa-compile
Aug 15 11:47:46.221 [21218] info: generic: base extraction starting. this can
take a while...
Aug 15 11:47:46.221 [21218] info: generic: extracting from rules of type body_0
100%
[================================================================================================]
568.89 rules/sec 00m02s DONE
100%
[================================================================================================]
235.82 bases/sec 00m13s DONE
Aug 15 11:48:02.864 [21218] info: body_0: 1817 base strings extracted in 16
seconds
cd /tmp/.spamassassin21218ELu9TJtmp
cd Mail-SpamAssassin-CompiledRegexps-body_0
re2c -i -b -o scanner1.c scanner1.re
re2c -i -b -o scanner2.c scanner2.re
re2c: error: line 207, column 2: unterminated string constant (missing ")
command failed: exit 1


I'm guessing the problem is that the string in column 1 contains a newline
(carriage return, linefeed, or both).  Quite possibly from SOUGHT.  So maybe a
bug in SOUGHT's rule generation.  Or maybe it would be nice to handle this in
re2c?

scanner2.re attached.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Fred Wittekind IV <ro...@twister.dyndns.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rom@twister.dyndns.org

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Daniel J McDonald <da...@austinenergy.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dan.mcdonald@austinenergy.c
                   |                            |om

--- Comment #6 from Daniel J McDonald <da...@austinenergy.com> 2011-08-15 21:32:19 UTC ---
I applied the patch, ran sa-update and sa-compile and reloaded amavisd.  There
were no errors during the compilation.  The JM_SOUGHT_3 rule has hit 4 messages
in the past two minutes.  So, it appears to be working, but I don't know if it
will detect a message with the particular sub-rule that was escaped.

(at JM_SOUGHT_3 just hit another 12 messages...)

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #4 from Darxus <Da...@ChaosReigns.com> 2011-08-15 21:10:50 UTC ---
Created attachment 4954
  --> https://issues.apache.org/SpamAssassin/attachment.cgi?id=4954
Patch to escape linefeeds in re2c input, against trunk

This might do it:

  $output =~ s/\n/\\n/g; # escape linefeeds

Near the end of lib/Mail/SpamAssassin/Plugin/BodyRuleBaseExtractor.pm.  It
makes the error go away.  Still need to test that it's actually creating useful
output.

http://re2c.org/manual.html says ANSI-C escape sequences (which \n is) are
allowed in strings.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Kevin A. McGrail <km...@pccc.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kmcgrail@pccc.com

--- Comment #5 from Kevin A. McGrail <km...@pccc.com> 2011-08-15 21:15:16 UTC ---
(In reply to comment #4)
> Created attachment 4954 [details]
> Patch to escape linefeeds in re2c input, against trunk
> 
> This might do it:
> 
>   $output =~ s/\n/\\n/g; # escape linefeeds
> 
> Near the end of lib/Mail/SpamAssassin/Plugin/BodyRuleBaseExtractor.pm.  It
> makes the error go away.  Still need to test that it's actually creating useful
> output.
> 
> http://re2c.org/manual.html says ANSI-C escape sequences (which \n is) are
> allowed in strings.

Excellent.  In theory sounds very good.  Any of the users complaining able to
test?  I don't use re2c on any production systems :-(

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #10 from Darxus <Da...@ChaosReigns.com> 2011-08-15 22:53:06 UTC ---
From
/tmp/.spamassassin9294X8Ocvktmp/Mail-SpamAssassin-CompiledRegexps-body_0/scanner2.re,
the file generated by sa-compile for input into re2c:

"<82><D7>ٕh<FC><F2><BC><8E>uo~u뉒<A4>z<BF>
<B2><99>^]$<D2>
^\"<FD>ka<BD><86><8E>w<FF>b<8D>><8A>;:<AC><E9><CB><F5><A3>#<B7><AF>staf6<93><F7><A3><ED>(<BC><B2><B6>"
           {RET("__SEEK_4CZTNZ,[l=1]");}

The rule it was generated from:

body __SEEK_4CZTNZ 
/\x{82}\x{d7}\x{d9}\x{95}H\x{fc}\x{f2}\x{bc}\x{8e}uo\~U\x{eb}\x{89}\x{92}\x{a4}z\x{bf}\x{01}d\x{e6}\~\x{04}\x{c9}L\x{81}\x{d2}\{\{l\x{a9}\x{f4}\x{a1}\x{87}m\x{e7}\x{bb}\/n
\x{19}\(\x{fa}\x{a3}\x{dd}\x{ac}\x{1a}\[\.R\x{1a}z\x{e8}a\x{bb}\x{fb}\x{98}\[\$r_\x{02}y\x{ab}i\x{c1}\x{e1}
,\x{9f}\x{f9}\x{f9}\x{d3}2_\x{f9}\x{d0}\x{fc}2R\x{13}\x{ee}g\x{aa}\x{9b}TK\\
\x{1c}\x{f8}qu\x{0f}\x{e9}k\{\x{ab}\x{aa}q \+u\^f\x{a5}\x{8f}\x{db}_\}
(truncated by me)

See where the original rule says z\x{bf}\x{01} the scanner2.re file should say
z<BF><01>, but it says z<BF><newline>.  There's actually about 223 characters
in the much longer original rule that got replaced by that one linefeed.  Any
ideas why?

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #3 from Darxus <Da...@ChaosReigns.com> 2011-08-15 20:47:05 UTC ---
scanner2.re is written by sa-compile.  In trunk, scanner2.re is opened for
writing on line 389, and it looks like the corrupted output is printed on line
431:

431   print $re "\t",
432          
Mail::SpamAssassin::Plugin::BodyRuleBaseExtractor::fixup_re($regexp),
433           "            {RET(\"$reason\");}\n"
434           or die "error writing: $!";

re2c is then run on this output on line 455.

I'm thinking the best solution would be to modify
Mail::SpamAssassin::Plugin::BodyRuleBaseExtractor::fixup_re() to escape the
linefeed in its output, if re2c (which is not part of spamassassin) has a
mechanism for escaping it.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #19 from Mark Martinec <Ma...@ijs.si> 2011-11-14 18:28:24 UTC ---
Perhaps I should explain: the BodyRuleBaseExtractor spawns a perl interpreter
in a regexp debug mode and gives it a regexp to compile - letting perl's regexp
machinery do the hard work.

Then it tries to parse the perl debug output and make some sense/use of it.
The trouble is that perl regexp debug output truncates its lines in a report,
and does so at fixed line sizes, which do not take into account possible
multi-character byte sequences (like backslash-escaping). So doing it right
is tricky if not impossible. I'm not volunteering.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #18 from Mark Martinec <Ma...@ijs.si> 2011-11-14 18:13:21 UTC ---
> sa-compile is breaking on input data (other than low ASCII) in a way
> nobody has been able to explain.

What happens is explained in Comment 16.
It's just that nobody is willing / able / knowledgeable enough
to fix it.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Darxus <Da...@ChaosReigns.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |major

--- Comment #17 from Darxus <Da...@ChaosReigns.com> 2011-11-14 17:54:29 UTC ---
Increasing priority from "normal" to "major" - sa-compile is breaking on input
data (other than low ASCII) in a way nobody has been able to explain.  And
which there's no build test for.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #12 from bpoliakoff <be...@reed.edu> 2011-09-21 17:00:37 UTC ---
Created attachment 4964
  --> https://issues.apache.org/SpamAssassin/attachment.cgi?id=4964
another problematic re2c input file from the sought ruleset

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #13 from Darxus <Da...@ChaosReigns.com> 2011-09-21 17:11:39 UTC ---
It would be nice if the build tests included a test to sa-compile a rule with
the full set of (binary) characters, verifying the output matches the input (as
I explain it doesn't in comment 10).

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Mark Martinec <Ma...@ijs.si> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|3.4.0                       |Undefined

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Darxus <Da...@ChaosReigns.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jm@jmason.org
            Summary|sa-compile fails with re2c: |sa-compile fails on SOUGHT
                   |error: line 207, column 2:  |rule with re2c: error: line
                   |unterminated string         |207, column 2: unterminated
                   |constant (missing ")        |string constant (missing ")

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #15 from Mark Martinec <Ma...@ijs.si> 2011-09-22 15:31:43 UTC ---
rule:
body __SEEK_UWCSGI
  /\x{de}\x{bd}\x{c9}x\@FE;\x{ba}w\x{87}T-\x{cb}
   \x{98}\x{e4}2\x{e2}U\x{e9}<\x{cf}\x{ed}
   \x{f2}r\x{f9}\x{19}\x{87}\x{b5}\x{13}\x{ef}9\*/

result:
  "\x{de}\x{bd}\x{c9}x@fe;\x{ba}w\x{87}t-\x{cb}
   \x{98}"\x{e2}u\x{e9}<\x{cf}\x{ed}"

note that  \x{e4}2  was transformed into a  "

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Darxus <Da...@ChaosReigns.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |Darxus@ChaosReigns.com

--- Comment #1 from Darxus <Da...@ChaosReigns.com> 2011-08-15 16:03:29 UTC ---
Yeah, the 21st character of line 207 in the attachment is hex 0A, a linefeed,
the unix standard for a newline.  


head -n 208
/tmp/.spamassassin21134S0BApJtmp/Mail-SpamAssassin-CompiledRegexps-body_0/scanner2.re
| tail -n 2 
        "��ٕh��uo~u뉒�z�
��$� ^\"�ka���w�b�>�;:����#��staf6���(���"           
{RET("__SEEK_4CZTNZ,[l=1]");}


And it's from sought.  

/var/lib/spamassassin/3.004000/sought_rules_yerp_org# grep __SEEK_4CZTNZ *
20_sought.cf:body __SEEK_4CZTNZ  
...nevermind, that's too long.

# ls -l 20_sought.cf
-rw-r--r-- 1 root root 265630 2011-08-15 03:52 20_sought.cf
(Eastern Daylight Time)

On the users list it was mentioned that just running sa-update again gets this
problem to go away, at Mon, 15 Aug 2011 14:13:31 +0000.


So, SOUGHT or re2c?  Or both?

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

Re: Missing Content-Type on UPS Phish

Posted by Henrik Krohns <he...@hege.li>.
On Tue, Aug 16, 2011 at 09:04:35AM -0500, Daniel McDonald wrote:
> 
> On 8/16/11 9:01 AM, "Henrik Krohns" <he...@hege.li> wrote:
> 
> > On Tue, Aug 16, 2011 at 06:47:42AM -0400, Kevin A. McGrail wrote:
> >> 
> >>> Apart from trusting the filename extension? Examining the first
> >>> few bytes of the attachment for non-ASCII characters (excluding
> >>> UTF encoding markers) is the only thing that springs to mind.
> >>> 
> >>> File::Type perhaps? Or is that overkill?
> >>> 
> >> File::Type wouldn't be overkill if Content Type is missing.
> > 
> > What function in SA needs to know it correctly? I think it's safe to assume
> > such as text (what do MUAs do?).  We have the binary problem regardless.
> 
> If you mis-classify binary as text, you hit a lot of funky rules like
> UNWANTED_LANGUAGE_BODY

I'm well aware of that if you didn't notice? Some binary detection would be
fine to bypass TextCat etc specific cases.  I don't see why we need to
detect all the file formats in the world for SA to function.


Re: Missing Content-Type on UPS Phish

Posted by Daniel McDonald <da...@austinenergy.com>.


On 8/16/11 9:01 AM, "Henrik Krohns" <he...@hege.li> wrote:

> On Tue, Aug 16, 2011 at 06:47:42AM -0400, Kevin A. McGrail wrote:
>> 
>>> Apart from trusting the filename extension? Examining the first
>>> few bytes of the attachment for non-ASCII characters (excluding
>>> UTF encoding markers) is the only thing that springs to mind.
>>> 
>>> File::Type perhaps? Or is that overkill?
>>> 
>> File::Type wouldn't be overkill if Content Type is missing.
> 
> What function in SA needs to know it correctly? I think it's safe to assume
> such as text (what do MUAs do?).  We have the binary problem regardless.

If you mis-classify binary as text, you hit a lot of funky rules like
UNWANTED_LANGUAGE_BODY


-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281


Re: Missing Content-Type on UPS Phish

Posted by Henrik Krohns <he...@hege.li>.
On Tue, Aug 16, 2011 at 06:47:42AM -0400, Kevin A. McGrail wrote:
> 
> >Apart from trusting the filename extension? Examining the first
> >few bytes of the attachment for non-ASCII characters (excluding
> >UTF encoding markers) is the only thing that springs to mind.
> >
> >File::Type perhaps? Or is that overkill?
> >
> File::Type wouldn't be overkill if Content Type is missing.

What function in SA needs to know it correctly? I think it's safe to assume
such as text (what do MUAs do?).  We have the binary problem regardless.

I guess some rules could be useful to detect faked content type etc, but
I've yet to see a good "magic" module in CPAN..  I'd prefer SA had efficient
internal code to detect few chosen types that might be useful (if any).


Re: Missing Content-Type on UPS Phish

Posted by Justin Mason <jm...@jmason.org>.
On Tue, Aug 16, 2011 at 14:04, John Hardin <jh...@impsec.org> wrote:
> On Tue, 16 Aug 2011, Kevin A. McGrail wrote:
>
>> I also think if the Content Type is missing completely, wouldn't that be a
>> good rule?  Any indicators that this happens with Ham?
>
> Dunno. I put a test rule in my sandbox last night, we'll see how it goes.

at a guess I'd say it'd be promising.

in this case it's important to identify what "real" MUAs do, and just
do that.  it's quite possible that they also display the binary as
"text", since the message is broken.

Re: Missing Content-Type on UPS Phish

Posted by John Hardin <jh...@impsec.org>.
On Tue, 16 Aug 2011, Kevin A. McGrail wrote:

> I also think if the Content Type is missing completely, wouldn't that be a 
> good rule?  Any indicators that this happens with Ham?

Dunno. I put a test rule in my sandbox last night, we'll see how it goes.

-- 
  John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
  jhardin@impsec.org    FALaholic #11174     pgpk -a jhardin@impsec.org
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
   The difference is that Unix has had thirty years of technical
   types demanding basic functionality of it. And the Macintosh has
   had fifteen years of interface fascist users shaping its progress.
   Windows has the hairpin turns of the Microsoft marketing machine
   and that's all.                                    -- Red Drag Diva
-----------------------------------------------------------------------
  8 days until the 1932nd anniversary of the destruction of Pompeii

Re: Missing Content-Type on UPS Phish

Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
> Apart from trusting the filename extension? Examining the first few 
> bytes of the attachment for non-ASCII characters (excluding UTF 
> encoding markers) is the only thing that springs to mind.
>
> File::Type perhaps? Or is that overkill?
>
File::Type wouldn't be overkill if Content Type is missing.

I also think if the Content Type is missing completely, wouldn't that be 
a good rule?  Any indicators that this happens with Ham?

Regards,
KAM

Re: Missing Content-Type on UPS Phish

Posted by Henrik Krohns <he...@hege.li>.
On Mon, Aug 15, 2011 at 05:06:10PM -0700, John Hardin wrote:
> On Mon, 15 Aug 2011, Michael Parker wrote:
> 
> >On Aug 15, 2011, at 5:14 PM, bugzilla-daemon@bugzilla.spamassassin.org wrote:
> >
> >>https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649
> >>
> >>--- Comment #8 from Justin Mason <jm...@jmason.org> 2011-08-15 22:14:02 UTC ---
> >>it's a phish containing the following MIME headers:
> >>
> >>Content-Type: ;
> >>       name="UPS_document.zip"
> >>Content-Transfer-Encoding: base64
> >>Content-Disposition: attachment;
> >>       filename="UPS_document.zip"
> >>
> >>and a phishing attachment which is then being interpreted as text.
> >
> >That's because it's missing a Content-Type and SpamAssassin is
> >interpreting that as text/plain.  Anyone have any thoughts on how
> >to prevent that?
> 
> Apart from trusting the filename extension? Examining the first few
> bytes of the attachment for non-ASCII characters (excluding UTF
> encoding markers) is the only thing that springs to mind.
> 
> File::Type perhaps? Or is that overkill?

Sought can make easy assumptions, but if we want to "fix" SA, there are many
gotchas.

For example the problem with real text/plain binary attachments[1]..  it
would be nice to solve, but it's hard to come up with a "foolproof" binary
detector.  I think best thing to do is just to limit the problems any binary
data might cause.  We need to make sure rules and plugins don't crap out,
some binary test cases would be nice..

[1] https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6582


Re: Missing Content-Type on UPS Phish

Posted by John Hardin <jh...@impsec.org>.
On Mon, 15 Aug 2011, Michael Parker wrote:

> On Aug 15, 2011, at 5:14 PM, bugzilla-daemon@bugzilla.spamassassin.org wrote:
>
>> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649
>>
>> --- Comment #8 from Justin Mason <jm...@jmason.org> 2011-08-15 22:14:02 UTC ---
>> it's a phish containing the following MIME headers:
>>
>> Content-Type: ;
>>        name="UPS_document.zip"
>> Content-Transfer-Encoding: base64
>> Content-Disposition: attachment;
>>        filename="UPS_document.zip"
>>
>> and a phishing attachment which is then being interpreted as text.
>
> That's because it's missing a Content-Type and SpamAssassin is 
> interpreting that as text/plain.  Anyone have any thoughts on how to 
> prevent that?

Apart from trusting the filename extension? Examining the first few bytes 
of the attachment for non-ASCII characters (excluding UTF encoding 
markers) is the only thing that springs to mind.

File::Type perhaps? Or is that overkill?

-- 
  John Hardin KA7OHZ                    http://www.impsec.org/~jhardin/
  jhardin@impsec.org    FALaholic #11174     pgpk -a jhardin@impsec.org
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
   Usually Microsoft doesn't develop products, we buy products.
                           -- Arno Edelmann, Microsoft product manager
-----------------------------------------------------------------------
  Today: the 66th anniversary of the end of World War II

Missing Content-Type on UPS Phish

Posted by Michael Parker <pa...@pobox.com>.
On Aug 15, 2011, at 5:14 PM, bugzilla-daemon@bugzilla.spamassassin.org wrote:

> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649
> 
> --- Comment #8 from Justin Mason <jm...@jmason.org> 2011-08-15 22:14:02 UTC ---
> it's a phish containing the following MIME headers:
> 
> Content-Type: ;
>        name="UPS_document.zip"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment;
>        filename="UPS_document.zip"
> 
> and a phishing attachment which is then being interpreted as text.


That's because it's missing a Content-Type and SpamAssassin is interpreting that as text/plain.  Anyone have any thoughts on how to prevent that?

Michael


> 
> I've updated the backend to strip these out (hopefully).
> 
> -- 
> Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
> ------- You are receiving this mail because: -------
> You are the assignee for the bug.


[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #8 from Justin Mason <jm...@jmason.org> 2011-08-15 22:14:02 UTC ---
it's a phish containing the following MIME headers:

Content-Type: ;
        name="UPS_document.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
        filename="UPS_document.zip"

and a phishing attachment which is then being interpreted as text.

I've updated the backend to strip these out (hopefully).

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #20 from Darxus <Da...@ChaosReigns.com> ---
Just want to record that this came up again today: 
http://www.gossamer-threads.com/lists/spamassassin/users/176122

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 6649] sa-compile fails with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #2 from Darxus <Da...@ChaosReigns.com> 2011-08-15 16:09:13 UTC ---
I just tried sa-update and then re-running sa-compile, and this problem did
*not* go away.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #14 from Mark Martinec <Ma...@ijs.si> 2011-09-22 15:22:35 UTC ---
> Created attachment 4964 [details]
> another problematic re2c input file from the sought ruleset

Right, the re2c crashes with a core dump.

The culprit line is:

  "\336\275\311x@fe;\272w\207t-\313\230"\342u\351<\317\355"
    {RET("__SEEK_UWCSGI,[l=1]");

Note that a " within a string is unescaped!

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #9 from Darxus <Da...@ChaosReigns.com> 2011-08-15 22:18:36 UTC ---
Created attachment 4955
  --> https://issues.apache.org/SpamAssassin/attachment.cgi?id=4955
Sample email that hits __SEEK_4CZTNZ.

I can delete my compiled rules and verify this attached email hits
__SEEK_4CZTNZ (the rule with the linefeed problem) via JM_SOUGHT_3.  Then I run
sa-compile, and it *still* hits.  Which shouldn't be possible, due to 0x01 in
the pre-compiled rule being converted to 0x0A in the input to re2c.

How do I verify it's the compiled form of a rule that I'm hitting?

I'm pretty sure there's something fantastically broken in here.

$ spamassassin -D -t < SEEK_4CZTNZ.mbox 2>&1 | grep 4CZTNZ
Aug 15 18:11:40.458 [6210] dbg: rules: ran body rule __SEEK_4CZTNZ ======> got
hit: "[binary gunk]"

BTW, I created that .mbox file by just copying and pasting the regex (stuff
between the //s) into a perl print statement and outputting it to a file.  Then
just adding a From header and blank line before it.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

bpoliakoff <be...@reed.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |benp@reed.edu

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

Mark Martinec <Ma...@ijs.si> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|Undefined                   |3.4.0

--- Comment #16 from Mark Martinec <Ma...@ijs.si> 2011-09-22 20:19:37 UTC ---
I'm beginning to understand what may have happened.

> \x{bf}\x{01}d\x{e6}...
> See where the original rule says z\x{bf}\x{01} the scanner2.re file
> should say z<BF><01>, but it says z<BF><newline>

See \x{01}d - note that \xd is a code for a newline.

> note that  \x{e4}2  was transformed into a  "

Cannot yet fully explain this, but note that \x22 is a code for "

Seems like the trouble originates from how the output
of a "perl -Mre=debug" command is being parsed by
Mail::SpamAssassin::Plugin::BodyRuleBaseExtractor::extract_hints().

The extract_hints() has to deal with truncted regexp debug output
(lines with "..." at their end). The RE debug output of perl has
changed subtly between 5.8 and 5.10:

  perl 5.8:              <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...>(18)
  perl 5.10, 5.12, 5.14: <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>... (18)

I'm not sure this is still being properly handled.
A side effect may be that a \x{dd} in a regexp may be arbitrarily
chopped for long strings, and an  \x{  may join later with some
text that follows. I do not fully understand what happens next
and how to properly fix it.

As a small hardening measure, I added escaping of NL and NULL
characters, following a suggestion in Comment 4. It does not solve
the origin of the trouble, but at least it limits the damage.

I'm also not sure that re2c knows how to handle null characters
(escaped or raw) in regexp strings in a scanner*.re source.
There are several of these in current SOUGHT rules.

trunk:
  Bug 6649: sa-compile fails on SOUGHT rule with re2c: unterminated
  string constant - protect special characters, some debuggings aids,
  perl -Mre=debug changed its output format with perl 5.10
Sending lib/Mail/SpamAssassin/Plugin/BodyRuleBaseExtractor.pm
Sending lib/Mail/SpamAssassin/Util.pm
Sending sa-compile.raw
Committed revision 1174349.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #11 from bpoliakoff <be...@reed.edu> 2011-09-21 16:59:41 UTC ---
We're seeing further instances of this behavior with SOUGHT rules.  I'm
attaching another file that causes re2c to segfault.

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #7 from Darxus <Da...@ChaosReigns.com> 2011-08-15 21:42:11 UTC ---
So... I'm trying to figure out exactly what this __SEEK_4CZTNZ rule was
supposed to hit, so I can try testing it.

\x{0a} would be the hex encoding of a linefeed character (0a).  But it doesn't
exist anywhere in the pre-compiled rule.  Where that character comes from,
there's a \x{01}.  Why is hex character 01 being converted to a linefeed, hex
character 0A for input into re2c??  Can it... result in a compiled rule that
works?

-- 
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 6649] sa-compile fails on SOUGHT rule with re2c: error: line 207, column 2: unterminated string constant (missing ")

Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6649

--- Comment #21 from Darxus <Da...@ChaosReigns.com> ---
There is a related bug 6336, which was fixed in 3.3.2, which Mark Martinec
believes my previous comment was actually caused by.

Summary of the status of this bug is in comment 19 and comment 16.

-- 
You are receiving this mail because:
You are the assignee for the bug.