Re: Slow processing

["Daniel F. Savarese" <df...@savarese.org>]

In message <CD...@hkstar.com>, "Robert Edgar" w
rites:
>Hi there,
>I am trying to load and parse a web server log file (about 3mb on average)
>with a regex as follows
>"(.*)\\s(.*)\\s(.*)\\s\\[([^\\]]+)\\]\\s\"(.*)\\s(.*)\\s(.*)\"\\s(.*)\\s(.*)
>\\s\"(.*)\"\\s\"(.*)\"\\s\"(.*)\""
>
>problem is it is takeing forever, I am getting only about 15 lines a second
>
>code is something like
>
>while((logEntry = bufferedreader.readLine()) != null){
> if (matcher.contains(logEntry, pattern)) {
>   MatchResult result=matcher.getMatch();
> }
>}

Half of the problem is calling readLine().  You're converting char arrays
into strings and then back again when you search for a match.  If the
files are about 3MB, you'll do better by reading the entire file into
a char array first and then searching for matches.  The other half of
the problem may be the regular expression, which is causing a lot of
backtracking.  Try replacing .* with \S* and [^\s"]* where appropriate.

daniel



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

RE: Slow processing

[Robert Edgar <ro...@hkstar.com>]

Thanks
I changed the regexp to
"(^\\S*)\\s(\\S*)\\s(\\S*)\\s\\[([^\\]]+)\\]\\s\"(\\S*)\\s(\\S*)\\s([^\\s\"]
*)\"\\s(\\S*)\\s(\\S*)\\s\"([^\\s\"]*)\"\\s\"([^\"]*)\""

which has got me up to about 450 line a second but that is still slow though
I am stil using the readline, but using readline and a string tokenizer I
can get 10x this speed which seem to me to indicate that the readline is not
really a bottleneck but that the regex still is or is regex not really
designed for this sort of processing and I would be better just doing a
simple hard coded parse of the string?.

Rob

-----Original Message-----
From: Daniel F. Savarese [mailto:dfs@savarese.org]
Sent: Saturday, April 27, 2002 10:37 PM
To: ORO Users List
Subject: Re: Slow processing



In message <CD...@hkstar.com>, "Robert
Edgar" w
rites:
>Hi there,
>I am trying to load and parse a web server log file (about 3mb on average)
>with a regex as follows
>"(.*)\\s(.*)\\s(.*)\\s\\[([^\\]]+)\\]\\s\"(.*)\\s(.*)\\s(.*)\"\\s(.*)\\s(.*
)
>\\s\"(.*)\"\\s\"(.*)\"\\s\"(.*)\""
>
>problem is it is takeing forever, I am getting only about 15 lines a second
>
>code is something like
>
>while((logEntry = bufferedreader.readLine()) != null){
> if (matcher.contains(logEntry, pattern)) {
>   MatchResult result=matcher.getMatch();
> }
>}

Half of the problem is calling readLine().  You're converting char arrays
into strings and then back again when you search for a match.  If the
files are about 3MB, you'll do better by reading the entire file into
a char array first and then searching for matches.  The other half of
the problem may be the regular expression, which is causing a lot of
backtracking.  Try replacing .* with \S* and [^\s"]* where appropriate.

daniel



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>