You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jmeter.apache.org by sebb <se...@gmail.com> on 2012/12/08 16:40:47 UTC
Re: svn commit: r1418666 - /jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
On 8 December 2012 13:03, <pm...@apache.org> wrote:
> Author: pmouawad
> Date: Sat Dec 8 13:03:30 2012
> New Revision: 1418666
>
> URL: http://svn.apache.org/viewvc?rev=1418666&view=rev
> Log:
> Bug 54268 - Improve memory usage
>
> Bugzilla Id: 54268
>
> Modified:
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
>
> Modified: jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> URL: http://svn.apache.org/viewvc/jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java?rev=1418666&r1=1418665&r2=1418666&view=diff
> ==============================================================================
> --- jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java (original)
> +++ jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java Sat Dec 8 13:03:30 2012
> @@ -110,11 +110,11 @@ public class RegexExtractor extends Abst
> if (defaultValue.length() > 0){// Only replace default if it is provided
> vars.put(refName, defaultValue);
> }
> -
> -
> + Perl5Matcher matcher = JMeterUtils.getMatcher();
> String regex = getRegex();
> + Pattern pattern = JMeterUtils.getPatternCache().getPattern(regex, Perl5Compiler.READ_ONLY_MASK);
> try {
> - List<MatchResult> matches = processMatches(regex, previousResult, matchNumber, vars);
> + List<MatchResult> matches = processMatches(pattern, regex, previousResult, matchNumber, vars);
> int prevCount = 0;
> String prevString = vars.get(refName + REF_MATCH_NR);
> if (prevString != null) {
> @@ -162,6 +162,22 @@ public class RegexExtractor extends Abst
> }
> } catch (MalformedCachePatternException e) {
> log.warn("Error in pattern: " + regex);
> + } finally {
> + clearMatcherMemory(matcher, pattern);
> + }
> + }
> +
> + /**
> + * Hack to make matcher clean the two internal buffers it keeps in memory which size is equivalent to
> + * the unzipped page size
> + * @param matcher {@link Perl5Matcher}
> + * @param pattern Pattern
> + */
> + private final void clearMatcherMemory(Perl5Matcher matcher, Pattern pattern) {
> + try {
> + matcher.matches("", pattern); // $NON-NLS-1$
> + } catch (Exception e) {
> + // NOOP
> }
Perhaps needs to be a utility method?
May be useful from other classes.
[However, if we end up dropping ORO it will become irrelevant.]
Does the Java regex processor behave in the same way?
Or is it better at tidying up?
> }
>
> @@ -180,13 +196,12 @@ public class RegexExtractor extends Abst
> return inputString;
> }
>
> - private List<MatchResult> processMatches(String regex, SampleResult result, int matchNumber, JMeterVariables vars) {
> + private List<MatchResult> processMatches(Pattern pattern, String regex, SampleResult result, int matchNumber, JMeterVariables vars) {
> if (log.isDebugEnabled()) {
> log.debug("Regex = " + regex);
> }
>
> Perl5Matcher matcher = JMeterUtils.getMatcher();
> - Pattern pattern = JMeterUtils.getPatternCache().getPattern(regex, Perl5Compiler.READ_ONLY_MASK);
> List<MatchResult> matches = new ArrayList<MatchResult>();
> int found = 0;
>
>
>
Re: svn commit: r1418666 - /jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
Posted by Philippe Mouawad <ph...@gmail.com>.
On Mon, Dec 10, 2012 at 8:00 PM, sebb <se...@gmail.com> wrote:
> On 8 December 2012 22:10, Philippe Mouawad <ph...@gmail.com>
> wrote:
> > Hello sebb,
> >
> > I will put it in some utility class, which one would be the best in your
> > opinion ?
> >
> > I didn't test yet with Java 6 but Java 6 class has a reset method
> although
> > It has an text zone that contains matched test .
> >
> > But regarding the replacement you are talking about, it seems to me it's
> > not that transparent as reading javadocs:
> >
> > -
> http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html
> > -
> >
> http://jakarta.apache.org/oro/api/org/apache/oro/text/regex/Perl5Matcher.html
> >
> > Java class does not seem to fully support PERL5 syntax no ? so it could
> > break existing test plan.
>
> AFAIK, ORO also does not support the full Perl5 syntax either.
>
> So it would be a question of checking whether the ORO features [1] [2]
> are supported by Java.
>
> [1]
> http://jakarta.apache.org/oro/api/org/apache/oro/text/regex/package-summary.html
> [2]
> http://www.savarese.org/oro/docs/OROMatcher/Syntax.html#Perl5Expressions
>
> One obvious omission from Java 6 is embedded comments.
> However, they aren't essential.
>
> > So if we switch it seems to me we should have a configuration of regexp
> > engine.
>
> I suppose that might be necessary initially, but ideally we would drop
> support for ORO (in a later release) if Java 6 is compatible.
> If Java 6 is not fully compatible, then it depends on how likely the
> features are to have been used, whether it's possible to detect that
> it has been used (so we could potentially report a suitable error) and
> whether there is a work-round
>
> Java 6 has features not supported by ORO, and at least some of these
> have been used by end-users.
>
> Ok
> >
> > Anyway, I made a performance comparison on a test plan calling:
> >
> > - 2 samples that generate pages of 280 and 200 Ko, gzipped
> respectively
> > to 1714 and 1475 bytes
> > - 200 threads
> > - 3 s pause between each sample
> > - 2 assertions
> > - 3 regexp
> >
> > Results are interesting:
> >
> > - Memory and GC behaviours are greatly improved:
> > - much less CPU consumed by GC activity 3 to 15% with 2.8, 2 to 8%
> > with 2.9
> > - A Full GC drops memory to 100 Mb while with 2.8, it drops only to
> > 200 Mb
>
> Sorry, I don't follow - what is being compared with what?
>
> I compared JMeter 2.8 with trunk version before the commits where I cached
responseAsString on the test plan I commited called
PerformanceTestPlanMemoryThread.jmx (which requires memory.jsp to be put in
webapps/examples/jsp folder of tomcat)
> Plan is maybe a bit agressive regarding page sizes. 200 Kb.
> >
> > Maybe we should create some kind of Base Test Plan.
> >
> > I think there is another potential improvement we could make which would
> > be to cache in SampleResult#getResponseDataAsString the result of
> > conversion.
> >
> > As if we have 3 regexp String will be decoded 3 times (CPU) and filled 3
> > times (memory).
>
> Not sure it uses extra memory - surely the decoded string will be
> dropped (and made eligible for GC) once used?
>
> No in this case it consumes CPU.
> >
> > What do you think about it ?
>
> OK, but remember that it will affect the total data stored.
>
Yes but with last commits mixing cached responseAsString + cleanAfter
sample I have memory close to what it was before caching and a neat CPU
decrease as byte to String decoding was called 4 times instead of one now.
>
> >
> > Regards
> >
> > Philippe
> >
> > On Sat, Dec 8, 2012 at 4:40 PM, sebb <se...@gmail.com> wrote:
> >
> >> On 8 December 2012 13:03, <pm...@apache.org> wrote:
> >> > Author: pmouawad
> >> > Date: Sat Dec 8 13:03:30 2012
> >> > New Revision: 1418666
> >> >
> >> > URL: http://svn.apache.org/viewvc?rev=1418666&view=rev
> >> > Log:
> >> > Bug 54268 - Improve memory usage
> >> >
> >> > Bugzilla Id: 54268
> >> >
> >> > Modified:
> >> >
> >>
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> >> >
> >> > Modified:
> >>
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> >> > URL:
> >>
> http://svn.apache.org/viewvc/jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java?rev=1418666&r1=1418665&r2=1418666&view=diff
> >> >
> >>
> ==============================================================================
> >> > ---
> >>
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> >> (original)
> >> > +++
> >>
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> >> Sat Dec 8 13:03:30 2012
> >> > @@ -110,11 +110,11 @@ public class RegexExtractor extends Abst
> >> > if (defaultValue.length() > 0){// Only replace default if it
> is
> >> provided
> >> > vars.put(refName, defaultValue);
> >> > }
> >> > -
> >> > -
> >> > + Perl5Matcher matcher = JMeterUtils.getMatcher();
> >> > String regex = getRegex();
> >> > + Pattern pattern =
> >> JMeterUtils.getPatternCache().getPattern(regex,
> >> Perl5Compiler.READ_ONLY_MASK);
> >> > try {
> >> > - List<MatchResult> matches = processMatches(regex,
> >> previousResult, matchNumber, vars);
> >> > + List<MatchResult> matches = processMatches(pattern,
> regex,
> >> previousResult, matchNumber, vars);
> >> > int prevCount = 0;
> >> > String prevString = vars.get(refName + REF_MATCH_NR);
> >> > if (prevString != null) {
> >> > @@ -162,6 +162,22 @@ public class RegexExtractor extends Abst
> >> > }
> >> > } catch (MalformedCachePatternException e) {
> >> > log.warn("Error in pattern: " + regex);
> >> > + } finally {
> >> > + clearMatcherMemory(matcher, pattern);
> >> > + }
> >> > + }
> >> > +
> >> > + /**
> >> > + * Hack to make matcher clean the two internal buffers it keeps
> in
> >> memory which size is equivalent to
> >> > + * the unzipped page size
> >> > + * @param matcher {@link Perl5Matcher}
> >> > + * @param pattern Pattern
> >> > + */
> >> > + private final void clearMatcherMemory(Perl5Matcher matcher,
> Pattern
> >> pattern) {
> >> > + try {
> >> > + matcher.matches("", pattern); // $NON-NLS-1$
> >> > + } catch (Exception e) {
> >> > + // NOOP
> >> > }
> >>
> >> Perhaps needs to be a utility method?
> >> May be useful from other classes.
> >>
> >> [However, if we end up dropping ORO it will become irrelevant.]
> >>
> >> Does the Java regex processor behave in the same way?
> >> Or is it better at tidying up?
> >>
> >> > }
> >> >
> >> > @@ -180,13 +196,12 @@ public class RegexExtractor extends Abst
> >> > return inputString;
> >> > }
> >> >
> >> > - private List<MatchResult> processMatches(String regex,
> SampleResult
> >> result, int matchNumber, JMeterVariables vars) {
> >> > + private List<MatchResult> processMatches(Pattern pattern, String
> >> regex, SampleResult result, int matchNumber, JMeterVariables vars) {
> >> > if (log.isDebugEnabled()) {
> >> > log.debug("Regex = " + regex);
> >> > }
> >> >
> >> > Perl5Matcher matcher = JMeterUtils.getMatcher();
> >> > - Pattern pattern =
> >> JMeterUtils.getPatternCache().getPattern(regex,
> >> Perl5Compiler.READ_ONLY_MASK);
> >> > List<MatchResult> matches = new ArrayList<MatchResult>();
> >> > int found = 0;
> >> >
> >> >
> >> >
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>
--
Cordialement.
Philippe Mouawad.
Re: svn commit: r1418666 - /jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
Posted by sebb <se...@gmail.com>.
On 8 December 2012 22:10, Philippe Mouawad <ph...@gmail.com> wrote:
> Hello sebb,
>
> I will put it in some utility class, which one would be the best in your
> opinion ?
>
> I didn't test yet with Java 6 but Java 6 class has a reset method although
> It has an text zone that contains matched test .
>
> But regarding the replacement you are talking about, it seems to me it's
> not that transparent as reading javadocs:
>
> - http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html
> -
> http://jakarta.apache.org/oro/api/org/apache/oro/text/regex/Perl5Matcher.html
>
> Java class does not seem to fully support PERL5 syntax no ? so it could
> break existing test plan.
AFAIK, ORO also does not support the full Perl5 syntax either.
So it would be a question of checking whether the ORO features [1] [2]
are supported by Java.
[1] http://jakarta.apache.org/oro/api/org/apache/oro/text/regex/package-summary.html
[2] http://www.savarese.org/oro/docs/OROMatcher/Syntax.html#Perl5Expressions
One obvious omission from Java 6 is embedded comments.
However, they aren't essential.
> So if we switch it seems to me we should have a configuration of regexp
> engine.
I suppose that might be necessary initially, but ideally we would drop
support for ORO (in a later release) if Java 6 is compatible.
If Java 6 is not fully compatible, then it depends on how likely the
features are to have been used, whether it's possible to detect that
it has been used (so we could potentially report a suitable error) and
whether there is a work-round
Java 6 has features not supported by ORO, and at least some of these
have been used by end-users.
>
> Anyway, I made a performance comparison on a test plan calling:
>
> - 2 samples that generate pages of 280 and 200 Ko, gzipped respectively
> to 1714 and 1475 bytes
> - 200 threads
> - 3 s pause between each sample
> - 2 assertions
> - 3 regexp
>
> Results are interesting:
>
> - Memory and GC behaviours are greatly improved:
> - much less CPU consumed by GC activity 3 to 15% with 2.8, 2 to 8%
> with 2.9
> - A Full GC drops memory to 100 Mb while with 2.8, it drops only to
> 200 Mb
Sorry, I don't follow - what is being compared with what?
> Plan is maybe a bit agressive regarding page sizes. 200 Kb.
>
> Maybe we should create some kind of Base Test Plan.
>
> I think there is another potential improvement we could make which would
> be to cache in SampleResult#getResponseDataAsString the result of
> conversion.
>
> As if we have 3 regexp String will be decoded 3 times (CPU) and filled 3
> times (memory).
Not sure it uses extra memory - surely the decoded string will be
dropped (and made eligible for GC) once used?
>
> What do you think about it ?
OK, but remember that it will affect the total data stored.
>
> Regards
>
> Philippe
>
> On Sat, Dec 8, 2012 at 4:40 PM, sebb <se...@gmail.com> wrote:
>
>> On 8 December 2012 13:03, <pm...@apache.org> wrote:
>> > Author: pmouawad
>> > Date: Sat Dec 8 13:03:30 2012
>> > New Revision: 1418666
>> >
>> > URL: http://svn.apache.org/viewvc?rev=1418666&view=rev
>> > Log:
>> > Bug 54268 - Improve memory usage
>> >
>> > Bugzilla Id: 54268
>> >
>> > Modified:
>> >
>> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
>> >
>> > Modified:
>> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
>> > URL:
>> http://svn.apache.org/viewvc/jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java?rev=1418666&r1=1418665&r2=1418666&view=diff
>> >
>> ==============================================================================
>> > ---
>> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
>> (original)
>> > +++
>> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
>> Sat Dec 8 13:03:30 2012
>> > @@ -110,11 +110,11 @@ public class RegexExtractor extends Abst
>> > if (defaultValue.length() > 0){// Only replace default if it is
>> provided
>> > vars.put(refName, defaultValue);
>> > }
>> > -
>> > -
>> > + Perl5Matcher matcher = JMeterUtils.getMatcher();
>> > String regex = getRegex();
>> > + Pattern pattern =
>> JMeterUtils.getPatternCache().getPattern(regex,
>> Perl5Compiler.READ_ONLY_MASK);
>> > try {
>> > - List<MatchResult> matches = processMatches(regex,
>> previousResult, matchNumber, vars);
>> > + List<MatchResult> matches = processMatches(pattern, regex,
>> previousResult, matchNumber, vars);
>> > int prevCount = 0;
>> > String prevString = vars.get(refName + REF_MATCH_NR);
>> > if (prevString != null) {
>> > @@ -162,6 +162,22 @@ public class RegexExtractor extends Abst
>> > }
>> > } catch (MalformedCachePatternException e) {
>> > log.warn("Error in pattern: " + regex);
>> > + } finally {
>> > + clearMatcherMemory(matcher, pattern);
>> > + }
>> > + }
>> > +
>> > + /**
>> > + * Hack to make matcher clean the two internal buffers it keeps in
>> memory which size is equivalent to
>> > + * the unzipped page size
>> > + * @param matcher {@link Perl5Matcher}
>> > + * @param pattern Pattern
>> > + */
>> > + private final void clearMatcherMemory(Perl5Matcher matcher, Pattern
>> pattern) {
>> > + try {
>> > + matcher.matches("", pattern); // $NON-NLS-1$
>> > + } catch (Exception e) {
>> > + // NOOP
>> > }
>>
>> Perhaps needs to be a utility method?
>> May be useful from other classes.
>>
>> [However, if we end up dropping ORO it will become irrelevant.]
>>
>> Does the Java regex processor behave in the same way?
>> Or is it better at tidying up?
>>
>> > }
>> >
>> > @@ -180,13 +196,12 @@ public class RegexExtractor extends Abst
>> > return inputString;
>> > }
>> >
>> > - private List<MatchResult> processMatches(String regex, SampleResult
>> result, int matchNumber, JMeterVariables vars) {
>> > + private List<MatchResult> processMatches(Pattern pattern, String
>> regex, SampleResult result, int matchNumber, JMeterVariables vars) {
>> > if (log.isDebugEnabled()) {
>> > log.debug("Regex = " + regex);
>> > }
>> >
>> > Perl5Matcher matcher = JMeterUtils.getMatcher();
>> > - Pattern pattern =
>> JMeterUtils.getPatternCache().getPattern(regex,
>> Perl5Compiler.READ_ONLY_MASK);
>> > List<MatchResult> matches = new ArrayList<MatchResult>();
>> > int found = 0;
>> >
>> >
>> >
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
Re: svn commit: r1418666 - /jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello sebb,
I will put it in some utility class, which one would be the best in your
opinion ?
I didn't test yet with Java 6 but Java 6 class has a reset method although
It has an text zone that contains matched test .
But regarding the replacement you are talking about, it seems to me it's
not that transparent as reading javadocs:
- http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html
-
http://jakarta.apache.org/oro/api/org/apache/oro/text/regex/Perl5Matcher.html
Java class does not seem to fully support PERL5 syntax no ? so it could
break existing test plan.
So if we switch it seems to me we should have a configuration of regexp
engine.
Anyway, I made a performance comparison on a test plan calling:
- 2 samples that generate pages of 280 and 200 Ko, gzipped respectively
to 1714 and 1475 bytes
- 200 threads
- 3 s pause between each sample
- 2 assertions
- 3 regexp
Results are interesting:
- Memory and GC behaviours are greatly improved:
- much less CPU consumed by GC activity 3 to 15% with 2.8, 2 to 8%
with 2.9
- A Full GC drops memory to 100 Mb while with 2.8, it drops only to
200 Mb
Plan is maybe a bit agressive regarding page sizes. 200 Kb.
Maybe we should create some kind of Base Test Plan.
I think there is another potential improvement we could make which would
be to cache in SampleResult#getResponseDataAsString the result of
conversion.
As if we have 3 regexp String will be decoded 3 times (CPU) and filled 3
times (memory).
What do you think about it ?
Regards
Philippe
On Sat, Dec 8, 2012 at 4:40 PM, sebb <se...@gmail.com> wrote:
> On 8 December 2012 13:03, <pm...@apache.org> wrote:
> > Author: pmouawad
> > Date: Sat Dec 8 13:03:30 2012
> > New Revision: 1418666
> >
> > URL: http://svn.apache.org/viewvc?rev=1418666&view=rev
> > Log:
> > Bug 54268 - Improve memory usage
> >
> > Bugzilla Id: 54268
> >
> > Modified:
> >
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> >
> > Modified:
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> > URL:
> http://svn.apache.org/viewvc/jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java?rev=1418666&r1=1418665&r2=1418666&view=diff
> >
> ==============================================================================
> > ---
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> (original)
> > +++
> jmeter/trunk/src/components/org/apache/jmeter/extractor/RegexExtractor.java
> Sat Dec 8 13:03:30 2012
> > @@ -110,11 +110,11 @@ public class RegexExtractor extends Abst
> > if (defaultValue.length() > 0){// Only replace default if it is
> provided
> > vars.put(refName, defaultValue);
> > }
> > -
> > -
> > + Perl5Matcher matcher = JMeterUtils.getMatcher();
> > String regex = getRegex();
> > + Pattern pattern =
> JMeterUtils.getPatternCache().getPattern(regex,
> Perl5Compiler.READ_ONLY_MASK);
> > try {
> > - List<MatchResult> matches = processMatches(regex,
> previousResult, matchNumber, vars);
> > + List<MatchResult> matches = processMatches(pattern, regex,
> previousResult, matchNumber, vars);
> > int prevCount = 0;
> > String prevString = vars.get(refName + REF_MATCH_NR);
> > if (prevString != null) {
> > @@ -162,6 +162,22 @@ public class RegexExtractor extends Abst
> > }
> > } catch (MalformedCachePatternException e) {
> > log.warn("Error in pattern: " + regex);
> > + } finally {
> > + clearMatcherMemory(matcher, pattern);
> > + }
> > + }
> > +
> > + /**
> > + * Hack to make matcher clean the two internal buffers it keeps in
> memory which size is equivalent to
> > + * the unzipped page size
> > + * @param matcher {@link Perl5Matcher}
> > + * @param pattern Pattern
> > + */
> > + private final void clearMatcherMemory(Perl5Matcher matcher, Pattern
> pattern) {
> > + try {
> > + matcher.matches("", pattern); // $NON-NLS-1$
> > + } catch (Exception e) {
> > + // NOOP
> > }
>
> Perhaps needs to be a utility method?
> May be useful from other classes.
>
> [However, if we end up dropping ORO it will become irrelevant.]
>
> Does the Java regex processor behave in the same way?
> Or is it better at tidying up?
>
> > }
> >
> > @@ -180,13 +196,12 @@ public class RegexExtractor extends Abst
> > return inputString;
> > }
> >
> > - private List<MatchResult> processMatches(String regex, SampleResult
> result, int matchNumber, JMeterVariables vars) {
> > + private List<MatchResult> processMatches(Pattern pattern, String
> regex, SampleResult result, int matchNumber, JMeterVariables vars) {
> > if (log.isDebugEnabled()) {
> > log.debug("Regex = " + regex);
> > }
> >
> > Perl5Matcher matcher = JMeterUtils.getMatcher();
> > - Pattern pattern =
> JMeterUtils.getPatternCache().getPattern(regex,
> Perl5Compiler.READ_ONLY_MASK);
> > List<MatchResult> matches = new ArrayList<MatchResult>();
> > int found = 0;
> >
> >
> >
>
--
Cordialement.
Philippe Mouawad.