You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jmeter.apache.org by bu...@apache.org on 2012/08/20 22:39:34 UTC
[Bug 53748] New: timer for the access log sampler
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
Priority: P2
Bug ID: 53748
Assignee: issues@jmeter.apache.org
Summary: timer for the access log sampler
Severity: normal
Classification: Unclassified
OS: Windows Vista
Reporter: enricjaen@yahoo.es
Hardware: PC
Status: NEW
Version: Nightly (Please specify date)
Component: HTTP
Product: JMeter
Created attachment 29255
--> https://issues.apache.org/bugzilla/attachment.cgi?id=29255&action=edit
contains the code of access log timer, plan generator, and tests
The actual access log sampler has two drawbacks: a) the rate of the requests
doens't replay the access log as the sampler doesn't considers the existing
delay betweeen requests, and b) the concurrency is neither real, as there isn't
distintion of which IP sends the request.
This enhancement adds two components: a timer for the access log sampler, and a
generator that creates a test plan for the access log, with one thread group
for each IP.
With these components, replaying an access log is now much more real, as
samples are sent respecting the delays in the access log, and each IP is
represented by a different thread group so the real access log conccurrency is
replayed.
With the generator you can also generate a test plan for a specific time range
of the access log being replayed, in case you don't want to replay the whole
access log.
I attach here the patch, and in the dev mailing list I will attach a PDF with
the design details and some screenshots.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #5 from ejaenv <en...@yahoo.es> ---
Created attachment 30434
--> https://issues.apache.org/bugzilla/attachment.cgi?id=30434&action=edit
writtable documentation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |p.mouawad@ubik-ingenierie.c
| |om
Hardware|PC |All
Version|Nightly (Please specify |2.7
|date) |
OS|Windows Vista |All
Severity|normal |enhancement
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #30440|0 |1
is patch| |
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #17 from Sebb <se...@apache.org> ---
(In reply to ejaenv from comment #16)
>
> How can the thread stop itself? I am assuming that threads (samplers) don't
> read themselves the log to avoid the file handler problem, but that there is
> a queue that feed the samplers (as you pointed). Someone has to notify them
> to stop, or tell them h
For example, the queue can contain a special marker entry that denotes EOF.
This technique is used in AsynchSampleSender - see FINAL_EVENT.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #6 from ejaenv <en...@yahoo.es> ---
Hi Philippe,
Sorry me too!
Thanks for your questions. I hope to clarify them:
- Why read lines from access log into memory instead of reading them from file
?
This is needed to keep at runtime the delays as similar as in the log file.
Reading them from disc caused delay deviations due to disc major latency.
- There is something that's not clear for me, does Access Log TImeer suppose
that access log file has been split into as many file as IPs and that only the
logs concerning the Thread Group under which Timer is located are passed to it
? because if not, it seems to me delay computation is weird ? could you clarify
?
yes, the generator (1) splits the log file in the way you say, and also
generates the jmeter plan file. Keep in mind that this computation is done
before running jmeter, in a preparation phase.
(1) the generator is implemented actually as a junit test, but the idea is to
have a command tool.
- Could you submit the PDF into a writable format ?
I attach it.
Best regards,
/Enric
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #14 from ejaenv <en...@yahoo.es> ---
so do you propose to create and start all the threads at the very test start?
how can you finalize the threads without knowing the number of requests to
send? are you proposing that all the threads live the whole test?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |PatchAvailable
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #15 from Sebb <se...@apache.org> ---
(In reply to ejaenv from comment #14)
> so do you propose to create and start all the threads at the very test start?
Ideally, yes.
> how can you finalize the threads without knowing the number of requests to
> send?
As already stated (twice), a thread can stop itself when it has no more samples
to process.
> are you proposing that all the threads live the whole test?
No.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #12 from ejaenv <en...@yahoo.es> ---
but knowing the number of IPs is not necessary as threads can be created in
demand whenever the sampler reads a new IP.
what It's needed is the number of requests per IP so a thread can finalize once
it has sent all its requests.
The preprocessing could be done by the sampler itself before it starts creating
threads.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #18 from ejaenv <en...@yahoo.es> ---
sorry I don't follow you.. Supose a 1-hour log where an IP just sends 1 request
at time-0. What do you mean, the thread should stop 1 hour later (at EOF) of
after sending the request? (I am fine with any option, I just propose in the
second case that before start sending requests someone read the whole log and
configure the thread to stop after 1 request in this case)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #3 from ejaenv <en...@yahoo.es> ---
Hi Philippe! I'll answer in a few days. Cheers
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #16 from ejaenv <en...@yahoo.es> ---
>> how can you finalize the threads without knowing the number of requests to
>> send?
>As already stated (twice), a thread can stop itself when it has no more samples >to process.
How can the thread stop itself? I am assuming that threads (samplers) don't
read themselves the log to avoid the file handler problem, but that there is a
queue that feed the samplers (as you pointed). Someone has to notify them to
stop, or tell them h
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #13 from Sebb <se...@apache.org> ---
(In reply to ejaenv from comment #12)
> but knowing the number of IPs is not necessary as threads can be created in
> demand whenever the sampler reads a new IP.
Thread creation is relatively expensive, so not ideal to create one as part of
the main test.
> what It's needed is the number of requests per IP so a thread can finalize
> once it has sent all its requests.
A thread can stop itself when it has nothing more to do.
> The preprocessing could be done by the sampler itself before it starts
> creating threads.
Samplers don't normally create threads in JMeter (except for download
resources).
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #29255|0 |1
is patch| |
Attachment #29255|application/octet-stream |text/plain
mime type| |
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #1 from ejaenv <en...@yahoo.es> ---
Created attachment 29256
--> https://issues.apache.org/bugzilla/attachment.cgi?id=29256&action=edit
design details and screenshots
I attach here the pdf
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #29255|0 |1
is obsolete| |
--- Comment #8 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Created attachment 30440
--> https://issues.apache.org/bugzilla/attachment.cgi?id=30440&action=edit
Updated patch
Attached is the updated patch with the following:
- Some code cleanup to follow naming conventions and others
- AccessLogTimer now only reads the time in memory instead of samples content
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #10 from ejaenv <en...@yahoo.es> ---
Yes this is a beta version and surely the design needs to be discussed. As it
is now:
>If the access log does need to be reformatted, it should be done as a separate >stage before starting the test proper, and the output should probably be >reformatted to make subsequent parsing easier.
Yes, this is done in a separate stage.
>I'm not sure why the access log needs to be rewritten, surely the sampler can >just read through the file until it finds the correct entry? Each sampler
> would need to know the IP address for which it is responsible.
As in the current plugin design, a log preprocessing is done:
1. to know in advance how many threadgroups will have the test plan (one for
IP)
2. to know how many requests each IP will send, and close the threadgroup when
it has finished. Rewritting the acceslog is done for efficiency, to avoid each
sampler read the whole log file, but apart of this, yes, it's possible for a
sampler to work directly with the original file.
>I don't think this should be added to JMeter trunk without further analysis.
>Maybe create an SVN branch so the feature can be tested further.
>If it looks OK it can later be merged with trunk.
I agree! Probably its better a design without preprocessing, where a master
sampler reads the log, and dynamically creates slave threadgroups on demand.
This would also solve the file hander problem you pointed earlier.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #11 from Sebb <se...@apache.org> ---
Separate Thread Groups are not essential to ensure concurrency; separate
threads are sufficient.
Also it's not necessary to know how in advance many samples there are in each
batch; a test element can cause its thread to stop when it has no more data to
process.
I think the only advance knowledge that may be needed is how many different IPs
there are, so sufficient threads can be started.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #2 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Hello,
Thanks for your contribution and sorry for very late feedback.
Few questions regarding Access Log Timer:
- Why read lines from access log into memory instead of reading them from file
?
- There is something that's not clear for me, does Access Log TImeer suppose
that access log file has been split into as many file as IPs and that only the
logs concerning the Thread Group under which Timer is located are passed to it
? because if not, it seems to me delay computation is weird ? could you clarify
?
- Could you submit the PDF into a writable format ?
Thanks
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #9 from Sebb <se...@apache.org> ---
I'm not sure why the access log needs to be rewritten, surely the sampler can
just read through the file until it finds the correct entry? Each sampler would
need to know the IP address for which it is responsible.
If the access log does need to be reformatted, it should be done as a separate
stage before starting the test proper, and the output should probably be
reformatted to make subsequent parsing easier.
I don't think this should be added to JMeter trunk without further analysis.
Maybe create an SVN branch so the feature can be tested further.
If it looks OK it can later be merged with trunk.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #4 from ejaenv <en...@yahoo.es> ---
Hi Philippe! I'll answer in a few days. Cheers
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 53748] timer for the access log sampler
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53748
--- Comment #7 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
(In reply to ejaenv from comment #6)
> Hi Philippe,
>
> Sorry me too!
>
> Thanks for your questions. I hope to clarify them:
>
> - Why read lines from access log into memory instead of reading them from
> file ?
>
> This is needed to keep at runtime the delays as similar as in the log file.
> Reading them from disc caused delay deviations due to disc major latency.
>
In this case , I think it's better to parse time during read and only store the
time instead of the whole sampler ?
Also I wonder if it would not be better to read from file to decrease memory
usage.
> - There is something that's not clear for me, does Access Log TImeer suppose
> that access log file has been split into as many file as IPs and that only
> the logs concerning the Thread Group under which Timer is located are passed
> to it ? because if not, it seems to me delay computation is weird ? could
> you clarify ?
>
> yes, the generator (1) splits the log file in the way you say, and also
> generates the jmeter plan file. Keep in mind that this computation is done
> before running jmeter, in a preparation phase.
>
> (1) the generator is implemented actually as a junit test, but the idea is
> to have a command tool.
>
> - Could you submit the PDF into a writable format ?
>
> I attach it.
>
>
> Best regards,
> /Enric
Could you join the dev mailing list, I started a discussion "AccessLogSampler &
Bug 53748" ?
--
You are receiving this mail because:
You are the assignee for the bug.