You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Oliver Erlewein <ol...@erlewein.net> on 2013/11/06 23:20:31 UTC
CSV file iteration in remote tests
Hi all,
I'm sure that this is a common problem for those using JMeter executions
across several machines. Can't really find any solution to this. So here
goes:
I have a plan that looks something like this:
Test Plan
|-- Thread Group
|--CSV Dataset
|--HTTP Sampler (login)
|--....
If I remotely distribute this all remotes will 1st start with line one of
the CSV file. In my case this will cause locking in the application,
thereby destroying the test. Ideally I'd like to give the CSV file a random
offset for each remote client, so that it would start iterating at various
points in the CSV. This is not quite safe but should give me enough
variance so that the chance of locking would be minimal.
But I am also open to other ideas. I do want to solve this in a standard
way in-script.
I have found a way to randomly skip CSV entires but it'd skip the same
number per remote as the randomizer does not get called per remote but per
test.
Test Plan (define variable randomCSV with a random number)
|-- Thread Group
|--Counter (1...randomCSV)
|--CSV Dataset
|--If (randomCSV < Counter)
|--HTTP Sampler (login)
|--....
Hope someone can help me. I probably can't see the forest for all the trees
atm.
Cheers & Thanks
Oliver Erlewein
Re: CSV file iteration in remote tests
Posted by Oliver Erlewein <ol...@erlewein.net>.
Hi Janusz,
The repo answers within 15-20ms for CSV files that are a few thousand lines
long. For me that impact is negligible.
Cheers Oliver
On 3 January 2014 13:43, Janusz Kowalczyk <ko...@gmail.com> wrote:
> Hi Oliver,
>
> Any solution that works for you is the good solution :)
> Yet in case of central CSV repository I'd be a bit concerned about delays
> introduced by requesting and delivering the data from the central repo.
>
> Cheers,
> Janusz
>
>
>
> On 30 December 2013 20:20, Oliver Erlewein <ol...@erlewein.net> wrote:
>
>> Hi Janusz,
>>
>> Cool solution to the problem. My other issue is with the distribution of
>> CSV files in a remote JMeter scenario. There's just no good solution. So
>> what I did I detailed here:
>> http://hellotestworld.com/2013/11/08/jmeter-csv-data-for-remote-clients/
>>
>> It's not right for every scenario but it works wonderfully for me. I like
>> that I can have a central CSV repository and then easily access it.
>>
>> Cheers Oliver
>>
>>
>> On 7 November 2013 23:07, Janusz Kowalczyk <ko...@gmail.com>wrote:
>>
>>> Hi Oliver,
>>>
>>> Have a look at this test plan
>>> https://github.com/kowalcj0/jmeter-example-test-plans/tree/master/selectNRandomLinesFromFile
>>> There's a JSR223 Sampler which takes a text file and randomly selects N
>>> number of lines from it (sometimes with duplicates) and saves it in a new
>>> file.
>>> If you modify it to simply skip the first (header) line and then it
>>> might be what you're after.
>>>
>>> Cheers,
>>> J
>>>
>>>
>>> On 7 November 2013 06:47, Philippe Mouawad <ph...@gmail.com>wrote:
>>>
>>>> you might be interested in this discussion:
>>>> -
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
>>>>
>>>>
>>>>
>>>> On Thursday, November 7, 2013, Oliver Erlewein wrote:
>>>>
>>>> > Oooh! I like that idea! That would also make it easier for me to
>>>> update
>>>> > CSVs as they don't need to be distributed to the clients anymore!
>>>> Would
>>>> > mean a bit of work but would be totally worth it.
>>>> >
>>>> >
>>>> >
>>>> > On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
>>>>
>>>> > wrote:
>>>> >
>>>> > > We approach this problem slightly differently, albeit we use an
>>>> external
>>>> > > process ..
>>>> > >
>>>> > > Users upload a CSV which we bulk import into redis. Users can then
>>>> make
>>>> > > HTTP requests to that data set via a performant API (webdis in
>>>> front of
>>>> > > redis). This allows calls like SPOP (remove a random member) or
>>>> > SRANDMEMBER
>>>> > > (retrieve random member). This takes care of the headache of trying
>>>> to
>>>> > > carve up CSVs across your load generators[1]
>>>> > >
>>>> > > A little bit more effort, but may give you some ideas.
>>>> > >
>>>> > > Regards,
>>>> > >
>>>> > > Tim Koopmans
>>>> > >
>>>> > > <https://flood.io>
>>>> > >
>>>> > > Level 27, 101 Collins Street
>>>> > > Melbourne, Vic 3000
>>>> > >
>>>> > > [1] https://flood.io/blog/9-sharing-test-data
>>>> > >
>>>> > >
>>>> > >
>>>> > > On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <
>>>> oliver@erlewein.net<javascript:;>
>>>>
>>>> > >wrote:
>>>> > >
>>>> > >> Hi Sergio,
>>>> > >>
>>>> > >> Sorry forgot to say that that isn't a good option for me. I thought
>>>> > about
>>>> > >> it but I dynamically build remote agents off a standard build
>>>> image and
>>>> > >> that makes my start procedure immensely difficult as I'd need to
>>>> split
>>>> > the
>>>> > >> files and copy them to the clients before the test. I also don't
>>>> have a
>>>> > >> standard number of clients so the splits are different. If I do
>>>> that I
>>>> > >> could also just re-randomize my files too.
>>>> > >>
>>>> > >> Cheers Oliver
>>>> > >>
>>>> > >>
>>>> > >> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it
>>>> <javascript:;>>
>>>>
>>>> > wrote:
>>>> > >>
>>>> > >> > Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>>>> > >> >
>>>> > >> > Hi all,
>>>> > >> >>
>>>> > >> >> I'm sure that this is a common problem for those using JMeter
>>>> > >> executions
>>>> > >> >> across several machines. Can't really find any solution to
>>>> this. So
>>>> > >> here
>>>> > >> >> goes:
>>>> > >> >>
>>>> > >> >> I have a plan that looks something like this:
>>>> > >> >>
>>>> > >> >> Test Plan
>>>> > >> >> |-- Thread Group
>>>> > >> >> |--CSV Dataset
>>>> > >> >> |--HTTP Sampler (login)
>>>> > >> >> |--....
>>>> > >> >>
>>>> > >> >> If I remotely distribute this all remotes will 1st start with
>>>> line
>>>> > one
>>>> > >> of
>>>> > >> >> the CSV file. In my case this will cause locking in the
>>>> application,
>>>> > >> >> thereby destroying the test. Ideally I'd like to give the CSV
>>>> file a
>>>> > >> >> random
>>>> > >> >> offset for each remote client, so that it would start iterating
>>>> at
>>>> > >> various
>>>> > >> >> points in the CSV. This is not quite safe but should give me
>>>> enough
>>>> > >> >> variance so that the chance of locking would be minimal.
>>>> > >> >>
>>>> > >> > The only way I have found to cope with this is to manually split
>>>> the
>>>> > CSV
>>>> > >> > file, and copy each of these parts to each remote system, so
>>>> that each
>>>> > >> > system uses its own set of lines.
>>>> > >> >
>>>> > >> > I'm looking forward to see if there is a better system
>>>> > >> > regards
>>>> > >> >
>>>> > >> > Sergio
>>>> > >> >
>>>> > >> > --
>>>> > >> >
>>>> > >> > Ing. Sergio Boso
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >> >
>>>> > >>
>>>> > >
>>>> > >
>>>> >
>>>>
>>>>
>>>> --
>>>> Cordialement.
>>>> Philippe Mouawad.
>>>>
>>>
>>>
>
Re: CSV file iteration in remote tests
Posted by Janusz Kowalczyk <ko...@gmail.com>.
Hi Oliver,
Any solution that works for you is the good solution :)
Yet in case of central CSV repository I'd be a bit concerned about delays
introduced by requesting and delivering the data from the central repo.
Cheers,
Janusz
On 30 December 2013 20:20, Oliver Erlewein <ol...@erlewein.net> wrote:
> Hi Janusz,
>
> Cool solution to the problem. My other issue is with the distribution of
> CSV files in a remote JMeter scenario. There's just no good solution. So
> what I did I detailed here:
> http://hellotestworld.com/2013/11/08/jmeter-csv-data-for-remote-clients/
>
> It's not right for every scenario but it works wonderfully for me. I like
> that I can have a central CSV repository and then easily access it.
>
> Cheers Oliver
>
>
> On 7 November 2013 23:07, Janusz Kowalczyk <ko...@gmail.com>wrote:
>
>> Hi Oliver,
>>
>> Have a look at this test plan
>> https://github.com/kowalcj0/jmeter-example-test-plans/tree/master/selectNRandomLinesFromFile
>> There's a JSR223 Sampler which takes a text file and randomly selects N
>> number of lines from it (sometimes with duplicates) and saves it in a new
>> file.
>> If you modify it to simply skip the first (header) line and then it might
>> be what you're after.
>>
>> Cheers,
>> J
>>
>>
>> On 7 November 2013 06:47, Philippe Mouawad <ph...@gmail.com>wrote:
>>
>>> you might be interested in this discussion:
>>> -
>>>
>>> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
>>>
>>>
>>>
>>> On Thursday, November 7, 2013, Oliver Erlewein wrote:
>>>
>>> > Oooh! I like that idea! That would also make it easier for me to update
>>> > CSVs as they don't need to be distributed to the clients anymore! Would
>>> > mean a bit of work but would be totally worth it.
>>> >
>>> >
>>> >
>>> > On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
>>>
>>> > wrote:
>>> >
>>> > > We approach this problem slightly differently, albeit we use an
>>> external
>>> > > process ..
>>> > >
>>> > > Users upload a CSV which we bulk import into redis. Users can then
>>> make
>>> > > HTTP requests to that data set via a performant API (webdis in front
>>> of
>>> > > redis). This allows calls like SPOP (remove a random member) or
>>> > SRANDMEMBER
>>> > > (retrieve random member). This takes care of the headache of trying
>>> to
>>> > > carve up CSVs across your load generators[1]
>>> > >
>>> > > A little bit more effort, but may give you some ideas.
>>> > >
>>> > > Regards,
>>> > >
>>> > > Tim Koopmans
>>> > >
>>> > > <https://flood.io>
>>> > >
>>> > > Level 27, 101 Collins Street
>>> > > Melbourne, Vic 3000
>>> > >
>>> > > [1] https://flood.io/blog/9-sharing-test-data
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net
>>> <javascript:;>
>>>
>>> > >wrote:
>>> > >
>>> > >> Hi Sergio,
>>> > >>
>>> > >> Sorry forgot to say that that isn't a good option for me. I thought
>>> > about
>>> > >> it but I dynamically build remote agents off a standard build image
>>> and
>>> > >> that makes my start procedure immensely difficult as I'd need to
>>> split
>>> > the
>>> > >> files and copy them to the clients before the test. I also don't
>>> have a
>>> > >> standard number of clients so the splits are different. If I do
>>> that I
>>> > >> could also just re-randomize my files too.
>>> > >>
>>> > >> Cheers Oliver
>>> > >>
>>> > >>
>>> > >> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it
>>> <javascript:;>>
>>>
>>> > wrote:
>>> > >>
>>> > >> > Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>>> > >> >
>>> > >> > Hi all,
>>> > >> >>
>>> > >> >> I'm sure that this is a common problem for those using JMeter
>>> > >> executions
>>> > >> >> across several machines. Can't really find any solution to this.
>>> So
>>> > >> here
>>> > >> >> goes:
>>> > >> >>
>>> > >> >> I have a plan that looks something like this:
>>> > >> >>
>>> > >> >> Test Plan
>>> > >> >> |-- Thread Group
>>> > >> >> |--CSV Dataset
>>> > >> >> |--HTTP Sampler (login)
>>> > >> >> |--....
>>> > >> >>
>>> > >> >> If I remotely distribute this all remotes will 1st start with
>>> line
>>> > one
>>> > >> of
>>> > >> >> the CSV file. In my case this will cause locking in the
>>> application,
>>> > >> >> thereby destroying the test. Ideally I'd like to give the CSV
>>> file a
>>> > >> >> random
>>> > >> >> offset for each remote client, so that it would start iterating
>>> at
>>> > >> various
>>> > >> >> points in the CSV. This is not quite safe but should give me
>>> enough
>>> > >> >> variance so that the chance of locking would be minimal.
>>> > >> >>
>>> > >> > The only way I have found to cope with this is to manually split
>>> the
>>> > CSV
>>> > >> > file, and copy each of these parts to each remote system, so that
>>> each
>>> > >> > system uses its own set of lines.
>>> > >> >
>>> > >> > I'm looking forward to see if there is a better system
>>> > >> > regards
>>> > >> >
>>> > >> > Sergio
>>> > >> >
>>> > >> > --
>>> > >> >
>>> > >> > Ing. Sergio Boso
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>>
>>> --
>>> Cordialement.
>>> Philippe Mouawad.
>>>
>>
>>
Re: CSV file iteration in remote tests
Posted by Janusz Kowalczyk <ko...@gmail.com>.
Hi Oliver,
Have a look at this test plan
https://github.com/kowalcj0/jmeter-example-test-plans/tree/master/selectNRandomLinesFromFile
There's a JSR223 Sampler which takes a text file and randomly selects N
number of lines from it (sometimes with duplicates) and saves it in a new
file.
If you modify it to simply skip the first (header) line and then it might
be what you're after.
Cheers,
J
On 7 November 2013 06:47, Philippe Mouawad <ph...@gmail.com>wrote:
> you might be interested in this discussion:
> -
>
> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
>
>
>
> On Thursday, November 7, 2013, Oliver Erlewein wrote:
>
> > Oooh! I like that idea! That would also make it easier for me to update
> > CSVs as they don't need to be distributed to the clients anymore! Would
> > mean a bit of work but would be totally worth it.
> >
> >
> >
> > On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
> > wrote:
> >
> > > We approach this problem slightly differently, albeit we use an
> external
> > > process ..
> > >
> > > Users upload a CSV which we bulk import into redis. Users can then make
> > > HTTP requests to that data set via a performant API (webdis in front of
> > > redis). This allows calls like SPOP (remove a random member) or
> > SRANDMEMBER
> > > (retrieve random member). This takes care of the headache of trying to
> > > carve up CSVs across your load generators[1]
> > >
> > > A little bit more effort, but may give you some ideas.
> > >
> > > Regards,
> > >
> > > Tim Koopmans
> > >
> > > <https://flood.io>
> > >
> > > Level 27, 101 Collins Street
> > > Melbourne, Vic 3000
> > >
> > > [1] https://flood.io/blog/9-sharing-test-data
> > >
> > >
> > >
> > > On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net
> <javascript:;>
> > >wrote:
> > >
> > >> Hi Sergio,
> > >>
> > >> Sorry forgot to say that that isn't a good option for me. I thought
> > about
> > >> it but I dynamically build remote agents off a standard build image
> and
> > >> that makes my start procedure immensely difficult as I'd need to split
> > the
> > >> files and copy them to the clients before the test. I also don't have
> a
> > >> standard number of clients so the splits are different. If I do that I
> > >> could also just re-randomize my files too.
> > >>
> > >> Cheers Oliver
> > >>
> > >>
> > >> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it
> <javascript:;>>
> > wrote:
> > >>
> > >> > Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
> > >> >
> > >> > Hi all,
> > >> >>
> > >> >> I'm sure that this is a common problem for those using JMeter
> > >> executions
> > >> >> across several machines. Can't really find any solution to this. So
> > >> here
> > >> >> goes:
> > >> >>
> > >> >> I have a plan that looks something like this:
> > >> >>
> > >> >> Test Plan
> > >> >> |-- Thread Group
> > >> >> |--CSV Dataset
> > >> >> |--HTTP Sampler (login)
> > >> >> |--....
> > >> >>
> > >> >> If I remotely distribute this all remotes will 1st start with line
> > one
> > >> of
> > >> >> the CSV file. In my case this will cause locking in the
> application,
> > >> >> thereby destroying the test. Ideally I'd like to give the CSV file
> a
> > >> >> random
> > >> >> offset for each remote client, so that it would start iterating at
> > >> various
> > >> >> points in the CSV. This is not quite safe but should give me enough
> > >> >> variance so that the chance of locking would be minimal.
> > >> >>
> > >> > The only way I have found to cope with this is to manually split the
> > CSV
> > >> > file, and copy each of these parts to each remote system, so that
> each
> > >> > system uses its own set of lines.
> > >> >
> > >> > I'm looking forward to see if there is a better system
> > >> > regards
> > >> >
> > >> > Sergio
> > >> >
> > >> > --
> > >> >
> > >> > Ing. Sergio Boso
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
--
Zapraszam na swojego foto-bloga: http://na100procentchyba.wordpress.com/
Autopodpis: Staraj się używać pola Ukryty do Wiadomości (UDW) przy
wysyłaniu wiadomości do wielu odbiorców, ograniczysz przez to
rozprzestrzenianie się spamu!
Autosignature: Try to use field BCC (blind carbon copy) when sending
message to many recepients, it will restrain spread of spam!
Re: CSV file iteration in remote tests
Posted by Oliver Erlewein <ol...@erlewein.net>.
That looks cool. I'll have a look at if I'll use it.
Did you do some load testing on this? i.e. how does Redis need to scale to
cope with large tests? How does it affect JMeter tests?...
Cheers Oliver
On 31 December 2013 03:11, Philippe Mouawad <ph...@gmail.com>wrote:
> Hello,
> As dev team didn't reach consensus on commiting the piece of code to
> JMeter core, it has been contributed to third party jmeter-plugins project,
> it should be in next version of the library.
> For those interested :
> Pull request:
> - https://github.com/undera/jmeter-plugins/pull/22<https://github.com/undera/jmeter-plugins/pull/22>
>
> - Developer Snapshots download:
> -
> http://jmeter-plugins.org/downloads/file/nightly/jmeter-plugins-extras-libs-1.1.2-2013-12-30_06-16.zip
>
>
> Regards
> Philippe M.
> @philmdot
>
> On Thursday, November 7, 2013, Oliver Lloyd wrote:
>
>> I really like the idea of JMeter having the ability to use a central repo
>> of data via something like Redis. It solves not only the problem of
>> distributing unique data over multiple remote servers but also the problem
>> of data that can only be used once. I see Sebb's point though that this
>> solution will only help where the slaves have internet access - not always
>> the case. But it is the case more and more; at least in my experience, that
>> cloud-based hardware is the way to go so it makes sense to develop
>> solutions that support this.
>>
>> I think though that such an implementation should ideally abstract a lot
>> of the hassle from the user. If it would accept a csv file and a URI
>> connection string, handle the import and then internalise the methods to
>> pop / read the data, only exposing to the user some vars like ${pop_email}
>> or ${read_email}, then that would be the perfect. That way I could enter my
>> redis URI, pass a file ref to my csv and then know that each time the var
>> ${pop_email} is resolved I would have one less email in my datastore -
>> pretty cool.
>>
>>
>> On 7 Nov 2013, at 06:47, Philippe Mouawad <ph...@gmail.com>
>> wrote:
>>
>> > you might be interested in this discussion:
>> > -
>> >
>> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
>> >
>> >
>> >
>> > On Thursday, November 7, 2013, Oliver Erlewein wrote:
>> >
>> >> Oooh! I like that idea! That would also make it easier for me to update
>> >> CSVs as they don't need to be distributed to the clients anymore! Would
>> >> mean a bit of work but would be totally worth it.
>> >>
>> >>
>> >>
>> >> On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
>> >> wrote:
>> >>
>> >>> We approach this problem slightly differently, albeit we use an
>> external
>> >>> process ..
>> >>>
>> >>> Users upload a CSV which we bulk import into redis. Users can then
>> make
>> >>> HTTP requests to that data set via a performant API (webdis in front
>> of
>> >>> redis). This allows calls like SPOP (remove a random member) or
>> >> SRANDMEMBER
>> >>> (retrieve random member). This takes care of the headache of trying to
>> >>> carve up CSVs across your load generators[1]
>> >>>
>> >>> A little bit more effort, but may give you some ideas.
>> >>>
>> >>> Regards,
>> >>>
>> >>> Tim Koopmans
>> >>>
>> >>> <https://flood.io>
>> >>>
>> >>> Level 27, 101 Collins Street
>> >>> Melbourne, Vic 3000
>> >>>
>> >>> [1] https://flood.io/blog/9-sharing-test-data
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net
>> <javascript:;>
>> >>> wrote:
>> >>>
>> >>>> Hi Sergio,
>> >>>>
>> >>>> Sorry forgot to say that that isn't a good option for me. I thought
>> >> about
>> >>>> it but I dynamically build remote agents off a standard build image
>> and
>> >>>> that makes my start procedure immensely difficult as I'd need to
>> split
>> >> the
>> >>>> files and copy them to the clients before the test. I also don't
>> have a
>> >>>> standard number of clients so the splits are different. If I do that
>> I
>> >>>> could also just re-randomize my files too.
>> >>>>
>> >>>> Cheers Oliver
>> >>>>
>> >>>>
>> >>>> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it
>> <javascript:;>>
>> >> wrote:
>> >>>>
>> >>>>> Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>> >>>>>
>> >>>>> Hi all,
>> >>>>>>
>> >>>>>> I'm sure that this is a common problem for those using JMeter
>> >>>> executions
>> >>>>>> across several machines. Can't really find any solution to this. So
>> >>>> here
>> >>>>>> goes:
>> >>>>>>
>> >>>>>> I have a plan that looks something like this:
>> >>>>>>
>> >>>>>> Test Plan
>> >>>>>> |-- Thread Group
>> >>>>>> |--CSV Dataset
>> >>>>>> |--HTTP Sampler (login)
>> >>>>>> |--....
>> >>>>>>
>> >>>>>> If I remotely distribute this all remotes will 1st start with line
>> >> one
>> >>>> of
>> >>>>>> the CSV file. In my case this will cause locking in the
>> application,
>> >>>>>> thereby destroying the test. Ideally I'd like to give the CSV file
>> a
>> >>>>>> random
>> >>>>>> offset for each remote client, so that it would start iterating at
>> >>>> various
>> >>>>>> points in the CSV. This is not quite safe but should give me enough
>> >>>>>> variance so that the chance of locking would be minimal.
>> >>>>>>
>> >>>>> The only way I have found to cope with this is to manually split the
>> >> CSV
>> >>>>> file, and copy each of these parts to each remote system, so that
>> each
>> >>>>> system uses its own set of lines.
>> >>>>>
>> >>>>> I'm looking forward to see if there is a better system
>> >>>>> regards
>> >>>>>
>> >>>>> Sergio
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>> Ing. Sergio Boso
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
Re: CSV file iteration in remote tests
Posted by Tim Koopmans <ti...@flood.io>.
Great work Philippe, thanks for contributing this plugin!
Regards,
Tim Koopmans
+61 3 9221 6309
Level 27, 101 Collins Street
Melbourne, Vic 3000
On Tue, Dec 31, 2013 at 1:11 AM, Philippe Mouawad
<ph...@gmail.com> wrote:
> Hello,
> As dev team didn't reach consensus on commiting the piece of code to JMeter
> core, it has been contributed to third party jmeter-plugins project, it
> should be in next version of the library.
> For those interested :
> Pull request:
> - https://github.com/undera/jmeter-plugins/pull/22<https://github.com/undera/jmeter-plugins/pull/22>
>
> - Developer Snapshots download:
> -
> http://jmeter-plugins.org/downloads/file/nightly/jmeter-plugins-extras-libs-1.1.2-2013-12-30_06-16.zip
>
>
> Regards
> Philippe M.
> @philmdot
>
> On Thursday, November 7, 2013, Oliver Lloyd wrote:
>
>> I really like the idea of JMeter having the ability to use a central repo
>> of data via something like Redis. It solves not only the problem of
>> distributing unique data over multiple remote servers but also the problem
>> of data that can only be used once. I see Sebb's point though that this
>> solution will only help where the slaves have internet access - not always
>> the case. But it is the case more and more; at least in my experience, that
>> cloud-based hardware is the way to go so it makes sense to develop
>> solutions that support this.
>>
>> I think though that such an implementation should ideally abstract a lot
>> of the hassle from the user. If it would accept a csv file and a URI
>> connection string, handle the import and then internalise the methods to
>> pop / read the data, only exposing to the user some vars like ${pop_email}
>> or ${read_email}, then that would be the perfect. That way I could enter my
>> redis URI, pass a file ref to my csv and then know that each time the var
>> ${pop_email} is resolved I would have one less email in my datastore -
>> pretty cool.
>>
>>
>> On 7 Nov 2013, at 06:47, Philippe Mouawad <ph...@gmail.com>
>> wrote:
>>
>> > you might be interested in this discussion:
>> > -
>> >
>> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
>> >
>> >
>> >
>> > On Thursday, November 7, 2013, Oliver Erlewein wrote:
>> >
>> >> Oooh! I like that idea! That would also make it easier for me to update
>> >> CSVs as they don't need to be distributed to the clients anymore! Would
>> >> mean a bit of work but would be totally worth it.
>> >>
>> >>
>> >>
>> >> On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
>> >> wrote:
>> >>
>> >>> We approach this problem slightly differently, albeit we use an
>> external
>> >>> process ..
>> >>>
>> >>> Users upload a CSV which we bulk import into redis. Users can then make
>> >>> HTTP requests to that data set via a performant API (webdis in front of
>> >>> redis). This allows calls like SPOP (remove a random member) or
>> >> SRANDMEMBER
>> >>> (retrieve random member). This takes care of the headache of trying to
>> >>> carve up CSVs across your load generators[1]
>> >>>
>> >>> A little bit more effort, but may give you some ideas.
>> >>>
>> >>> Regards,
>> >>>
>> >>> Tim Koopmans
>> >>>
>> >>> <https://flood.io>
>> >>>
>> >>> Level 27, 101 Collins Street
>> >>> Melbourne, Vic 3000
>> >>>
>> >>> [1] https://flood.io/blog/9-sharing-test-data
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net
>> <javascript:;>
>> >>> wrote:
>> >>>
>> >>>> Hi Sergio,
>> >>>>
>> >>>> Sorry forgot to say that that isn't a good option for me. I thought
>> >> about
>> >>>> it but I dynamically build remote agents off a standard build image
>> and
>> >>>> that makes my start procedure immensely difficult as I'd need to split
>> >> the
>> >>>> files and copy them to the clients before the test. I also don't have
>> a
>> >>>> standard number of clients so the splits are different. If I do that I
>> >>>> could also just re-randomize my files too.
>> >>>>
>> >>>> Cheers Oliver
>> >>>>
>> >>>>
>> >>>> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it
>> <javascript:;>>
>> >> wrote:
>> >>>>
>> >>>>> Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>> >>>>>
>> >>>>> Hi all,
>> >>>>>>
>> >>>>>> I'm sure that this is a common problem for those using JMeter
>> >>>> executions
>> >>>>>> across several machines. Can't really find any solution to this. So
>> >>>> here
>> >>>>>> goes:
>> >>>>>>
>> >>>>>> I have a plan that looks something like this:
>> >>>>>>
>> >>>>>> Test Plan
>> >>>>>> |-- Thread Group
>> >>>>>> |--CSV Dataset
>> >>>>>> |--HTTP Sampler (login)
>> >>>>>> |--....
>> >>>>>>
>> >>>>>> If I remotely distribute this all remotes will 1st start with line
>> >> one
>> >>>> of
>> >>>>>> the CSV file. In my case this will cause locking in the application,
>> >>>>>> thereby destroying the test. Ideally I'd like to give the CSV file a
>> >>>>>> random
>> >>>>>> offset for each remote client, so that it would start iterating at
>> >>>> various
>> >>>>>> points in the CSV. This is not quite safe but should give me enough
>> >>>>>> variance so that the chance of locking would be minimal.
>> >>>>>>
>> >>>>> The only way I have found to cope with this is to manually split the
>> >> CSV
>> >>>>> file, and copy each of these parts to each remote system, so that
>> each
>> >>>>> system uses its own set of lines.
>> >>>>>
>> >>>>> I'm looking forward to see if there is a better system
>> >>>>> regards
>> >>>>>
>> >>>>> Sergio
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>> Ing. Sergio Boso
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
CSV file iteration in remote tests
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
As dev team didn't reach consensus on commiting the piece of code to JMeter
core, it has been contributed to third party jmeter-plugins project, it
should be in next version of the library.
For those interested :
Pull request:
- https://github.com/undera/jmeter-plugins/pull/22<https://github.com/undera/jmeter-plugins/pull/22>
- Developer Snapshots download:
-
http://jmeter-plugins.org/downloads/file/nightly/jmeter-plugins-extras-libs-1.1.2-2013-12-30_06-16.zip
Regards
Philippe M.
@philmdot
On Thursday, November 7, 2013, Oliver Lloyd wrote:
> I really like the idea of JMeter having the ability to use a central repo
> of data via something like Redis. It solves not only the problem of
> distributing unique data over multiple remote servers but also the problem
> of data that can only be used once. I see Sebb's point though that this
> solution will only help where the slaves have internet access - not always
> the case. But it is the case more and more; at least in my experience, that
> cloud-based hardware is the way to go so it makes sense to develop
> solutions that support this.
>
> I think though that such an implementation should ideally abstract a lot
> of the hassle from the user. If it would accept a csv file and a URI
> connection string, handle the import and then internalise the methods to
> pop / read the data, only exposing to the user some vars like ${pop_email}
> or ${read_email}, then that would be the perfect. That way I could enter my
> redis URI, pass a file ref to my csv and then know that each time the var
> ${pop_email} is resolved I would have one less email in my datastore -
> pretty cool.
>
>
> On 7 Nov 2013, at 06:47, Philippe Mouawad <ph...@gmail.com>
> wrote:
>
> > you might be interested in this discussion:
> > -
> >
> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
> >
> >
> >
> > On Thursday, November 7, 2013, Oliver Erlewein wrote:
> >
> >> Oooh! I like that idea! That would also make it easier for me to update
> >> CSVs as they don't need to be distributed to the clients anymore! Would
> >> mean a bit of work but would be totally worth it.
> >>
> >>
> >>
> >> On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
> >> wrote:
> >>
> >>> We approach this problem slightly differently, albeit we use an
> external
> >>> process ..
> >>>
> >>> Users upload a CSV which we bulk import into redis. Users can then make
> >>> HTTP requests to that data set via a performant API (webdis in front of
> >>> redis). This allows calls like SPOP (remove a random member) or
> >> SRANDMEMBER
> >>> (retrieve random member). This takes care of the headache of trying to
> >>> carve up CSVs across your load generators[1]
> >>>
> >>> A little bit more effort, but may give you some ideas.
> >>>
> >>> Regards,
> >>>
> >>> Tim Koopmans
> >>>
> >>> <https://flood.io>
> >>>
> >>> Level 27, 101 Collins Street
> >>> Melbourne, Vic 3000
> >>>
> >>> [1] https://flood.io/blog/9-sharing-test-data
> >>>
> >>>
> >>>
> >>> On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net
> <javascript:;>
> >>> wrote:
> >>>
> >>>> Hi Sergio,
> >>>>
> >>>> Sorry forgot to say that that isn't a good option for me. I thought
> >> about
> >>>> it but I dynamically build remote agents off a standard build image
> and
> >>>> that makes my start procedure immensely difficult as I'd need to split
> >> the
> >>>> files and copy them to the clients before the test. I also don't have
> a
> >>>> standard number of clients so the splits are different. If I do that I
> >>>> could also just re-randomize my files too.
> >>>>
> >>>> Cheers Oliver
> >>>>
> >>>>
> >>>> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it
> <javascript:;>>
> >> wrote:
> >>>>
> >>>>> Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
> >>>>>
> >>>>> Hi all,
> >>>>>>
> >>>>>> I'm sure that this is a common problem for those using JMeter
> >>>> executions
> >>>>>> across several machines. Can't really find any solution to this. So
> >>>> here
> >>>>>> goes:
> >>>>>>
> >>>>>> I have a plan that looks something like this:
> >>>>>>
> >>>>>> Test Plan
> >>>>>> |-- Thread Group
> >>>>>> |--CSV Dataset
> >>>>>> |--HTTP Sampler (login)
> >>>>>> |--....
> >>>>>>
> >>>>>> If I remotely distribute this all remotes will 1st start with line
> >> one
> >>>> of
> >>>>>> the CSV file. In my case this will cause locking in the application,
> >>>>>> thereby destroying the test. Ideally I'd like to give the CSV file a
> >>>>>> random
> >>>>>> offset for each remote client, so that it would start iterating at
> >>>> various
> >>>>>> points in the CSV. This is not quite safe but should give me enough
> >>>>>> variance so that the chance of locking would be minimal.
> >>>>>>
> >>>>> The only way I have found to cope with this is to manually split the
> >> CSV
> >>>>> file, and copy each of these parts to each remote system, so that
> each
> >>>>> system uses its own set of lines.
> >>>>>
> >>>>> I'm looking forward to see if there is a better system
> >>>>> regards
> >>>>>
> >>>>> Sergio
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Ing. Sergio Boso
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
Re: CSV file iteration in remote tests
Posted by Oliver Lloyd <ol...@hotmail.com>.
I really like the idea of JMeter having the ability to use a central repo of data via something like Redis. It solves not only the problem of distributing unique data over multiple remote servers but also the problem of data that can only be used once. I see Sebb's point though that this solution will only help where the slaves have internet access - not always the case. But it is the case more and more; at least in my experience, that cloud-based hardware is the way to go so it makes sense to develop solutions that support this.
I think though that such an implementation should ideally abstract a lot of the hassle from the user. If it would accept a csv file and a URI connection string, handle the import and then internalise the methods to pop / read the data, only exposing to the user some vars like ${pop_email} or ${read_email}, then that would be the perfect. That way I could enter my redis URI, pass a file ref to my csv and then know that each time the var ${pop_email} is resolved I would have one less email in my datastore - pretty cool.
On 7 Nov 2013, at 06:47, Philippe Mouawad <ph...@gmail.com> wrote:
> you might be interested in this discussion:
> -
> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
>
>
>
> On Thursday, November 7, 2013, Oliver Erlewein wrote:
>
>> Oooh! I like that idea! That would also make it easier for me to update
>> CSVs as they don't need to be distributed to the clients anymore! Would
>> mean a bit of work but would be totally worth it.
>>
>>
>>
>> On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
>> wrote:
>>
>>> We approach this problem slightly differently, albeit we use an external
>>> process ..
>>>
>>> Users upload a CSV which we bulk import into redis. Users can then make
>>> HTTP requests to that data set via a performant API (webdis in front of
>>> redis). This allows calls like SPOP (remove a random member) or
>> SRANDMEMBER
>>> (retrieve random member). This takes care of the headache of trying to
>>> carve up CSVs across your load generators[1]
>>>
>>> A little bit more effort, but may give you some ideas.
>>>
>>> Regards,
>>>
>>> Tim Koopmans
>>>
>>> <https://flood.io>
>>>
>>> Level 27, 101 Collins Street
>>> Melbourne, Vic 3000
>>>
>>> [1] https://flood.io/blog/9-sharing-test-data
>>>
>>>
>>>
>>> On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net<javascript:;>
>>> wrote:
>>>
>>>> Hi Sergio,
>>>>
>>>> Sorry forgot to say that that isn't a good option for me. I thought
>> about
>>>> it but I dynamically build remote agents off a standard build image and
>>>> that makes my start procedure immensely difficult as I'd need to split
>> the
>>>> files and copy them to the clients before the test. I also don't have a
>>>> standard number of clients so the splits are different. If I do that I
>>>> could also just re-randomize my files too.
>>>>
>>>> Cheers Oliver
>>>>
>>>>
>>>> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it<javascript:;>>
>> wrote:
>>>>
>>>>> Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>>>>>
>>>>> Hi all,
>>>>>>
>>>>>> I'm sure that this is a common problem for those using JMeter
>>>> executions
>>>>>> across several machines. Can't really find any solution to this. So
>>>> here
>>>>>> goes:
>>>>>>
>>>>>> I have a plan that looks something like this:
>>>>>>
>>>>>> Test Plan
>>>>>> |-- Thread Group
>>>>>> |--CSV Dataset
>>>>>> |--HTTP Sampler (login)
>>>>>> |--....
>>>>>>
>>>>>> If I remotely distribute this all remotes will 1st start with line
>> one
>>>> of
>>>>>> the CSV file. In my case this will cause locking in the application,
>>>>>> thereby destroying the test. Ideally I'd like to give the CSV file a
>>>>>> random
>>>>>> offset for each remote client, so that it would start iterating at
>>>> various
>>>>>> points in the CSV. This is not quite safe but should give me enough
>>>>>> variance so that the chance of locking would be minimal.
>>>>>>
>>>>> The only way I have found to cope with this is to manually split the
>> CSV
>>>>> file, and copy each of these parts to each remote system, so that each
>>>>> system uses its own set of lines.
>>>>>
>>>>> I'm looking forward to see if there is a better system
>>>>> regards
>>>>>
>>>>> Sergio
>>>>>
>>>>> --
>>>>>
>>>>> Ing. Sergio Boso
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: CSV file iteration in remote tests
Posted by Philippe Mouawad <ph...@gmail.com>.
you might be interested in this discussion:
-
http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
On Thursday, November 7, 2013, Oliver Erlewein wrote:
> Oooh! I like that idea! That would also make it easier for me to update
> CSVs as they don't need to be distributed to the clients anymore! Would
> mean a bit of work but would be totally worth it.
>
>
>
> On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
> wrote:
>
> > We approach this problem slightly differently, albeit we use an external
> > process ..
> >
> > Users upload a CSV which we bulk import into redis. Users can then make
> > HTTP requests to that data set via a performant API (webdis in front of
> > redis). This allows calls like SPOP (remove a random member) or
> SRANDMEMBER
> > (retrieve random member). This takes care of the headache of trying to
> > carve up CSVs across your load generators[1]
> >
> > A little bit more effort, but may give you some ideas.
> >
> > Regards,
> >
> > Tim Koopmans
> >
> > <https://flood.io>
> >
> > Level 27, 101 Collins Street
> > Melbourne, Vic 3000
> >
> > [1] https://flood.io/blog/9-sharing-test-data
> >
> >
> >
> > On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net<javascript:;>
> >wrote:
> >
> >> Hi Sergio,
> >>
> >> Sorry forgot to say that that isn't a good option for me. I thought
> about
> >> it but I dynamically build remote agents off a standard build image and
> >> that makes my start procedure immensely difficult as I'd need to split
> the
> >> files and copy them to the clients before the test. I also don't have a
> >> standard number of clients so the splits are different. If I do that I
> >> could also just re-randomize my files too.
> >>
> >> Cheers Oliver
> >>
> >>
> >> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it<javascript:;>>
> wrote:
> >>
> >> > Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
> >> >
> >> > Hi all,
> >> >>
> >> >> I'm sure that this is a common problem for those using JMeter
> >> executions
> >> >> across several machines. Can't really find any solution to this. So
> >> here
> >> >> goes:
> >> >>
> >> >> I have a plan that looks something like this:
> >> >>
> >> >> Test Plan
> >> >> |-- Thread Group
> >> >> |--CSV Dataset
> >> >> |--HTTP Sampler (login)
> >> >> |--....
> >> >>
> >> >> If I remotely distribute this all remotes will 1st start with line
> one
> >> of
> >> >> the CSV file. In my case this will cause locking in the application,
> >> >> thereby destroying the test. Ideally I'd like to give the CSV file a
> >> >> random
> >> >> offset for each remote client, so that it would start iterating at
> >> various
> >> >> points in the CSV. This is not quite safe but should give me enough
> >> >> variance so that the chance of locking would be minimal.
> >> >>
> >> > The only way I have found to cope with this is to manually split the
> CSV
> >> > file, and copy each of these parts to each remote system, so that each
> >> > system uses its own set of lines.
> >> >
> >> > I'm looking forward to see if there is a better system
> >> > regards
> >> >
> >> > Sergio
> >> >
> >> > --
> >> >
> >> > Ing. Sergio Boso
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>
--
Cordialement.
Philippe Mouawad.
Re: CSV file iteration in remote tests
Posted by Philippe Mouawad <ph...@gmail.com>.
You might me interested in this discussion.
On Thursday, November 7, 2013, Oliver Erlewein wrote:
> Oooh! I like that idea! That would also make it easier for me to update
> CSVs as they don't need to be distributed to the clients anymore! Would
> mean a bit of work but would be totally worth it.
>
>
>
> On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
> wrote:
>
> > We approach this problem slightly differently, albeit we use an external
> > process ..
> >
> > Users upload a CSV which we bulk import into redis. Users can then make
> > HTTP requests to that data set via a performant API (webdis in front of
> > redis). This allows calls like SPOP (remove a random member) or
> SRANDMEMBER
> > (retrieve random member). This takes care of the headache of trying to
> > carve up CSVs across your load generators[1]
> >
> > A little bit more effort, but may give you some ideas.
> >
> > Regards,
> >
> > Tim Koopmans
> >
> > <https://flood.io>
> >
> > Level 27, 101 Collins Street
> > Melbourne, Vic 3000
> >
> > [1] https://flood.io/blog/9-sharing-test-data
> >
> >
> >
> > On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net<javascript:;>
> >wrote:
> >
> >> Hi Sergio,
> >>
> >> Sorry forgot to say that that isn't a good option for me. I thought
> about
> >> it but I dynamically build remote agents off a standard build image and
> >> that makes my start procedure immensely difficult as I'd need to split
> the
> >> files and copy them to the clients before the test. I also don't have a
> >> standard number of clients so the splits are different. If I do that I
> >> could also just re-randomize my files too.
> >>
> >> Cheers Oliver
> >>
> >>
> >> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it<javascript:;>>
> wrote:
> >>
> >> > Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
> >> >
> >> > Hi all,
> >> >>
> >> >> I'm sure that this is a common problem for those using JMeter
> >> executions
> >> >> across several machines. Can't really find any solution to this. So
> >> here
> >> >> goes:
> >> >>
> >> >> I have a plan that looks something like this:
> >> >>
> >> >> Test Plan
> >> >> |-- Thread Group
> >> >> |--CSV Dataset
> >> >> |--HTTP Sampler (login)
> >> >> |--....
> >> >>
> >> >> If I remotely distribute this all remotes will 1st start with line
> one
> >> of
> >> >> the CSV file. In my case this will cause locking in the application,
> >> >> thereby destroying the test. Ideally I'd like to give the CSV file a
> >> >> random
> >> >> offset for each remote client, so that it would start iterating at
> >> various
> >> >> points in the CSV. This is not quite safe but should give me enough
> >> >> variance so that the chance of locking would be minimal.
> >> >>
> >> > The only way I have found to cope with this is to manually split the
> CSV
> >> > file, and copy each of these parts to each remote system, so that each
> >> > system uses its own set of lines.
> >> >
> >> > I'm looking forward to see if there is a better system
> >> > regards
> >> >
> >> > Sergio
> >> >
> >> > --
> >> >
> >> > Ing. Sergio Boso
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>
--
Cordialement.
Philippe Mouawad.
Re: CSV file iteration in remote tests
Posted by Oliver Erlewein <ol...@erlewein.net>.
Oooh! I like that idea! That would also make it easier for me to update
CSVs as they don't need to be distributed to the clients anymore! Would
mean a bit of work but would be totally worth it.
On 7 November 2013 11:46, Tim Koopmans <ti...@flood.io> wrote:
> We approach this problem slightly differently, albeit we use an external
> process ..
>
> Users upload a CSV which we bulk import into redis. Users can then make
> HTTP requests to that data set via a performant API (webdis in front of
> redis). This allows calls like SPOP (remove a random member) or SRANDMEMBER
> (retrieve random member). This takes care of the headache of trying to
> carve up CSVs across your load generators[1]
>
> A little bit more effort, but may give you some ideas.
>
> Regards,
>
> Tim Koopmans
>
> <https://flood.io>
>
> Level 27, 101 Collins Street
> Melbourne, Vic 3000
>
> [1] https://flood.io/blog/9-sharing-test-data
>
>
>
> On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <ol...@erlewein.net>wrote:
>
>> Hi Sergio,
>>
>> Sorry forgot to say that that isn't a good option for me. I thought about
>> it but I dynamically build remote agents off a standard build image and
>> that makes my start procedure immensely difficult as I'd need to split the
>> files and copy them to the clients before the test. I also don't have a
>> standard number of clients so the splits are different. If I do that I
>> could also just re-randomize my files too.
>>
>> Cheers Oliver
>>
>>
>> On 7 November 2013 11:28, Sergio Boso <se...@bosoconsulting.it> wrote:
>>
>> > Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>> >
>> > Hi all,
>> >>
>> >> I'm sure that this is a common problem for those using JMeter
>> executions
>> >> across several machines. Can't really find any solution to this. So
>> here
>> >> goes:
>> >>
>> >> I have a plan that looks something like this:
>> >>
>> >> Test Plan
>> >> |-- Thread Group
>> >> |--CSV Dataset
>> >> |--HTTP Sampler (login)
>> >> |--....
>> >>
>> >> If I remotely distribute this all remotes will 1st start with line one
>> of
>> >> the CSV file. In my case this will cause locking in the application,
>> >> thereby destroying the test. Ideally I'd like to give the CSV file a
>> >> random
>> >> offset for each remote client, so that it would start iterating at
>> various
>> >> points in the CSV. This is not quite safe but should give me enough
>> >> variance so that the chance of locking would be minimal.
>> >>
>> > The only way I have found to cope with this is to manually split the CSV
>> > file, and copy each of these parts to each remote system, so that each
>> > system uses its own set of lines.
>> >
>> > I'm looking forward to see if there is a better system
>> > regards
>> >
>> > Sergio
>> >
>> > --
>> >
>> > Ing. Sergio Boso
>> >
>> >
>> >
>> >
>>
>
>
Re: CSV file iteration in remote tests
Posted by Tim Koopmans <ti...@flood.io>.
We approach this problem slightly differently, albeit we use an external
process ..
Users upload a CSV which we bulk import into redis. Users can then make
HTTP requests to that data set via a performant API (webdis in front of
redis). This allows calls like SPOP (remove a random member) or SRANDMEMBER
(retrieve random member). This takes care of the headache of trying to
carve up CSVs across your load generators[1]
A little bit more effort, but may give you some ideas.
Regards,
Tim Koopmans
<https://flood.io>
Level 27, 101 Collins Street
Melbourne, Vic 3000
[1] https://flood.io/blog/9-sharing-test-data
On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <ol...@erlewein.net> wrote:
> Hi Sergio,
>
> Sorry forgot to say that that isn't a good option for me. I thought about
> it but I dynamically build remote agents off a standard build image and
> that makes my start procedure immensely difficult as I'd need to split the
> files and copy them to the clients before the test. I also don't have a
> standard number of clients so the splits are different. If I do that I
> could also just re-randomize my files too.
>
> Cheers Oliver
>
>
> On 7 November 2013 11:28, Sergio Boso <se...@bosoconsulting.it> wrote:
>
> > Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
> >
> > Hi all,
> >>
> >> I'm sure that this is a common problem for those using JMeter executions
> >> across several machines. Can't really find any solution to this. So here
> >> goes:
> >>
> >> I have a plan that looks something like this:
> >>
> >> Test Plan
> >> |-- Thread Group
> >> |--CSV Dataset
> >> |--HTTP Sampler (login)
> >> |--....
> >>
> >> If I remotely distribute this all remotes will 1st start with line one
> of
> >> the CSV file. In my case this will cause locking in the application,
> >> thereby destroying the test. Ideally I'd like to give the CSV file a
> >> random
> >> offset for each remote client, so that it would start iterating at
> various
> >> points in the CSV. This is not quite safe but should give me enough
> >> variance so that the chance of locking would be minimal.
> >>
> > The only way I have found to cope with this is to manually split the CSV
> > file, and copy each of these parts to each remote system, so that each
> > system uses its own set of lines.
> >
> > I'm looking forward to see if there is a better system
> > regards
> >
> > Sergio
> >
> > --
> >
> > Ing. Sergio Boso
> >
> >
> >
> >
>
Re: CSV file iteration in remote tests
Posted by Oliver Erlewein <ol...@erlewein.net>.
Hi Sergio,
Sorry forgot to say that that isn't a good option for me. I thought about
it but I dynamically build remote agents off a standard build image and
that makes my start procedure immensely difficult as I'd need to split the
files and copy them to the clients before the test. I also don't have a
standard number of clients so the splits are different. If I do that I
could also just re-randomize my files too.
Cheers Oliver
On 7 November 2013 11:28, Sergio Boso <se...@bosoconsulting.it> wrote:
> Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>
> Hi all,
>>
>> I'm sure that this is a common problem for those using JMeter executions
>> across several machines. Can't really find any solution to this. So here
>> goes:
>>
>> I have a plan that looks something like this:
>>
>> Test Plan
>> |-- Thread Group
>> |--CSV Dataset
>> |--HTTP Sampler (login)
>> |--....
>>
>> If I remotely distribute this all remotes will 1st start with line one of
>> the CSV file. In my case this will cause locking in the application,
>> thereby destroying the test. Ideally I'd like to give the CSV file a
>> random
>> offset for each remote client, so that it would start iterating at various
>> points in the CSV. This is not quite safe but should give me enough
>> variance so that the chance of locking would be minimal.
>>
> The only way I have found to cope with this is to manually split the CSV
> file, and copy each of these parts to each remote system, so that each
> system uses its own set of lines.
>
> I'm looking forward to see if there is a better system
> regards
>
> Sergio
>
> --
>
> Ing. Sergio Boso
>
>
>
>
Re: CSV file iteration in remote tests
Posted by Sergio Boso <se...@bosoconsulting.it>.
Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
> Hi all,
>
> I'm sure that this is a common problem for those using JMeter executions
> across several machines. Can't really find any solution to this. So here
> goes:
>
> I have a plan that looks something like this:
>
> Test Plan
> |-- Thread Group
> |--CSV Dataset
> |--HTTP Sampler (login)
> |--....
>
> If I remotely distribute this all remotes will 1st start with line one of
> the CSV file. In my case this will cause locking in the application,
> thereby destroying the test. Ideally I'd like to give the CSV file a random
> offset for each remote client, so that it would start iterating at various
> points in the CSV. This is not quite safe but should give me enough
> variance so that the chance of locking would be minimal.
The only way I have found to cope with this is to manually split the CSV file, and copy each of these parts to each remote system,
so that each system uses its own set of lines.
I'm looking forward to see if there is a better system
regards
Sergio
--
Ing. Sergio Boso
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org