Random extraction in csv dataset

[Philippe Mouawad <ph...@gmail.com>]

Hello,
I think it would be useful to implement a mechanism to extract a line
randomly from csv,
I often meet this case and it's annoying to handle, it would be nice to
have it oob.

One problem is how to ensure uniquness if it's another requirement.

Work with a modulo per thread or something like it ?
compute at start chunks of values ?




-- 
Cordialement.
Philippe Mouawad.

Re: Random extraction in csv dataset

[Philippe Mouawad <ph...@gmail.com>]

Ok, let's close this one.

On Thursday, September 5, 2013, sebb wrote:

> On 5 September 2013 05:29, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>>
> wrote:
> > can't we open the file in random access and go to a line ?
> > Kind of RandomAccessFile + search newline character ?
> > Not sure it works
>
> Only if the file has fixed length records, which is a bit of a restriction.
> Alternatively, the user or JMeter would have to create an index file
> of fixed length records that point to the start of each record before
> starting the test run.
>
> As I wrote already, it's much better to solve this before starting the
> JMeter test.
>
> > On Wednesday, September 4, 2013, sebb wrote:
> >
> >> On 4 September 2013 09:27, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>
> <javascript:;>>
> >> wrote:
> >> > One useful case is distributed testing where you don't want to bother
> >> > splitting files for each injector.
> >> > Without this all injector will process lines in same order.
> >> > With it, no need for splitting.
> >>
> >> Separately randomised files may still result in two servers processing
> >> the same entry.
> >> Probably not at the same time, though that is not impossible either.
> >>
> >> The larger the input file, the less likely it is that the same entry
> >> will be processed more than once, but the larger the file, the larger
> >> the processing overheads.
> >>
> >> > of course I am not talking about uniqueness which is another matter.
> >> >
> >> > As always, it is a productivity feature, of course you can do it by
> hand.
> >>
> >> The problem is that randomising the lines at run-time is bound to
> >> increase the CPU and/or memory requirements.
> >>
> >> I think it would be a mistake to include this feature in standard
> JMeter.
> >>
> >> > Regards
> >> > Philippe
> >> >
> >> > On Tuesday, September 3, 2013, sebb wrote:
> >> >
> >> >> On 3 September 2013 07:45, Philippe Mouawad <
> philippe.mouawad@gmail.com <javascript:;><javascript:;>
> >> <javascript:;>>
> >> >> wrote:
> >> >> > Hello,
> >> >> > I think it would be useful to implement a mechanism to extract a
> line
> >> >> > randomly from csv,
> >> >> > I often meet this case
> >> >>
> >> >> What is the use case?
> >> >>
> >> >> > and it's annoying to handle, it would be nice to
> >> >> > have it oob.
> >> >> >
> >> >> > One problem is how to ensure uniquness if it's another requirement.
> >> >>
> >> >> That's easy. Just randomise the CSV file before starting the test
> run.
> >> >>
> >> >> That's how String From File came about.
> >> >> We had a long-running test that used millions of PINs that could only
> >> >> be used once, so we randomised them and split them into separate
> files
> >> >> which could be used once.
> >> >>
> >> >> > Work with a modulo per thread or something like it ?
> >> >> > compute at start chunks of values ?
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Cordialement.
> >> >> > Philippe Mouawad.
> >> >>
> >> >
> >> >
> >> > --
> >> > Cordialement.
> >> > Philippe Mouawad.
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>


-- 
Cordialement.
Philippe Mouawad.

Re: Random extraction in csv dataset

[sebb <se...@gmail.com>]

On 5 September 2013 05:29, Philippe Mouawad <ph...@gmail.com> wrote:
> can't we open the file in random access and go to a line ?
> Kind of RandomAccessFile + search newline character ?
> Not sure it works

Only if the file has fixed length records, which is a bit of a restriction.
Alternatively, the user or JMeter would have to create an index file
of fixed length records that point to the start of each record before
starting the test run.

As I wrote already, it's much better to solve this before starting the
JMeter test.

> On Wednesday, September 4, 2013, sebb wrote:
>
>> On 4 September 2013 09:27, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>>
>> wrote:
>> > One useful case is distributed testing where you don't want to bother
>> > splitting files for each injector.
>> > Without this all injector will process lines in same order.
>> > With it, no need for splitting.
>>
>> Separately randomised files may still result in two servers processing
>> the same entry.
>> Probably not at the same time, though that is not impossible either.
>>
>> The larger the input file, the less likely it is that the same entry
>> will be processed more than once, but the larger the file, the larger
>> the processing overheads.
>>
>> > of course I am not talking about uniqueness which is another matter.
>> >
>> > As always, it is a productivity feature, of course you can do it by hand.
>>
>> The problem is that randomising the lines at run-time is bound to
>> increase the CPU and/or memory requirements.
>>
>> I think it would be a mistake to include this feature in standard JMeter.
>>
>> > Regards
>> > Philippe
>> >
>> > On Tuesday, September 3, 2013, sebb wrote:
>> >
>> >> On 3 September 2013 07:45, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>
>> <javascript:;>>
>> >> wrote:
>> >> > Hello,
>> >> > I think it would be useful to implement a mechanism to extract a line
>> >> > randomly from csv,
>> >> > I often meet this case
>> >>
>> >> What is the use case?
>> >>
>> >> > and it's annoying to handle, it would be nice to
>> >> > have it oob.
>> >> >
>> >> > One problem is how to ensure uniquness if it's another requirement.
>> >>
>> >> That's easy. Just randomise the CSV file before starting the test run.
>> >>
>> >> That's how String From File came about.
>> >> We had a long-running test that used millions of PINs that could only
>> >> be used once, so we randomised them and split them into separate files
>> >> which could be used once.
>> >>
>> >> > Work with a modulo per thread or something like it ?
>> >> > compute at start chunks of values ?
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Cordialement.
>> >> > Philippe Mouawad.
>> >>
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.

Re: Random extraction in csv dataset

[Philippe Mouawad <ph...@gmail.com>]

can't we open the file in random access and go to a line ?
Kind of RandomAccessFile + search newline character ?
Not sure it works

On Wednesday, September 4, 2013, sebb wrote:

> On 4 September 2013 09:27, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>>
> wrote:
> > One useful case is distributed testing where you don't want to bother
> > splitting files for each injector.
> > Without this all injector will process lines in same order.
> > With it, no need for splitting.
>
> Separately randomised files may still result in two servers processing
> the same entry.
> Probably not at the same time, though that is not impossible either.
>
> The larger the input file, the less likely it is that the same entry
> will be processed more than once, but the larger the file, the larger
> the processing overheads.
>
> > of course I am not talking about uniqueness which is another matter.
> >
> > As always, it is a productivity feature, of course you can do it by hand.
>
> The problem is that randomising the lines at run-time is bound to
> increase the CPU and/or memory requirements.
>
> I think it would be a mistake to include this feature in standard JMeter.
>
> > Regards
> > Philippe
> >
> > On Tuesday, September 3, 2013, sebb wrote:
> >
> >> On 3 September 2013 07:45, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>
> <javascript:;>>
> >> wrote:
> >> > Hello,
> >> > I think it would be useful to implement a mechanism to extract a line
> >> > randomly from csv,
> >> > I often meet this case
> >>
> >> What is the use case?
> >>
> >> > and it's annoying to handle, it would be nice to
> >> > have it oob.
> >> >
> >> > One problem is how to ensure uniquness if it's another requirement.
> >>
> >> That's easy. Just randomise the CSV file before starting the test run.
> >>
> >> That's how String From File came about.
> >> We had a long-running test that used millions of PINs that could only
> >> be used once, so we randomised them and split them into separate files
> >> which could be used once.
> >>
> >> > Work with a modulo per thread or something like it ?
> >> > compute at start chunks of values ?
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Cordialement.
> >> > Philippe Mouawad.
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>


-- 
Cordialement.
Philippe Mouawad.

Re: Random extraction in csv dataset

[sebb <se...@gmail.com>]

On 4 September 2013 09:27, Philippe Mouawad <ph...@gmail.com> wrote:
> One useful case is distributed testing where you don't want to bother
> splitting files for each injector.
> Without this all injector will process lines in same order.
> With it, no need for splitting.

Separately randomised files may still result in two servers processing
the same entry.
Probably not at the same time, though that is not impossible either.

The larger the input file, the less likely it is that the same entry
will be processed more than once, but the larger the file, the larger
the processing overheads.

> of course I am not talking about uniqueness which is another matter.
>
> As always, it is a productivity feature, of course you can do it by hand.

The problem is that randomising the lines at run-time is bound to
increase the CPU and/or memory requirements.

I think it would be a mistake to include this feature in standard JMeter.

> Regards
> Philippe
>
> On Tuesday, September 3, 2013, sebb wrote:
>
>> On 3 September 2013 07:45, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>>
>> wrote:
>> > Hello,
>> > I think it would be useful to implement a mechanism to extract a line
>> > randomly from csv,
>> > I often meet this case
>>
>> What is the use case?
>>
>> > and it's annoying to handle, it would be nice to
>> > have it oob.
>> >
>> > One problem is how to ensure uniquness if it's another requirement.
>>
>> That's easy. Just randomise the CSV file before starting the test run.
>>
>> That's how String From File came about.
>> We had a long-running test that used millions of PINs that could only
>> be used once, so we randomised them and split them into separate files
>> which could be used once.
>>
>> > Work with a modulo per thread or something like it ?
>> > compute at start chunks of values ?
>> >
>> >
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.

Re: Random extraction in csv dataset

[Philippe Mouawad <ph...@gmail.com>]

One useful case is distributed testing where you don't want to bother
splitting files for each injector.
Without this all injector will process lines in same order.
With it, no need for splitting.

of course I am not talking about uniqueness which is another matter.

As always, it is a productivity feature, of course you can do it by hand.

Regards
Philippe

On Tuesday, September 3, 2013, sebb wrote:

> On 3 September 2013 07:45, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:;>>
> wrote:
> > Hello,
> > I think it would be useful to implement a mechanism to extract a line
> > randomly from csv,
> > I often meet this case
>
> What is the use case?
>
> > and it's annoying to handle, it would be nice to
> > have it oob.
> >
> > One problem is how to ensure uniquness if it's another requirement.
>
> That's easy. Just randomise the CSV file before starting the test run.
>
> That's how String From File came about.
> We had a long-running test that used millions of PINs that could only
> be used once, so we randomised them and split them into separate files
> which could be used once.
>
> > Work with a modulo per thread or something like it ?
> > compute at start chunks of values ?
> >
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>


-- 
Cordialement.
Philippe Mouawad.

Re: Random extraction in csv dataset

[sebb <se...@gmail.com>]

On 3 September 2013 07:45, Philippe Mouawad <ph...@gmail.com> wrote:
> Hello,
> I think it would be useful to implement a mechanism to extract a line
> randomly from csv,
> I often meet this case

What is the use case?

> and it's annoying to handle, it would be nice to
> have it oob.
>
> One problem is how to ensure uniquness if it's another requirement.

That's easy. Just randomise the CSV file before starting the test run.

That's how String From File came about.
We had a long-running test that used millions of PINs that could only
be used once, so we randomised them and split them into separate files
which could be used once.

> Work with a modulo per thread or something like it ?
> compute at start chunks of values ?
>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.

Re: Random extraction in csv dataset

[Philippe Mouawad <ph...@gmail.com>]

http://stackoverflow.com/questions/18479960/random-selection-from-csv-file-in-jmeter

On Tuesday, September 3, 2013, Philippe Mouawad wrote:

> Hello,
> I think it would be useful to implement a mechanism to extract a line
> randomly from csv,
> I often meet this case and it's annoying to handle, it would be nice to
> have it oob.
>
> One problem is how to ensure uniquness if it's another requirement.
>
> Work with a modulo per thread or something like it ?
> compute at start chunks of values ?
>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>

-- 
Cordialement.
Philippe Mouawad.