Re: New DataSet implementation

[Philippe Mouawad <ph...@gmail.com>]

On Sat, Nov 30, 2013 at 10:52 AM, sebb <se...@gmail.com> wrote:

> On 27 November 2013 21:21, Philippe Mouawad <ph...@gmail.com>
> wrote:
> > On Tue, Nov 26, 2013 at 1:05 AM, sebb <se...@gmail.com> wrote:
> >
> >> On 23 November 2013 13:48, Philippe Mouawad <philippe.mouawad@gmail.com
> >
> >> wrote:
> >> > Hello sebb,
> >> > What's your decision on this point ?
> >>
> >> I don't make the decisions - they are ideally arrived at by consensus;
> >> failing that, majority.
> >>
> >> > Still having objection to code commit ?
> >>
> >> Yes, because it seems wrong to include code for a specific database
> >> implementation.
> >> We should be implementing generic solutions.
> >>
> >> So would you be ok to go for Spring-data based implementation ?
> > It will impact number of embedded libraries.
>
> There are other possibilities for NoSQL as well, for example
> http://appscale.cs.ucsb.edu/datastores.html - AppScale
>
Commercial no ?

> http://thrift.apache.org/ - Apache Thrift
>

Didn't dig into it yet, but could you point me to a way to do it if you
already did ?

>
> See also
>
> http://www.oracle.com/technetwork/database/nosqldb/overview/nosql-api-497225.html
>
> http://stackoverflow.com/questions/9252034/is-there-an-emerging-standard-api-for-nosql-databases
>
> http://stackoverflow.com/questions/3439878/are-there-any-nosql-standards-emerging
>
> Given that there is no standard like JDBC available (and there may
> never be), what I think needs to happen is to decide where in JMeter
> such databases may be useful (probably not only CSV Dataset) and then
> define a suitable API for this.
> Users can then implement the API in terms of whatever database they
> wish to use (which might include a JDBC implementation).
> This is not something to be rushed into.
>

I don't totally agree with you, I prefer sometimes that we move forward
with "non perfect" thing and improve it in next steps.

Also I am a bit annoyed that time I spent on this development is kind of
thrown away while it is really easy to use and have tested it rather
thoroughly.


>
> In any case, whatever is decided on belongs in a later version of JMeter.
> We first need to get out the version with various fixes to keytool etc.
>

Dev is ready and could have been commited but would have agreed to wait.

>
> >
> >> > Thanks
> >> > Regards
> >> > Philippe
> >> >
> >> >
> >> > On Fri, Nov 15, 2013 at 10:37 PM, Philippe Mouawad <
> >> > philippe.mouawad@gmail.com> wrote:
> >> >
> >> >> We could sue Spring-Data but it would introduce a lot of
> dependencies.
> >> >>
> >> >>
> >> >> On Fri, Nov 15, 2013 at 12:54 AM, sebb <se...@gmail.com> wrote:
> >> >>
> >> >>> On 13 November 2013 21:59, Philippe Mouawad <
> >> philippe.mouawad@gmail.com>
> >> >>> wrote:
> >> >>> > Hello sebb,
> >> >>> > Shall I commit this development or do we wait for 2.10.1 to be
> >> released
> >> >>> ?
> >> >>>
> >> >>> I am still concerned that this addition is specific to the Redis
> >> server.
> >> >>> I would much prefer to see a generic solution that can use various
> >> >>> different kinds of servers.
> >> >>>
> >> >>> > When do you plan to commit your devs on keytool ?
> >> >>>
> >> >>
> >> >> If I have time I will do it , I plan the following, if you want more
> let
> >> >> me know:
> >> >> - Add keytool path property
> >> >> - Add popup to be clear about error
> >> >> - Add info about how to fix it
> >> >>
> >> >>>
> >> >>> Sorry, have not been able to spend any time on this recently.
> >> >>>
> >> >>> > Thank you
> >> >>> > Regards
> >> >>> > Philippe
> >> >>> >
> >> >>> >
> >> >>> > On Sat, Nov 2, 2013 at 3:05 PM, Philippe Mouawad <
> >> >>> philippe.mouawad@gmail.com
> >> >>> >> wrote:
> >> >>> >
> >> >>> >> Hello Sebb,
> >> >>> >> My answers below.
> >> >>> >>
> >> >>> >> On Sat, Nov 2, 2013 at 10:35 AM, sebb <se...@gmail.com> wrote:
> >> >>> >>
> >> >>> >>> On 2 November 2013 08:28, Philippe Mouawad <
> >> >>> philippe.mouawad@gmail.com>
> >> >>> >>> wrote:
> >> >>> >>> > Hello,
> >> >>> >>> > Can I proceed with commit these up coming days ?
> >> >>> >>>
> >> >>> >>> I'm not sure that the discussion was completed.
> >> >>> >>>
> >> >>> >>> As far as I can tell, the proposal only suits some types of
> >> >>> >>> cloud-based test, and relies on 3rd party servers to hold the
> data.
> >> >>> >>>
> >> >>> >>
> >> >>> >> No in my opinion if fits many scenarios:
> >> >>> >> - Cloud based tests, this one seems to me an important one, as
> Cloud
> >> >>> based
> >> >>> >> usage are increasing
> >> >>> >> - Distributed tests, even if it is possible to do it with CSV,
> >> having
> >> >>> the
> >> >>> >> data in a remote server is much easier to manage than having to
> >> >>> >> split/distribute on servers. Even it is true it requires some
> >> skills to
> >> >>> >> manage correctly the Redis server
> >> >>> >> - Continuous integration tests where also having the data in a
> >> >>> centralised
> >> >>> >> , remote servers is easier than managing CSVs. In this case Redis
> >> >>> server
> >> >>> >> plays the same role as a JDBC repository
> >> >>> >> - Compared to a database it should perform better for high load
> >> tests
> >> >>> >> since it's an in-memory repo (although this can be done in SQL
> >> >>> databases),
> >> >>> >> see http://redis.io/topics/benchmarks
> >> >>> >>
> >> >>> >>
> >> >>> >>> I'm not yet convinced that this is how we should be extending
> >> JMeter.
> >> >>> >>>
> >> >>> >>
> >> >>> >> I don't understand this argument, it would be another string for
> >> >>> JMeter,
> >> >>> >> my implementation is only 2 classes +  properties for I18N ?
> >> >>> >>
> >> >>> >> What do you propose in this case ?
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>> > Thanks
> >> >>> >>> > Regards
> >> >>> >>> > Philippe
> >> >>> >>> >
> >> >>> >>> > On Monday, October 28, 2013, Philippe Mouawad wrote:
> >> >>> >>> >
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> On Monday, October 28, 2013, sebb wrote:
> >> >>> >>> >>
> >> >>> >>> >>> On 28 October 2013 01:26, sebb <se...@gmail.com> wrote:
> >> >>> >>> >>> > On 26 October 2013 20:23, Philippe Mouawad <
> >> >>> >>> philippe.mouawad@gmail.com>
> >> >>> >>> >>> wrote:
> >> >>> >>> >>> >> Hello,
> >> >>> >>> >>> >> These days Cloud based testing is becoming popular and
> >> having
> >> >>> to
> >> >>> >>> >>> distribute
> >> >>> >>> >>> >> test data accross many servers through CSV can become
> >> painful
> >> >>> if
> >> >>> >>> not
> >> >>> >>> >>> >> impossible.
> >> >>> >>> >>> >>
> >> >>> >>> >>> >> Even without Cloud, when using distributed testing you
> >> always
> >> >>> have
> >> >>> >>> to
> >> >>> >>> >>> >> replicate your data on all servers, which is a painful
> >> manual
> >> >>> step.
> >> >>> >>> >>> >>
> >> >>> >>> >>> >> Shouldn't we introduce a new DataSet more suitable for
> these
> >> >>> use
> >> >>> >>> cases
> >> >>> >>> >>> ?
> >> >>> >>> >>> >>
> >> >>> >>> >>> >> We could do it in many different ways:
> >> >>> >>> >>> >> - Integrate an automatic CSV replicator, this would
> remain
> >> >>> simple
> >> >>> >>> and
> >> >>> >>> >>> would
> >> >>> >>> >>> >> not introduce any tier, but with cloud based it would not
> >> >>> >>> horizontally
> >> >>> >>> >>> >> scale easily
> >> >>> >>> >>> >
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>> > Not sure I follow the scaling argument. The file would
> only
> >> >>> have to
> >> >>> >>> be
> >> >>> >>> >>> > copied once before the test proper starts.
> >> >>> >>> >>> > From then on, the data is accessed locally.
> >> >>> >>> >>> >
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> The scale word was not good, In my thinking the issue is more
> >> about
> >> >>> >>> >> deploying/copying/splitting the file among nodes or cloud
> >> members.
> >> >>> A
> >> >>> >>> >> centralized access makes it far easier.
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>> > However, with a database, each node will need at least one
> >> >>> >>> connection
> >> >>> >>> >>> > to the database, and every time more data is needed there
> >> will
> >> >>> be
> >> >>> >>> >>> > network traffic.
> >> >>> >>> >>> > Or the database has to be running on the server node.
> >> >>> >>> >>> >
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> Agree, I was not saying anything different. But as I said
> this
> >> can
> >> >>> be
> >> >>> >>> >> useful for middle or low volume
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>> >> - Use a JDBC DataSet, but we would need to ensure it
> >> performs
> >> >>> >>> fine, and
> >> >>> >>> >>> >> jdbc protocol is not well suited for cloud based
> deployment
> >> >>> (But it
> >> >>> >>> >>> could
> >> >>> >>> >>> >> also be an interesting feature for Continuous
> Integration)
> >> >>> >>> >>> >> - Use a NOSQL repository  (Redis seems to me the best
> >> choice)
> >> >>> , see
> >> >>> >>> >>> this
> >> >>> >>> >>> >> high level summary which I find interesting
> >> >>> >>> >>> >>
> >> >>> >>> >>>
> >> >>> >>>
> >> >>>
> >>
> http://www.journaldunet.com/developpeur/outils/comparatif-des-bases-nosql/comparatif-des-bases-nosql-tableau-de-synthese.shtml
> >> >>> >>> >>> >>
> >> >>> >>> >>> >> I have implemented a new Redis (based on Java library for
> >> >>> Redis)
> >> >>> >>> >>> DataSet
> >> >>> >>> >>> >> which I plan to commit if no objection.
> >> >>> >>> >>> >>
> >> >>> >>> >>> >> It will introduce 2 new dependencies with Apache License:
> >> >>> >>> >>> >> - Jedis (http://code.google.com/p/jedis/)
> >> >>> >>> >>> >> - commons-pool
> >> >>> >>> >>> >
> >> >>> >>> >>> > There is also a dependency on Redis, but I guess that
> would
> >> not
> >> >>> be
> >> >>> >>> >>> bundled.
> >> >>> >>> >>>
> >> >>> >>> >>>
> >> >>> >>> >> no need to anything else than jedis + commons-pool
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>> I've just noticed that Redis is provided as source which
> needs
> >> to
> >> >>> be
> >> >>> >>> >>> built before use.
> >> >>> >>> >>> If it's difficult to copy CSV files to cloud nodes, it seems
> >> to me
> >> >>> >>> >>> it's going to be much harder to install Redis.
> >> >>> >>> >>> In which case additional network traffic will be needed to
> >> access
> >> >>> the
> >> >>> >>> >>> database.
> >> >>> >>> >>>
> >> >>> >>> >>> Also there is no official Windows release.
> >> >>> >>> >>>
> >> >>> >>> >>> >> Thoughts ?
> >> >>> >>> >>> >
> >> >>> >>> >>> > Is MongoDB not suitable?
> >> >>> >>> >>> > We already include a jar for it.
> >> >>> >>> >>> >
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>  Mongodb is more document oriented and has another type of
> use
> >> >>> cases.
> >> >>> >>> One
> >> >>> >>> >> interesting feature of redis is lists where you can pop a
> line
> >> it
> >> >>> will
> >> >>> >>> not
> >> >>> >>> >> be available to next calls, it is suitable for tests that
> need
> >> >>> >>> uniqueness
> >> >>> >>> >> accross all nodes.
> >> >>> >>> >>
> >> >>> >>> >>> >> --
> >> >>> >>> >>> >> Regards.
> >> >>> >>> >>> >> Philippe M.
> >> >>> >>> >>> >> @philmdot <https://twitter.com/philmdot>
> >> >>> >>> >>>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >> --
> >> >>> >>> >> Cordialement.
> >> >>> >>> >> Philippe Mouawad.
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >>
> >> >>> >>> >
> >> >>> >>> > --
> >> >>> >>> > Cordialement.
> >> >>> >>> > Philippe Mouawad.
> >> >>> >>>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Cordialement.
> >> >>> >> Philippe Mouawad.
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > Cordialement.
> >> >>> > Philippe Mouawad.
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Cordialement.
> >> >> Philippe Mouawad.
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Cordialement.
> >> > Philippe Mouawad.
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
>



-- 
Cordialement.
Philippe Mouawad.

Re: New DataSet implementation

[Philippe Mouawad <ph...@gmail.com>]

Hello,
For information, as we didn't get consensus on commiting the piece of code
to JMeter core and I didn't want to throw away the time I spent
implementing it I have contributed it to third party jmeter-plugins
project, it should be in next version of the plugins.

https://github.com/undera/jmeter-plugins/pull/22

I still think it would have been nice as is in core, creating a common
gui/api for cloud based libs being a bit complex as config properties
really differ as long as purposes.


Regards
Philippe M.

On Thursday, December 19, 2013, Philippe Mouawad wrote:

>
>
>
> On Sat, Nov 30, 2013 at 10:52 AM, sebb <sebbaz@gmail.com<javascript:_e({}, 'cvml', 'sebbaz@gmail.com');>
> > wrote:
>
>> On 27 November 2013 21:21, Philippe Mouawad <philippe.mouawad@gmail.com<javascript:_e({}, 'cvml', 'philippe.mouawad@gmail.com');>>
>> wrote:
>> > On Tue, Nov 26, 2013 at 1:05 AM, sebb <sebbaz@gmail.com<javascript:_e({}, 'cvml', 'sebbaz@gmail.com');>>
>> wrote:
>> >
>> >> On 23 November 2013 13:48, Philippe Mouawad <
>> philippe.mouawad@gmail.com <javascript:_e({}, 'cvml',
>> 'philippe.mouawad@gmail.com');>>
>> >> wrote:
>> >> > Hello sebb,
>> >> > What's your decision on this point ?
>> >>
>> >> I don't make the decisions - they are ideally arrived at by consensus;
>> >> failing that, majority.
>> >>
>> >> > Still having objection to code commit ?
>> >>
>> >> Yes, because it seems wrong to include code for a specific database
>> >> implementation.
>> >> We should be implementing generic solutions.
>> >>
>> >> So would you be ok to go for Spring-data based implementation ?
>> > It will impact number of embedded libraries.
>>
>> There are other possibilities for NoSQL as well, for example
>> http://appscale.cs.ucsb.edu/datastores.html - AppScale
>>
> Commercial no ?
>
>> http://thrift.apache.org/ - Apache Thrift
>>
>
> Didn't dig into it yet, but could you point me to a way to do it if you
> already did ?
>
>>
>> See also
>>
>> http://www.oracle.com/technetwork/database/nosqldb/overview/nosql-api-497225.html
>>
>> http://stackoverflow.com/questions/9252034/is-there-an-emerging-standard-api-for-nosql-databases
>>
>> http://stackoverflow.com/questions/3439878/are-there-any-nosql-standards-emerging
>>
>> Given that there is no standard like JDBC available (and there may
>> never be), what I think needs to happen is to decide where in JMeter
>> such databases may be useful (probably not only CSV Dataset) and then
>> define a suitable API for this.
>> Users can then implement the API in terms of whatever database they
>> wish to use (which might include a JDBC implementation).
>> This is not something to be rushed into.
>>
>
> I don't totally agree with you, I prefer sometimes that we move forward
> with "non perfect" thing and improve it in next steps.
>
> Also I am a bit annoyed that time I spent on this development is kind of
> thrown away while it is really easy to use and have tested it rather
> thoroughly.
>
>
>>
>> In any case, whatever is decided on belongs in a later version of JMeter.
>> We first need to get out the version with various fixes to keytool etc.
>>
>
> Dev is ready and could have been commited but would have agreed to wait.
>
>
> >
> >> > Thanks
> >> > Regards
> >> > Philippe
> >> >
> >> >
> >> > On Fri, Nov 15, 2013 at 10:37 PM, Philippe Mouawad <
> >> > philippe.mouawad@gmail.com> wrote:
> >> >
> >> >> We could sue Spring-Data but it would introduce a lot of
> dependencies.
> >> >>
> >> >>
> >> >> On Fri, Nov 15, 2013 at 12:54 AM, sebb <se...@gmail.com> wrote:
> >> >>
> >> >>> On 13 November 2013 21:59, Philippe Mouawad <
> >> philippe.mouawad@gmail.com>
> >> >>> wrote:
> >> >>> > Hello sebb,
> >> >>> > Shall I commit this development or do we wait for 2.10.1 to be
> >> released
> >> >>> ?
> >> >>>
> >> >>> I am still concerned that this addition is specific to the Redis
> >> server.
> >> >>> I would much prefer to see a generic solution that can use various
> >> >>> different kinds of servers.
> >> >>>
> >> >>> > When do you plan to commit your devs on keytool ?
> >> >>>
> >> >>
> >> >> If I have time I will do it , I plan the following, if you want more
> let
> >> >> me know:
> >> >> - Add keytool path property
> >> >> - Add popup to be clear about error
> >> >> - Add info about how to fix it
> >> >>
> >> >>>
> >> >>> Sorry, have not been able to spend any time on this recently.
> >> >>>
> >> >>> > Thank you
> >> >>> > Regards
> >> >>> > Philippe
> >> >>> >
> >> >>> >
> >> >>> > On Sat, Nov 2, 2013 at 3:05 PM, Philippe Mouawad <
> >> >>> philippe.mouawad@gmail.com
> >> >>> >> wrote:
> >> >>> >
> >> >>> >> Hello Sebb,
> >> >>> >> My answers below.
> >> >>> >>
> >> >>> >> On Sat, Nov 2, 2013 at 10:35 AM, sebb <se...@gmail.com> wrote:
> >> >>> >>
> >> >>> >>> On 2 November 2013 08:28, Philippe Mouawad <
> >> >>> philippe.mouawad@gmail.com>
> >> >>> >>> wrote:
> >> >>> >>> > Hello,
> >> >>> >>> > Can I proceed with commit these up coming days ?
> >> >>> >>>
> >> >>> >>> I'm not sure that the discussion was completed.
> >> >>> >>>
> >> >>> >>> As far as I can tell, the proposal only suits some types of
> >> >>> >>> cloud-based test, and relies on 3rd party servers to hold the
> data.
> >> >>> >>>
> >> >>> >>
> >> >>> >> No in my opinion if fits many scenarios:
> >> >>> >> - Cloud based tests, this one seems to me an important one, as
> Cloud
> >> >>> based
> >> >>> >> usage are increasing
> >> >>> >> - Distributed tests, even if it is possible to do it with CSV,
> >> having
> >> >>> the
> >> >>> >> data in a remote server is much easier to manage than having to
> >> >>> >> split/distribute on servers. Even it is true it requires some
> >> skills to
> >> >>> >> manage correctly the Redis server
> >> >>> >> - Continuous integration tests where also having the data in a
> >> >>> centralised
> >> >>> >> , remote servers is easier than managing CSVs. In this case Redis
> >> >>> server
> >> >>> >> plays the same role as a JDBC repository
> >> >>> >> - Compared to a database it should perform better for high load
> >> tests
> >> >>> >> si
>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>

-- 
Cordialement.
Philippe Mouawad.