You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dmitry Kan <so...@gmail.com> on 2013/09/02 13:46:29 UTC
Re: Measuring SOLR performance
Hi Roman,
What's the format for running the facet+filter queries?
Would something like this work:
field:foo >=50 fq=other_field:bar facet=true facet.field=facet_field_name
Thanks,
Dmitry
On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <so...@gmail.com> wrote:
> Hi Roman,
>
> With adminPath="/admin" or adminPath="/admin/cores", no. Interestingly
> enough, though, I can access
> http://localhost:8983/solr/statements/admin/system
>
> But I can access http://localhost:8983/solr/admin/cores, only when with
> adminPath="/admin/cores" (which suggests that this is the right value to be
> used for cores), and not with adminPath="/admin".
>
> Bottom line, these core configuration is not self-evident.
>
> Dmitry
>
>
>
>
> On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <ro...@gmail.com>wrote:
>
>> Hi Dmitry,
>> So it seems solrjmeter should not assume the adminPath - and perhaps needs
>> to be passed as an argument. When you set the adminPath, are you able to
>> access localhost:8983/solr/statements/admin/cores ?
>>
>> roman
>>
>>
>> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <so...@gmail.com> wrote:
>>
>> > Hi Roman,
>> >
>> > I have noticed a difference with different solr.xml config contents. It
>> is
>> > probably legit, but thought to let you know (tests run on fresh
>> checkout as
>> > of today).
>> >
>> > As mentioned before, I have two cores configured in solr.xml. If the
>> file
>> > is:
>> >
>> > [code]
>> > <solr persistent="false">
>> >
>> > <!--
>> > adminPath: RequestHandler path to manage cores.
>> > If 'null' (or absent), cores will not be manageable via request
>> handler
>> > -->
>> > <cores adminPath="/admin/cores" host="${host:}"
>> > hostPort="${jetty.port:8983}" hostContext="${hostContext:solr}">
>> > <core name="metadata" instanceDir="metadata" />
>> > <core name="statements" instanceDir="statements" />
>> > </cores>
>> > </solr>
>> > [/code]
>> >
>> > then the instruction:
>> >
>> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
>> > ./queries/demo/demo.queries -s localhost -p 8983 -a --durationInSecs 60
>> -R
>> > cms -t /solr/statements -e statements -U 100
>> >
>> > works just fine. If however the solr.xml has adminPath set to "/admin"
>> > solrjmeter produces an error:
>> >
>> > [error]
>> > **ERROR**
>> > File "solrjmeter.py", line 1386, in <module>
>> > main(sys.argv)
>> > File "solrjmeter.py", line 1278, in main
>> > check_prerequisities(options)
>> > File "solrjmeter.py", line 375, in check_prerequisities
>> > error('Cannot find admin pages: %s, please report a bug' % apath)
>> > File "solrjmeter.py", line 66, in error
>> > traceback.print_stack()
>> > Cannot find admin pages: http://localhost:8983/solr/admin, please
>> report a
>> > bug
>> > [/error]
>> >
>> > With both solr.xml configs the following url returns just fine:
>> >
>> > http://localhost:8983/solr/statements/admin/system?wt=json
>> >
>> > Regards,
>> >
>> > Dmitry
>> >
>> >
>> >
>> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <so...@gmail.com>
>> wrote:
>> >
>> > > Hi Roman,
>> > >
>> > > This looks much better, thanks! The ordinary non-comarison mode works.
>> > > I'll post here, if there are other findings.
>> > >
>> > > Thanks for quick turnarounds,
>> > >
>> > > Dmitry
>> > >
>> > >
>> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <roman.chyla@gmail.com
>> > >wrote:
>> > >
>> > >> Hi Dmitry, oh yes, late night fixes... :) The latest commit should
>> make
>> > it
>> > >> work for you.
>> > >> Thanks!
>> > >>
>> > >> roman
>> > >>
>> > >>
>> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <so...@gmail.com>
>> > wrote:
>> > >>
>> > >> > Hi Roman,
>> > >> >
>> > >> > Something bad happened in fresh checkout:
>> > >> >
>> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
>> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
>> --durationInSecs
>> > 60
>> > >> -R
>> > >> > cms -t /solr/statements -e statements -U 100
>> > >> >
>> > >> > Traceback (most recent call last):
>> > >> > File "solrjmeter.py", line 1392, in <module>
>> > >> > main(sys.argv)
>> > >> > File "solrjmeter.py", line 1347, in main
>> > >> > save_into_file('before-test.json',
>> simplejson.dumps(before_test))
>> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
>> line
>> > >> 286,
>> > >> > in dumps
>> > >> > return _default_encoder.encode(obj)
>> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
>> line
>> > >> 226,
>> > >> > in encode
>> > >> > chunks = self.iterencode(o, _one_shot=True)
>> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
>> line
>> > >> 296,
>> > >> > in iterencode
>> > >> > return _iterencode(o, 0)
>> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
>> line
>> > >> 202,
>> > >> > in default
>> > >> > raise TypeError(repr(o) + " is not JSON serializable")
>> > >> > TypeError: <__main__.ForgivingValue object at 0x7fc6d4040fd0> is
>> not
>> > >> JSON
>> > >> > serializable
>> > >> >
>> > >> >
>> > >> > Regards,
>> > >> >
>> > >> > D.
>> > >> >
>> > >> >
>> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
>> roman.chyla@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Dmitry,
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
>> solrexpert@gmail.com>
>> > >> > wrote:
>> > >> > >
>> > >> > > > Hi Roman,
>> > >> > > >
>> > >> > > > Good point. I managed to run the command with -C and double
>> > quotes:
>> > >> > > >
>> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour -x
>> > >> ./jmx/SolrQueryTest.jmx
>> > >> > > >
>> > >> > > > As a result got several files (html, css, js, csv) in the
>> running
>> > >> > > directory
>> > >> > > > (any way to specify where the output should be stored in this
>> > case?)
>> > >> > > >
>> > >> > >
>> > >> > > i know it is confusing, i plan to change it - but later, now it
>> is
>> > too
>> > >> > busy
>> > >> > > here...
>> > >> > >
>> > >> > >
>> > >> > > >
>> > >> > > > When I look onto the comparison dashboard, I see this:
>> > >> > > >
>> > >> > > > http://pbrd.co/17IRI0b
>> > >> > > >
>> > >> > >
>> > >> > > two things: the tests probably took more than one hour to
>> finish, so
>> > >> they
>> > >> > > are not aligned - try generating the comparison with '-c 14400'
>> > (ie.
>> > >> > > 4x3600 secs)
>> > >> > >
>> > >> > > the other thing: if you have only two datapoints, the dygraph
>> will
>> > not
>> > >> > show
>> > >> > > anything - there must be more datapoints/measurements
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > >
>> > >> > > > One more thing: all the previous tests were run with softCommit
>> > >> > disabled.
>> > >> > > > After enabling it, the tests started to fail:
>> > >> > > >
>> > >> > > > $ python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
>> > >> > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
>> > >> --durationInSecs 60
>> > >> > > -R
>> > >> > > > g1 -t /solr/statements -e statements -U 100
>> > >> > > > $ cd g1
>> > >> > > > Reading results of the previous test
>> > >> > > > $ cd 2013.08.12.16.32.48
>> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
>> > >> > > > $ mkdir 2013.08.12.16.33.02
>> > >> > > > $ cd 2013.08.12.16.33.02
>> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
>> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
>> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
>> > >> > > > Traceback (most recent call last):
>> > >> > > > File "solrjmeter.py", line 1427, in <module>
>> > >> > > > main(sys.argv)
>> > >> > > > File "solrjmeter.py", line 1381, in main
>> > >> > > > before_test = harvest_details_about_montysolr(options)
>> > >> > > > File "solrjmeter.py", line 562, in
>> > harvest_details_about_montysolr
>> > >> > > > indexLstModified =
>> > >> > cores_data['status'][cn]['index']['lastModified'],
>> > >> > > > KeyError: 'lastModified'
>> > >> > > >
>> > >> > >
>> > >> > > Thanks for letting me know, that info is probably not available
>> in
>> > >> this
>> > >> > > situation - i've cooked st quick to fix it, please try the latest
>> > >> commit
>> > >> > > (hope it doesn't do more harm, i should get some sleep ..;))
>> > >> > >
>> > >> > > roman
>> > >> > >
>> > >> > >
>> > >> > > >
>> > >> > > > In case it matters: Python 2.7.3, ubuntu, solr 4.3.1.
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > >
>> > >> > > > Dmitry
>> > >> > > >
>> > >> > > >
>> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla <
>> > roman.chyla@gmail.com>
>> > >> > > wrote:
>> > >> > > >
>> > >> > > > > Hi Dmitry,
>> > >> > > > > The command seems good. Are you sure your shell is not doing
>> > >> > something
>> > >> > > > > funny with the params? You could try:
>> > >> > > > >
>> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
>> > >> ./jmx/SolrQueryTest.jmx
>> > >> > -a
>> > >> > > > >
>> > >> > > > > where g1 and foo are results of the individual runs, ie.
>> > something
>> > >> > that
>> > >> > > > was
>> > >> > > > > started and saved with '-R g1' and '-R foo' respectively
>> > >> > > > >
>> > >> > > > > so, for example, i have these comparisons inside
>> > >> > > > > '/var/lib/montysolr/different-java-settings/solrjmeter', so
>> I am
>> > >> > > > generating
>> > >> > > > > the comparison by:
>> > >> > > > >
>> > >> > > > > export
>> > >> > > > >
>> > >> SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
>> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
>> > >> ./jmx/SolrQueryTest.jmx
>> > >> > -a
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > roman
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry Kan <
>> > solrexpert@gmail.com
>> > >> >
>> > >> > > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi Roman,
>> > >> > > > > >
>> > >> > > > > > One more question. I tried to compare different runs (g1 vs
>> > cms)
>> > >> > > using
>> > >> > > > > the
>> > >> > > > > > command below, but get an error. Should I attach some other
>> > >> > param(s)?
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > python solrjmeter.py -C g1,foo -c hour -x
>> > >> ./jmx/SolrQueryTest.jmx
>> > >> > > > > > **ERROR**
>> > >> > > > > > File "solrjmeter.py", line 1427, in <module>
>> > >> > > > > > main(sys.argv)
>> > >> > > > > > File "solrjmeter.py", line 1303, in main
>> > >> > > > > > check_options(options, args)
>> > >> > > > > > File "solrjmeter.py", line 185, in check_options
>> > >> > > > > > error("The folder '%s' does not exist" % rf)
>> > >> > > > > > File "solrjmeter.py", line 66, in error
>> > >> > > > > > traceback.print_stack()
>> > >> > > > > > The folder '0' does not exist
>> > >> > > > > >
>> > >> > > > > > Dmitry
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM, Dmitry Kan <
>> > >> solrexpert@gmail.com>
>> > >> > > > wrote:
>> > >> > > > > >
>> > >> > > > > > > Hi Roman,
>> > >> > > > > > >
>> > >> > > > > > > Finally, this has worked! Thanks for quick support.
>> > >> > > > > > >
>> > >> > > > > > > The graphs look awesome. At least on the index sample :)
>> It
>> > is
>> > >> > > quite
>> > >> > > > > easy
>> > >> > > > > > > to setup and run + possible to run directly on the shard
>> > >> server
>> > >> > in
>> > >> > > > > > > background mode.
>> > >> > > > > > >
>> > >> > > > > > > my test run was:
>> > >> > > > > > >
>> > >> > > > > > > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
>> > >> > > > > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
>> > >> > > --durationInSecs
>> > >> > > > 60
>> > >> > > > > > -R
>> > >> > > > > > > foo -t /solr/statements -e statements
>> > >> > > > > > >
>> > >> > > > > > > Thanks!
>> > >> > > > > > >
>> > >> > > > > > > Dmitry
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54 AM, Roman Chyla <
>> > >> > roman.chyla@gmail.com
>> > >> > > >
>> > >> > > > > > wrote:
>> > >> > > > > > >
>> > >> > > > > > >> Hi Dmitry,
>> > >> > > > > > >>
>> > >> > > > > > >> I've modified the solrjmeter to retrieve data from under
>> > the
>> > >> > core
>> > >> > > > (the
>> > >> > > > > > -t
>> > >> > > > > > >> parameter) and the rest from the /solr/admin - I could
>> test
>> > >> it
>> > >> > > only
>> > >> > > > > > >> against
>> > >> > > > > > >> 4.0, but it is there the same as 4.3 - it seems...so you
>> > can
>> > >> try
>> > >> > > the
>> > >> > > > > > fresh
>> > >> > > > > > >> checkout
>> > >> > > > > > >>
>> > >> > > > > > >> my test was: python solrjmeter.py -a -x
>> > >> ./jmx/SolrQueryTest.jmx
>> > >> > -t
>> > >> > > > > > >> /solr/collection1 -R foo -q ./queries/demo/* -p 9002 -s
>> > >> adsate
>> > >> > > > > > >>
>> > >> > > > > > >> Thanks!
>> > >> > > > > > >>
>> > >> > > > > > >> roman
>> > >> > > > > > >>
>> > >> > > > > > >>
>> > >> > > > > > >> On Tue, Aug 6, 2013 at 9:46 AM, Dmitry Kan <
>> > >> > solrexpert@gmail.com>
>> > >> > > > > > wrote:
>> > >> > > > > > >>
>> > >> > > > > > >> > Hi,
>> > >> > > > > > >> >
>> > >> > > > > > >> > Thanks for the clarification, Shawn!
>> > >> > > > > > >> >
>> > >> > > > > > >> > So with this in mind, the following work:
>> > >> > > > > > >> >
>> > >> > > > > > >> >
>> > http://localhost:8983/solr/statements/admin/system?wt=json
>> > >> > > > > > >> >
>> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
>> > >> > > > > > >> >
>> > >> > > > > > >> > not copying their output to save space.
>> > >> > > > > > >> >
>> > >> > > > > > >> > Roman:
>> > >> > > > > > >> >
>> > >> > > > > > >> > is this something that should be set via -t parameter
>> as
>> > >> well?
>> > >> > > > > > >> >
>> > >> > > > > > >> > Dmitry
>> > >> > > > > > >> >
>> > >> > > > > > >> >
>> > >> > > > > > >> >
>> > >> > > > > > >> > On Tue, Aug 6, 2013 at 4:34 PM, Shawn Heisey <
>> > >> > solr@elyograg.org
>> > >> > > >
>> > >> > > > > > wrote:
>> > >> > > > > > >> >
>> > >> > > > > > >> > > On 8/6/2013 6:17 AM, Dmitry Kan wrote:
>> > >> > > > > > >> > > > Of three URLs you asked for, only the 3rd one gave
>> > >> > response:
>> > >> > > > > > >> > > <snip>
>> > >> > > > > > >> > > > The rest report 404.
>> > >> > > > > > >> > > >
>> > >> > > > > > >> > > > On Mon, Aug 5, 2013 at 8:38 PM, Roman Chyla <
>> > >> > > > > > roman.chyla@gmail.com>
>> > >> > > > > > >> > > wrote:
>> > >> > > > > > >> > > >
>> > >> > > > > > >> > > >> Hi Dmitry,
>> > >> > > > > > >> > > >> So I think the admin pages are different on your
>> > >> version
>> > >> > of
>> > >> > > > > solr,
>> > >> > > > > > >> what
>> > >> > > > > > >> > > do
>> > >> > > > > > >> > > >> you see when you request... ?
>> > >> > > > > > >> > > >>
>> > >> > > > > > >> > > >> http://localhost:8983/solr/admin/system?wt=json
>> > >> > > > > > >> > > >> http://localhost:8983/solr/admin/mbeans?wt=json
>> > >> > > > > > >> > > >> http://localhost:8983/solr/admin/cores?wt=json
>> > >> > > > > > >> > >
>> > >> > > > > > >> > > Unless you have a valid defaultCoreName set in your
>> > >> > > (old-style)
>> > >> > > > > > >> > > solr.xml, the first two URLs won't work, as you've
>> > >> > discovered.
>> > >> > > > > > >> Without
>> > >> > > > > > >> > > that valid defaultCoreName (or if you wanted info
>> from
>> > a
>> > >> > > > different
>> > >> > > > > > >> > > core), you'd need to add a core name to the URL for
>> > them
>> > >> to
>> > >> > > > work.
>> > >> > > > > > >> > >
>> > >> > > > > > >> > > The third one, which works for you, is a global
>> handler
>> > >> for
>> > >> > > > > > >> manipulating
>> > >> > > > > > >> > > cores, so naturally it doesn't need a core name to
>> > >> function.
>> > >> > > > The
>> > >> > > > > > URL
>> > >> > > > > > >> > > path for this handler is defined by solr.xml.
>> > >> > > > > > >> > >
>> > >> > > > > > >> > > Thanks,
>> > >> > > > > > >> > > Shawn
>> > >> > > > > > >> > >
>> > >> > > > > > >> > >
>> > >> > > > > > >> >
>> > >> > > > > > >>
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: Measuring SOLR performance
Posted by Dmitry Kan <so...@gmail.com>.
Hi Roman,
Ok, I will. Thanks!
Cheers,
Dmitry
On Tue, Sep 3, 2013 at 4:46 PM, Roman Chyla <ro...@gmail.com> wrote:
> Hi Dmitry,
>
> Thanks for the feedback. Yes, it is indeed jmeter issue (or rather, the
> issue of the plugin we use to generate charts). You may want to use the
> github for whatever comes next
>
> https://github.com/romanchyla/solrjmeter/issues
>
> Cheers,
>
> roman
>
>
> On Tue, Sep 3, 2013 at 7:54 AM, Dmitry Kan <so...@gmail.com> wrote:
>
> > Hi Roman,
> >
> > Thanks, the --additionalSolrParams was just what I wanted and works fine.
> >
> > BTW, if you have some special "bug tracking forum" for the tool, I'm
> happy
> > to submit questions / bug reports there. Otherwise, this email list is ok
> > (for me at least).
> >
> > One other thing I have noticed in the err logs was a series of messages
> of
> > this sort upon generating the perf test report. Seems to be jmeter
> related
> > (the err messages disappear, if extra lib dir is present under ext
> > directory).
> >
> > java.lang.Throwable: Could not access
> > /home/dmitry/projects/lab/solrjmeter7/solrjmeter/jmeter/lib/ext/lib
> > at
> >
> kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
> > at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
> > at
> >
> kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
> > at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
> >
> > at
> >
> kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
> > at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
> >
> >
> >
> > On Tue, Sep 3, 2013 at 2:50 AM, Roman Chyla <ro...@gmail.com>
> wrote:
> >
> > > Hi Dmitry,
> > >
> > > If it is something you want to pass with every request (which is my use
> > > case), you can pass it as additional solr params, eg.
> > >
> > > python solrjmeter
> > >
> > >
> >
> --additionalSolrParams="fq=other_field:bar+facet=true+facet.field=facet_field_name"
> > > ....
> > >
> > > the string should be url encoded.
> > >
> > > If it is something that changes with every request, you should modify
> the
> > > jmeter test. If you open/load it with jmeter GUI, in the HTTP request
> > > processor you can define other additional fields to pass with the
> > request.
> > > These values can come from the CSV file, you'll see an example how to
> use
> > > that when you open the test difinition file.
> > >
> > > Cheers,
> > >
> > > roman
> > >
> > >
> > >
> > >
> > > On Mon, Sep 2, 2013 at 3:12 PM, Dmitry Kan <so...@gmail.com>
> wrote:
> > >
> > > > Hi Erick,
> > > >
> > > > Agree, this is perfectly fine to mix them in solr. But my question is
> > > about
> > > > solrjmeter input query format. Just couldn't find a suitable example
> on
> > > the
> > > > solrjmeter's github.
> > > >
> > > > Dmitry
> > > >
> > > >
> > > >
> > > > On Mon, Sep 2, 2013 at 5:40 PM, Erick Erickson <
> > erickerickson@gmail.com
> > > > >wrote:
> > > >
> > > > > filter and facet queries can be freely intermixed, it's not a
> > problem.
> > > > > What problem are you seeing when you try this?
> > > > >
> > > > > Best,
> > > > > Erick
> > > > >
> > > > >
> > > > > On Mon, Sep 2, 2013 at 7:46 AM, Dmitry Kan <so...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Roman,
> > > > > >
> > > > > > What's the format for running the facet+filter queries?
> > > > > >
> > > > > > Would something like this work:
> > > > > >
> > > > > > field:foo >=50 fq=other_field:bar facet=true
> > > > > facet.field=facet_field_name
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitry
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <
> solrexpert@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Roman,
> > > > > > >
> > > > > > > With adminPath="/admin" or adminPath="/admin/cores", no.
> > > > Interestingly
> > > > > > > enough, though, I can access
> > > > > > > http://localhost:8983/solr/statements/admin/system
> > > > > > >
> > > > > > > But I can access http://localhost:8983/solr/admin/cores, only
> > when
> > > > > with
> > > > > > > adminPath="/admin/cores" (which suggests that this is the right
> > > value
> > > > > to
> > > > > > be
> > > > > > > used for cores), and not with adminPath="/admin".
> > > > > > >
> > > > > > > Bottom line, these core configuration is not self-evident.
> > > > > > >
> > > > > > > Dmitry
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <
> > > roman.chyla@gmail.com
> > > > > > >wrote:
> > > > > > >
> > > > > > >> Hi Dmitry,
> > > > > > >> So it seems solrjmeter should not assume the adminPath - and
> > > perhaps
> > > > > > needs
> > > > > > >> to be passed as an argument. When you set the adminPath, are
> you
> > > > able
> > > > > to
> > > > > > >> access localhost:8983/solr/statements/admin/cores ?
> > > > > > >>
> > > > > > >> roman
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <
> > solrexpert@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi Roman,
> > > > > > >> >
> > > > > > >> > I have noticed a difference with different solr.xml config
> > > > contents.
> > > > > > It
> > > > > > >> is
> > > > > > >> > probably legit, but thought to let you know (tests run on
> > fresh
> > > > > > >> checkout as
> > > > > > >> > of today).
> > > > > > >> >
> > > > > > >> > As mentioned before, I have two cores configured in
> solr.xml.
> > If
> > > > the
> > > > > > >> file
> > > > > > >> > is:
> > > > > > >> >
> > > > > > >> > [code]
> > > > > > >> > <solr persistent="false">
> > > > > > >> >
> > > > > > >> > <!--
> > > > > > >> > adminPath: RequestHandler path to manage cores.
> > > > > > >> > If 'null' (or absent), cores will not be manageable via
> > > > request
> > > > > > >> handler
> > > > > > >> > -->
> > > > > > >> > <cores adminPath="/admin/cores" host="${host:}"
> > > > > > >> > hostPort="${jetty.port:8983}"
> > hostContext="${hostContext:solr}">
> > > > > > >> > <core name="metadata" instanceDir="metadata" />
> > > > > > >> > <core name="statements" instanceDir="statements" />
> > > > > > >> > </cores>
> > > > > > >> > </solr>
> > > > > > >> > [/code]
> > > > > > >> >
> > > > > > >> > then the instruction:
> > > > > > >> >
> > > > > > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > > > > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > > --durationInSecs
> > > > > > 60
> > > > > > >> -R
> > > > > > >> > cms -t /solr/statements -e statements -U 100
> > > > > > >> >
> > > > > > >> > works just fine. If however the solr.xml has adminPath set
> to
> > > > > "/admin"
> > > > > > >> > solrjmeter produces an error:
> > > > > > >> >
> > > > > > >> > [error]
> > > > > > >> > **ERROR**
> > > > > > >> > File "solrjmeter.py", line 1386, in <module>
> > > > > > >> > main(sys.argv)
> > > > > > >> > File "solrjmeter.py", line 1278, in main
> > > > > > >> > check_prerequisities(options)
> > > > > > >> > File "solrjmeter.py", line 375, in check_prerequisities
> > > > > > >> > error('Cannot find admin pages: %s, please report a
> bug' %
> > > > > apath)
> > > > > > >> > File "solrjmeter.py", line 66, in error
> > > > > > >> > traceback.print_stack()
> > > > > > >> > Cannot find admin pages: http://localhost:8983/solr/admin,
> > > please
> > > > > > >> report a
> > > > > > >> > bug
> > > > > > >> > [/error]
> > > > > > >> >
> > > > > > >> > With both solr.xml configs the following url returns just
> > fine:
> > > > > > >> >
> > > > > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> >
> > > > > > >> > Dmitry
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <
> > > solrexpert@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > Hi Roman,
> > > > > > >> > >
> > > > > > >> > > This looks much better, thanks! The ordinary non-comarison
> > > mode
> > > > > > works.
> > > > > > >> > > I'll post here, if there are other findings.
> > > > > > >> > >
> > > > > > >> > > Thanks for quick turnarounds,
> > > > > > >> > >
> > > > > > >> > > Dmitry
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <
> > > > > roman.chyla@gmail.com
> > > > > > >> > >wrote:
> > > > > > >> > >
> > > > > > >> > >> Hi Dmitry, oh yes, late night fixes... :) The latest
> commit
> > > > > should
> > > > > > >> make
> > > > > > >> > it
> > > > > > >> > >> work for you.
> > > > > > >> > >> Thanks!
> > > > > > >> > >>
> > > > > > >> > >> roman
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <
> > > > > solrexpert@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> > >>
> > > > > > >> > >> > Hi Roman,
> > > > > > >> > >> >
> > > > > > >> > >> > Something bad happened in fresh checkout:
> > > > > > >> > >> >
> > > > > > >> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > > > > >> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > > > > >> --durationInSecs
> > > > > > >> > 60
> > > > > > >> > >> -R
> > > > > > >> > >> > cms -t /solr/statements -e statements -U 100
> > > > > > >> > >> >
> > > > > > >> > >> > Traceback (most recent call last):
> > > > > > >> > >> > File "solrjmeter.py", line 1392, in <module>
> > > > > > >> > >> > main(sys.argv)
> > > > > > >> > >> > File "solrjmeter.py", line 1347, in main
> > > > > > >> > >> > save_into_file('before-test.json',
> > > > > > >> simplejson.dumps(before_test))
> > > > > > >> > >> > File
> > > > > "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
> > > > > > >> line
> > > > > > >> > >> 286,
> > > > > > >> > >> > in dumps
> > > > > > >> > >> > return _default_encoder.encode(obj)
> > > > > > >> > >> > File
> > > > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > > > >> line
> > > > > > >> > >> 226,
> > > > > > >> > >> > in encode
> > > > > > >> > >> > chunks = self.iterencode(o, _one_shot=True)
> > > > > > >> > >> > File
> > > > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > > > >> line
> > > > > > >> > >> 296,
> > > > > > >> > >> > in iterencode
> > > > > > >> > >> > return _iterencode(o, 0)
> > > > > > >> > >> > File
> > > > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > > > >> line
> > > > > > >> > >> 202,
> > > > > > >> > >> > in default
> > > > > > >> > >> > raise TypeError(repr(o) + " is not JSON
> > serializable")
> > > > > > >> > >> > TypeError: <__main__.ForgivingValue object at
> > > 0x7fc6d4040fd0>
> > > > > is
> > > > > > >> not
> > > > > > >> > >> JSON
> > > > > > >> > >> > serializable
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > Regards,
> > > > > > >> > >> >
> > > > > > >> > >> > D.
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
> > > > > > >> roman.chyla@gmail.com>
> > > > > > >> > >> > wrote:
> > > > > > >> > >> >
> > > > > > >> > >> > > Hi Dmitry,
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
> > > > > > >> solrexpert@gmail.com>
> > > > > > >> > >> > wrote:
> > > > > > >> > >> > >
> > > > > > >> > >> > > > Hi Roman,
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > Good point. I managed to run the command with -C
> and
> > > > double
> > > > > > >> > quotes:
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour -x
> > > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > As a result got several files (html, css, js, csv)
> in
> > > the
> > > > > > >> running
> > > > > > >> > >> > > directory
> > > > > > >> > >> > > > (any way to specify where the output should be
> stored
> > > in
> > > > > this
> > > > > > >> > case?)
> > > > > > >> > >> > > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > i know it is confusing, i plan to change it - but
> > later,
> > > > now
> > > > > it
> > > > > > >> is
> > > > > > >> > too
> > > > > > >> > >> > busy
> > > > > > >> > >> > > here...
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > When I look onto the comparison dashboard, I see
> > this:
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > http://pbrd.co/17IRI0b
> > > > > > >> > >> > > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > two things: the tests probably took more than one
> hour
> > to
> > > > > > >> finish, so
> > > > > > >> > >> they
> > > > > > >> > >> > > are not aligned - try generating the comparison with
> > '-c
> > > > > > 14400'
> > > > > > >> > (ie.
> > > > > > >> > >> > > 4x3600 secs)
> > > > > > >> > >> > >
> > > > > > >> > >> > > the other thing: if you have only two datapoints, the
> > > > dygraph
> > > > > > >> will
> > > > > > >> > not
> > > > > > >> > >> > show
> > > > > > >> > >> > > anything - there must be more datapoints/measurements
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > One more thing: all the previous tests were run
> with
> > > > > > softCommit
> > > > > > >> > >> > disabled.
> > > > > > >> > >> > > > After enabling it, the tests started to fail:
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > $ python solrjmeter.py -a -x
> ./jmx/SolrQueryTest.jmx
> > -q
> > > > > > >> > >> > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > > > > >> > >> --durationInSecs 60
> > > > > > >> > >> > > -R
> > > > > > >> > >> > > > g1 -t /solr/statements -e statements -U 100
> > > > > > >> > >> > > > $ cd g1
> > > > > > >> > >> > > > Reading results of the previous test
> > > > > > >> > >> > > > $ cd 2013.08.12.16.32.48
> > > > > > >> > >> > > > $ cd
> > > /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > > > > >> > >> > > > $ mkdir 2013.08.12.16.33.02
> > > > > > >> > >> > > > $ cd 2013.08.12.16.33.02
> > > > > > >> > >> > > > $ cd
> > > /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > > > > >> > >> > > > $ cd
> /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > > > > >> > >> > > > $ cd
> /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > > > > >> > >> > > > Traceback (most recent call last):
> > > > > > >> > >> > > > File "solrjmeter.py", line 1427, in <module>
> > > > > > >> > >> > > > main(sys.argv)
> > > > > > >> > >> > > > File "solrjmeter.py", line 1381, in main
> > > > > > >> > >> > > > before_test =
> > > > harvest_details_about_montysolr(options)
> > > > > > >> > >> > > > File "solrjmeter.py", line 562, in
> > > > > > >> > harvest_details_about_montysolr
> > > > > > >> > >> > > > indexLstModified =
> > > > > > >> > >> > cores_data['status'][cn]['index']['lastModified'],
> > > > > > >> > >> > > > KeyError: 'lastModified'
> > > > > > >> > >> > > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > Thanks for letting me know, that info is probably not
> > > > > available
> > > > > > >> in
> > > > > > >> > >> this
> > > > > > >> > >> > > situation - i've cooked st quick to fix it, please
> try
> > > the
> > > > > > latest
> > > > > > >> > >> commit
> > > > > > >> > >> > > (hope it doesn't do more harm, i should get some
> sleep
> > > > ..;))
> > > > > > >> > >> > >
> > > > > > >> > >> > > roman
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > In case it matters: Python 2.7.3, ubuntu, solr
> > 4.3.1.
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > Thanks,
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > Dmitry
> > > > > > >> > >> > > >
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla <
> > > > > > >> > roman.chyla@gmail.com>
> > > > > > >> > >> > > wrote:
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > > Hi Dmitry,
> > > > > > >> > >> > > > > The command seems good. Are you sure your shell
> is
> > > not
> > > > > > doing
> > > > > > >> > >> > something
> > > > > > >> > >> > > > > funny with the params? You could try:
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > > >> > >> > -a
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > where g1 and foo are results of the individual
> > runs,
> > > > ie.
> > > > > > >> > something
> > > > > > >> > >> > that
> > > > > > >> > >> > > > was
> > > > > > >> > >> > > > > started and saved with '-R g1' and '-R foo'
> > > > respectively
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > so, for example, i have these comparisons inside
> > > > > > >> > >> > > > >
> > > > '/var/lib/montysolr/different-java-settings/solrjmeter',
> > > > > so
> > > > > > >> I am
> > > > > > >> > >> > > > generating
> > > > > > >> > >> > > > > the comparison by:
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > export
> > > > > > >> > >> > > > >
> > > > > > >> > >>
> > > > > >
> > SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
> > > > > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > > >> > >> > -a
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > roman
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry Kan <
> > > > > > >> > solrexpert@gmail.com
> > > > > > >> > >> >
> > > > > > >> > >> > > > wrote:
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > > Hi Roman,
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > > One more question. I tried to compare different
> > > runs
> > > > > (g1
> > > > > > vs
> > > > > > >> > cms)
> > > > > > >> > >> > > using
> > > > > > >> > >> > > > > the
> > > > > > >> > >> > > > > > command below, but get an error. Should I
> attach
> > > some
> > > > > > other
> > > > > > >> > >> > param(s)?
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > > python solrjmeter.py -C g1,foo -c hour -x
> > > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > > >> > >> > > > > > **ERROR**
> > > > > > >> > >> > > > > > File "solrjmeter.py", line 1427, in <module>
> > > > > > >> > >> > > > > > main(sys.argv)
> > > > > > >> > >> > > > > > File "solrjmeter.py", line 1303, in main
> > > > > > >> > >> > > > > > check_options(options, args)
> > > > > > >> > >> > > > > > File "solrjmeter.py", line 185, in
> > check_options
> > > > > > >> > >> > > > > > error("The folder '%s' does not exist" %
> rf)
> > > > > > >> > >> > > > > > File "solrjmeter.py", line 66, in error
> > > > > > >> > >> > > > > > traceback.print_stack()
> > > > > > >> > >> > > > > > The folder '0' does not exist
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > > Dmitry
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM, Dmitry Kan <
> > > > > > >> > >> solrexpert@gmail.com>
> > > > > > >> > >> > > > wrote:
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > > > Hi Roman,
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > > Finally, this has worked! Thanks for quick
> > > support.
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > > The graphs look awesome. At least on the
> index
> > > > sample
> > > > > > :)
> > > > > > >> It
> > > > > > >> > is
> > > > > > >> > >> > > quite
> > > > > > >> > >> > > > > easy
> > > > > > >> > >> > > > > > > to setup and run + possible to run directly
> on
> > > the
> > > > > > shard
> > > > > > >> > >> server
> > > > > > >> > >> > in
> > > > > > >> > >> > > > > > > background mode.
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > > my test run was:
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > > python solrjmeter.py -a -x
> > > ./jmx/SolrQueryTest.jmx
> > > > -q
> > > > > > >> > >> > > > > > > ./queries/demo/demo.queries -s localhost -p
> > 8983
> > > -a
> > > > > > >> > >> > > --durationInSecs
> > > > > > >> > >> > > > 60
> > > > > > >> > >> > > > > > -R
> > > > > > >> > >> > > > > > > foo -t /solr/statements -e statements
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > > Thanks!
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > > Dmitry
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54 AM, Roman Chyla <
> > > > > > >> > >> > roman.chyla@gmail.com
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > > > wrote:
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > >> Hi Dmitry,
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >> I've modified the solrjmeter to retrieve
> data
> > > from
> > > > > > under
> > > > > > >> > the
> > > > > > >> > >> > core
> > > > > > >> > >> > > > (the
> > > > > > >> > >> > > > > > -t
> > > > > > >> > >> > > > > > >> parameter) and the rest from the /solr/admin
> > - I
> > > > > could
> > > > > > >> test
> > > > > > >> > >> it
> > > > > > >> > >> > > only
> > > > > > >> > >> > > > > > >> against
> > > > > > >> > >> > > > > > >> 4.0, but it is there the same as 4.3 - it
> > > > seems...so
> > > > > > you
> > > > > > >> > can
> > > > > > >> > >> try
> > > > > > >> > >> > > the
> > > > > > >> > >> > > > > > fresh
> > > > > > >> > >> > > > > > >> checkout
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >> my test was: python solrjmeter.py -a -x
> > > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > > >> > >> > -t
> > > > > > >> > >> > > > > > >> /solr/collection1 -R foo -q ./queries/demo/*
> > -p
> > > > 9002
> > > > > > -s
> > > > > > >> > >> adsate
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >> Thanks!
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >> roman
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >> On Tue, Aug 6, 2013 at 9:46 AM, Dmitry Kan <
> > > > > > >> > >> > solrexpert@gmail.com>
> > > > > > >> > >> > > > > > wrote:
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >> > Hi,
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > Thanks for the clarification, Shawn!
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > So with this in mind, the following work:
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > not copying their output to save space.
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > Roman:
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > is this something that should be set via
> -t
> > > > > > parameter
> > > > > > >> as
> > > > > > >> > >> well?
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > Dmitry
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > On Tue, Aug 6, 2013 at 4:34 PM, Shawn
> > Heisey <
> > > > > > >> > >> > solr@elyograg.org
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > > > wrote:
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >> > > On 8/6/2013 6:17 AM, Dmitry Kan wrote:
> > > > > > >> > >> > > > > > >> > > > Of three URLs you asked for, only the
> > 3rd
> > > > one
> > > > > > gave
> > > > > > >> > >> > response:
> > > > > > >> > >> > > > > > >> > > <snip>
> > > > > > >> > >> > > > > > >> > > > The rest report 404.
> > > > > > >> > >> > > > > > >> > > >
> > > > > > >> > >> > > > > > >> > > > On Mon, Aug 5, 2013 at 8:38 PM, Roman
> > > Chyla
> > > > <
> > > > > > >> > >> > > > > > roman.chyla@gmail.com>
> > > > > > >> > >> > > > > > >> > > wrote:
> > > > > > >> > >> > > > > > >> > > >
> > > > > > >> > >> > > > > > >> > > >> Hi Dmitry,
> > > > > > >> > >> > > > > > >> > > >> So I think the admin pages are
> > different
> > > on
> > > > > > your
> > > > > > >> > >> version
> > > > > > >> > >> > of
> > > > > > >> > >> > > > > solr,
> > > > > > >> > >> > > > > > >> what
> > > > > > >> > >> > > > > > >> > > do
> > > > > > >> > >> > > > > > >> > > >> you see when you request... ?
> > > > > > >> > >> > > > > > >> > > >>
> > > > > > >> > >> > > > > > >> > > >>
> > > > > > http://localhost:8983/solr/admin/system?wt=json
> > > > > > >> > >> > > > > > >> > > >>
> > > > > > http://localhost:8983/solr/admin/mbeans?wt=json
> > > > > > >> > >> > > > > > >> > > >>
> > > > > http://localhost:8983/solr/admin/cores?wt=json
> > > > > > >> > >> > > > > > >> > >
> > > > > > >> > >> > > > > > >> > > Unless you have a valid defaultCoreName
> > set
> > > in
> > > > > > your
> > > > > > >> > >> > > (old-style)
> > > > > > >> > >> > > > > > >> > > solr.xml, the first two URLs won't work,
> > as
> > > > > you've
> > > > > > >> > >> > discovered.
> > > > > > >> > >> > > > > > >> Without
> > > > > > >> > >> > > > > > >> > > that valid defaultCoreName (or if you
> > wanted
> > > > > info
> > > > > > >> from
> > > > > > >> > a
> > > > > > >> > >> > > > different
> > > > > > >> > >> > > > > > >> > > core), you'd need to add a core name to
> > the
> > > > URL
> > > > > > for
> > > > > > >> > them
> > > > > > >> > >> to
> > > > > > >> > >> > > > work.
> > > > > > >> > >> > > > > > >> > >
> > > > > > >> > >> > > > > > >> > > The third one, which works for you, is a
> > > > global
> > > > > > >> handler
> > > > > > >> > >> for
> > > > > > >> > >> > > > > > >> manipulating
> > > > > > >> > >> > > > > > >> > > cores, so naturally it doesn't need a
> core
> > > > name
> > > > > to
> > > > > > >> > >> function.
> > > > > > >> > >> > > > The
> > > > > > >> > >> > > > > > URL
> > > > > > >> > >> > > > > > >> > > path for this handler is defined by
> > > solr.xml.
> > > > > > >> > >> > > > > > >> > >
> > > > > > >> > >> > > > > > >> > > Thanks,
> > > > > > >> > >> > > > > > >> > > Shawn
> > > > > > >> > >> > > > > > >> > >
> > > > > > >> > >> > > > > > >> > >
> > > > > > >> > >> > > > > > >> >
> > > > > > >> > >> > > > > > >>
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > > >
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > >
> > > > > > >> > >> > >
> > > > > > >> > >> >
> > > > > > >> > >>
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Measuring SOLR performance
Posted by Roman Chyla <ro...@gmail.com>.
Hi Dmitry,
Thanks for the feedback. Yes, it is indeed jmeter issue (or rather, the
issue of the plugin we use to generate charts). You may want to use the
github for whatever comes next
https://github.com/romanchyla/solrjmeter/issues
Cheers,
roman
On Tue, Sep 3, 2013 at 7:54 AM, Dmitry Kan <so...@gmail.com> wrote:
> Hi Roman,
>
> Thanks, the --additionalSolrParams was just what I wanted and works fine.
>
> BTW, if you have some special "bug tracking forum" for the tool, I'm happy
> to submit questions / bug reports there. Otherwise, this email list is ok
> (for me at least).
>
> One other thing I have noticed in the err logs was a series of messages of
> this sort upon generating the perf test report. Seems to be jmeter related
> (the err messages disappear, if extra lib dir is present under ext
> directory).
>
> java.lang.Throwable: Could not access
> /home/dmitry/projects/lab/solrjmeter7/solrjmeter/jmeter/lib/ext/lib
> at
> kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
> at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
> at
> kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
> at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
>
> at
> kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
> at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
>
>
>
> On Tue, Sep 3, 2013 at 2:50 AM, Roman Chyla <ro...@gmail.com> wrote:
>
> > Hi Dmitry,
> >
> > If it is something you want to pass with every request (which is my use
> > case), you can pass it as additional solr params, eg.
> >
> > python solrjmeter
> >
> >
> --additionalSolrParams="fq=other_field:bar+facet=true+facet.field=facet_field_name"
> > ....
> >
> > the string should be url encoded.
> >
> > If it is something that changes with every request, you should modify the
> > jmeter test. If you open/load it with jmeter GUI, in the HTTP request
> > processor you can define other additional fields to pass with the
> request.
> > These values can come from the CSV file, you'll see an example how to use
> > that when you open the test difinition file.
> >
> > Cheers,
> >
> > roman
> >
> >
> >
> >
> > On Mon, Sep 2, 2013 at 3:12 PM, Dmitry Kan <so...@gmail.com> wrote:
> >
> > > Hi Erick,
> > >
> > > Agree, this is perfectly fine to mix them in solr. But my question is
> > about
> > > solrjmeter input query format. Just couldn't find a suitable example on
> > the
> > > solrjmeter's github.
> > >
> > > Dmitry
> > >
> > >
> > >
> > > On Mon, Sep 2, 2013 at 5:40 PM, Erick Erickson <
> erickerickson@gmail.com
> > > >wrote:
> > >
> > > > filter and facet queries can be freely intermixed, it's not a
> problem.
> > > > What problem are you seeing when you try this?
> > > >
> > > > Best,
> > > > Erick
> > > >
> > > >
> > > > On Mon, Sep 2, 2013 at 7:46 AM, Dmitry Kan <so...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Roman,
> > > > >
> > > > > What's the format for running the facet+filter queries?
> > > > >
> > > > > Would something like this work:
> > > > >
> > > > > field:foo >=50 fq=other_field:bar facet=true
> > > > facet.field=facet_field_name
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Dmitry
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <so...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi Roman,
> > > > > >
> > > > > > With adminPath="/admin" or adminPath="/admin/cores", no.
> > > Interestingly
> > > > > > enough, though, I can access
> > > > > > http://localhost:8983/solr/statements/admin/system
> > > > > >
> > > > > > But I can access http://localhost:8983/solr/admin/cores, only
> when
> > > > with
> > > > > > adminPath="/admin/cores" (which suggests that this is the right
> > value
> > > > to
> > > > > be
> > > > > > used for cores), and not with adminPath="/admin".
> > > > > >
> > > > > > Bottom line, these core configuration is not self-evident.
> > > > > >
> > > > > > Dmitry
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <
> > roman.chyla@gmail.com
> > > > > >wrote:
> > > > > >
> > > > > >> Hi Dmitry,
> > > > > >> So it seems solrjmeter should not assume the adminPath - and
> > perhaps
> > > > > needs
> > > > > >> to be passed as an argument. When you set the adminPath, are you
> > > able
> > > > to
> > > > > >> access localhost:8983/solr/statements/admin/cores ?
> > > > > >>
> > > > > >> roman
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <
> solrexpert@gmail.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > Hi Roman,
> > > > > >> >
> > > > > >> > I have noticed a difference with different solr.xml config
> > > contents.
> > > > > It
> > > > > >> is
> > > > > >> > probably legit, but thought to let you know (tests run on
> fresh
> > > > > >> checkout as
> > > > > >> > of today).
> > > > > >> >
> > > > > >> > As mentioned before, I have two cores configured in solr.xml.
> If
> > > the
> > > > > >> file
> > > > > >> > is:
> > > > > >> >
> > > > > >> > [code]
> > > > > >> > <solr persistent="false">
> > > > > >> >
> > > > > >> > <!--
> > > > > >> > adminPath: RequestHandler path to manage cores.
> > > > > >> > If 'null' (or absent), cores will not be manageable via
> > > request
> > > > > >> handler
> > > > > >> > -->
> > > > > >> > <cores adminPath="/admin/cores" host="${host:}"
> > > > > >> > hostPort="${jetty.port:8983}"
> hostContext="${hostContext:solr}">
> > > > > >> > <core name="metadata" instanceDir="metadata" />
> > > > > >> > <core name="statements" instanceDir="statements" />
> > > > > >> > </cores>
> > > > > >> > </solr>
> > > > > >> > [/code]
> > > > > >> >
> > > > > >> > then the instruction:
> > > > > >> >
> > > > > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > > > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > --durationInSecs
> > > > > 60
> > > > > >> -R
> > > > > >> > cms -t /solr/statements -e statements -U 100
> > > > > >> >
> > > > > >> > works just fine. If however the solr.xml has adminPath set to
> > > > "/admin"
> > > > > >> > solrjmeter produces an error:
> > > > > >> >
> > > > > >> > [error]
> > > > > >> > **ERROR**
> > > > > >> > File "solrjmeter.py", line 1386, in <module>
> > > > > >> > main(sys.argv)
> > > > > >> > File "solrjmeter.py", line 1278, in main
> > > > > >> > check_prerequisities(options)
> > > > > >> > File "solrjmeter.py", line 375, in check_prerequisities
> > > > > >> > error('Cannot find admin pages: %s, please report a bug' %
> > > > apath)
> > > > > >> > File "solrjmeter.py", line 66, in error
> > > > > >> > traceback.print_stack()
> > > > > >> > Cannot find admin pages: http://localhost:8983/solr/admin,
> > please
> > > > > >> report a
> > > > > >> > bug
> > > > > >> > [/error]
> > > > > >> >
> > > > > >> > With both solr.xml configs the following url returns just
> fine:
> > > > > >> >
> > > > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > > > >> >
> > > > > >> > Regards,
> > > > > >> >
> > > > > >> > Dmitry
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <
> > solrexpert@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Hi Roman,
> > > > > >> > >
> > > > > >> > > This looks much better, thanks! The ordinary non-comarison
> > mode
> > > > > works.
> > > > > >> > > I'll post here, if there are other findings.
> > > > > >> > >
> > > > > >> > > Thanks for quick turnarounds,
> > > > > >> > >
> > > > > >> > > Dmitry
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <
> > > > roman.chyla@gmail.com
> > > > > >> > >wrote:
> > > > > >> > >
> > > > > >> > >> Hi Dmitry, oh yes, late night fixes... :) The latest commit
> > > > should
> > > > > >> make
> > > > > >> > it
> > > > > >> > >> work for you.
> > > > > >> > >> Thanks!
> > > > > >> > >>
> > > > > >> > >> roman
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <
> > > > solrexpert@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >>
> > > > > >> > >> > Hi Roman,
> > > > > >> > >> >
> > > > > >> > >> > Something bad happened in fresh checkout:
> > > > > >> > >> >
> > > > > >> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > > > >> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > > > >> --durationInSecs
> > > > > >> > 60
> > > > > >> > >> -R
> > > > > >> > >> > cms -t /solr/statements -e statements -U 100
> > > > > >> > >> >
> > > > > >> > >> > Traceback (most recent call last):
> > > > > >> > >> > File "solrjmeter.py", line 1392, in <module>
> > > > > >> > >> > main(sys.argv)
> > > > > >> > >> > File "solrjmeter.py", line 1347, in main
> > > > > >> > >> > save_into_file('before-test.json',
> > > > > >> simplejson.dumps(before_test))
> > > > > >> > >> > File
> > > > "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
> > > > > >> line
> > > > > >> > >> 286,
> > > > > >> > >> > in dumps
> > > > > >> > >> > return _default_encoder.encode(obj)
> > > > > >> > >> > File
> > > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > > >> line
> > > > > >> > >> 226,
> > > > > >> > >> > in encode
> > > > > >> > >> > chunks = self.iterencode(o, _one_shot=True)
> > > > > >> > >> > File
> > > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > > >> line
> > > > > >> > >> 296,
> > > > > >> > >> > in iterencode
> > > > > >> > >> > return _iterencode(o, 0)
> > > > > >> > >> > File
> > > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > > >> line
> > > > > >> > >> 202,
> > > > > >> > >> > in default
> > > > > >> > >> > raise TypeError(repr(o) + " is not JSON
> serializable")
> > > > > >> > >> > TypeError: <__main__.ForgivingValue object at
> > 0x7fc6d4040fd0>
> > > > is
> > > > > >> not
> > > > > >> > >> JSON
> > > > > >> > >> > serializable
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > Regards,
> > > > > >> > >> >
> > > > > >> > >> > D.
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
> > > > > >> roman.chyla@gmail.com>
> > > > > >> > >> > wrote:
> > > > > >> > >> >
> > > > > >> > >> > > Hi Dmitry,
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
> > > > > >> solrexpert@gmail.com>
> > > > > >> > >> > wrote:
> > > > > >> > >> > >
> > > > > >> > >> > > > Hi Roman,
> > > > > >> > >> > > >
> > > > > >> > >> > > > Good point. I managed to run the command with -C and
> > > double
> > > > > >> > quotes:
> > > > > >> > >> > > >
> > > > > >> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour -x
> > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > >> > >> > > >
> > > > > >> > >> > > > As a result got several files (html, css, js, csv) in
> > the
> > > > > >> running
> > > > > >> > >> > > directory
> > > > > >> > >> > > > (any way to specify where the output should be stored
> > in
> > > > this
> > > > > >> > case?)
> > > > > >> > >> > > >
> > > > > >> > >> > >
> > > > > >> > >> > > i know it is confusing, i plan to change it - but
> later,
> > > now
> > > > it
> > > > > >> is
> > > > > >> > too
> > > > > >> > >> > busy
> > > > > >> > >> > > here...
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > > >
> > > > > >> > >> > > > When I look onto the comparison dashboard, I see
> this:
> > > > > >> > >> > > >
> > > > > >> > >> > > > http://pbrd.co/17IRI0b
> > > > > >> > >> > > >
> > > > > >> > >> > >
> > > > > >> > >> > > two things: the tests probably took more than one hour
> to
> > > > > >> finish, so
> > > > > >> > >> they
> > > > > >> > >> > > are not aligned - try generating the comparison with
> '-c
> > > > > 14400'
> > > > > >> > (ie.
> > > > > >> > >> > > 4x3600 secs)
> > > > > >> > >> > >
> > > > > >> > >> > > the other thing: if you have only two datapoints, the
> > > dygraph
> > > > > >> will
> > > > > >> > not
> > > > > >> > >> > show
> > > > > >> > >> > > anything - there must be more datapoints/measurements
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > > >
> > > > > >> > >> > > > One more thing: all the previous tests were run with
> > > > > softCommit
> > > > > >> > >> > disabled.
> > > > > >> > >> > > > After enabling it, the tests started to fail:
> > > > > >> > >> > > >
> > > > > >> > >> > > > $ python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx
> -q
> > > > > >> > >> > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > > > >> > >> --durationInSecs 60
> > > > > >> > >> > > -R
> > > > > >> > >> > > > g1 -t /solr/statements -e statements -U 100
> > > > > >> > >> > > > $ cd g1
> > > > > >> > >> > > > Reading results of the previous test
> > > > > >> > >> > > > $ cd 2013.08.12.16.32.48
> > > > > >> > >> > > > $ cd
> > /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > > > >> > >> > > > $ mkdir 2013.08.12.16.33.02
> > > > > >> > >> > > > $ cd 2013.08.12.16.33.02
> > > > > >> > >> > > > $ cd
> > /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > > > >> > >> > > > Traceback (most recent call last):
> > > > > >> > >> > > > File "solrjmeter.py", line 1427, in <module>
> > > > > >> > >> > > > main(sys.argv)
> > > > > >> > >> > > > File "solrjmeter.py", line 1381, in main
> > > > > >> > >> > > > before_test =
> > > harvest_details_about_montysolr(options)
> > > > > >> > >> > > > File "solrjmeter.py", line 562, in
> > > > > >> > harvest_details_about_montysolr
> > > > > >> > >> > > > indexLstModified =
> > > > > >> > >> > cores_data['status'][cn]['index']['lastModified'],
> > > > > >> > >> > > > KeyError: 'lastModified'
> > > > > >> > >> > > >
> > > > > >> > >> > >
> > > > > >> > >> > > Thanks for letting me know, that info is probably not
> > > > available
> > > > > >> in
> > > > > >> > >> this
> > > > > >> > >> > > situation - i've cooked st quick to fix it, please try
> > the
> > > > > latest
> > > > > >> > >> commit
> > > > > >> > >> > > (hope it doesn't do more harm, i should get some sleep
> > > ..;))
> > > > > >> > >> > >
> > > > > >> > >> > > roman
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > > >
> > > > > >> > >> > > > In case it matters: Python 2.7.3, ubuntu, solr
> 4.3.1.
> > > > > >> > >> > > >
> > > > > >> > >> > > > Thanks,
> > > > > >> > >> > > >
> > > > > >> > >> > > > Dmitry
> > > > > >> > >> > > >
> > > > > >> > >> > > >
> > > > > >> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla <
> > > > > >> > roman.chyla@gmail.com>
> > > > > >> > >> > > wrote:
> > > > > >> > >> > > >
> > > > > >> > >> > > > > Hi Dmitry,
> > > > > >> > >> > > > > The command seems good. Are you sure your shell is
> > not
> > > > > doing
> > > > > >> > >> > something
> > > > > >> > >> > > > > funny with the params? You could try:
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > >> > >> > -a
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > where g1 and foo are results of the individual
> runs,
> > > ie.
> > > > > >> > something
> > > > > >> > >> > that
> > > > > >> > >> > > > was
> > > > > >> > >> > > > > started and saved with '-R g1' and '-R foo'
> > > respectively
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > so, for example, i have these comparisons inside
> > > > > >> > >> > > > >
> > > '/var/lib/montysolr/different-java-settings/solrjmeter',
> > > > so
> > > > > >> I am
> > > > > >> > >> > > > generating
> > > > > >> > >> > > > > the comparison by:
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > export
> > > > > >> > >> > > > >
> > > > > >> > >>
> > > > >
> SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
> > > > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > >> > >> > -a
> > > > > >> > >> > > > >
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > roman
> > > > > >> > >> > > > >
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry Kan <
> > > > > >> > solrexpert@gmail.com
> > > > > >> > >> >
> > > > > >> > >> > > > wrote:
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > > Hi Roman,
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > > One more question. I tried to compare different
> > runs
> > > > (g1
> > > > > vs
> > > > > >> > cms)
> > > > > >> > >> > > using
> > > > > >> > >> > > > > the
> > > > > >> > >> > > > > > command below, but get an error. Should I attach
> > some
> > > > > other
> > > > > >> > >> > param(s)?
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > > python solrjmeter.py -C g1,foo -c hour -x
> > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > >> > >> > > > > > **ERROR**
> > > > > >> > >> > > > > > File "solrjmeter.py", line 1427, in <module>
> > > > > >> > >> > > > > > main(sys.argv)
> > > > > >> > >> > > > > > File "solrjmeter.py", line 1303, in main
> > > > > >> > >> > > > > > check_options(options, args)
> > > > > >> > >> > > > > > File "solrjmeter.py", line 185, in
> check_options
> > > > > >> > >> > > > > > error("The folder '%s' does not exist" % rf)
> > > > > >> > >> > > > > > File "solrjmeter.py", line 66, in error
> > > > > >> > >> > > > > > traceback.print_stack()
> > > > > >> > >> > > > > > The folder '0' does not exist
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > > Dmitry
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM, Dmitry Kan <
> > > > > >> > >> solrexpert@gmail.com>
> > > > > >> > >> > > > wrote:
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > > > Hi Roman,
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > > Finally, this has worked! Thanks for quick
> > support.
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > > The graphs look awesome. At least on the index
> > > sample
> > > > > :)
> > > > > >> It
> > > > > >> > is
> > > > > >> > >> > > quite
> > > > > >> > >> > > > > easy
> > > > > >> > >> > > > > > > to setup and run + possible to run directly on
> > the
> > > > > shard
> > > > > >> > >> server
> > > > > >> > >> > in
> > > > > >> > >> > > > > > > background mode.
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > > my test run was:
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > > python solrjmeter.py -a -x
> > ./jmx/SolrQueryTest.jmx
> > > -q
> > > > > >> > >> > > > > > > ./queries/demo/demo.queries -s localhost -p
> 8983
> > -a
> > > > > >> > >> > > --durationInSecs
> > > > > >> > >> > > > 60
> > > > > >> > >> > > > > > -R
> > > > > >> > >> > > > > > > foo -t /solr/statements -e statements
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > > Thanks!
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > > Dmitry
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54 AM, Roman Chyla <
> > > > > >> > >> > roman.chyla@gmail.com
> > > > > >> > >> > > >
> > > > > >> > >> > > > > > wrote:
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > >> Hi Dmitry,
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >> I've modified the solrjmeter to retrieve data
> > from
> > > > > under
> > > > > >> > the
> > > > > >> > >> > core
> > > > > >> > >> > > > (the
> > > > > >> > >> > > > > > -t
> > > > > >> > >> > > > > > >> parameter) and the rest from the /solr/admin
> - I
> > > > could
> > > > > >> test
> > > > > >> > >> it
> > > > > >> > >> > > only
> > > > > >> > >> > > > > > >> against
> > > > > >> > >> > > > > > >> 4.0, but it is there the same as 4.3 - it
> > > seems...so
> > > > > you
> > > > > >> > can
> > > > > >> > >> try
> > > > > >> > >> > > the
> > > > > >> > >> > > > > > fresh
> > > > > >> > >> > > > > > >> checkout
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >> my test was: python solrjmeter.py -a -x
> > > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > > >> > >> > -t
> > > > > >> > >> > > > > > >> /solr/collection1 -R foo -q ./queries/demo/*
> -p
> > > 9002
> > > > > -s
> > > > > >> > >> adsate
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >> Thanks!
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >> roman
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >> On Tue, Aug 6, 2013 at 9:46 AM, Dmitry Kan <
> > > > > >> > >> > solrexpert@gmail.com>
> > > > > >> > >> > > > > > wrote:
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >> > Hi,
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > Thanks for the clarification, Shawn!
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > So with this in mind, the following work:
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> >
> > > > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > > > >> > >> > > > > > >> >
> > > > > >> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > not copying their output to save space.
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > Roman:
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > is this something that should be set via -t
> > > > > parameter
> > > > > >> as
> > > > > >> > >> well?
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > Dmitry
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > On Tue, Aug 6, 2013 at 4:34 PM, Shawn
> Heisey <
> > > > > >> > >> > solr@elyograg.org
> > > > > >> > >> > > >
> > > > > >> > >> > > > > > wrote:
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >> > > On 8/6/2013 6:17 AM, Dmitry Kan wrote:
> > > > > >> > >> > > > > > >> > > > Of three URLs you asked for, only the
> 3rd
> > > one
> > > > > gave
> > > > > >> > >> > response:
> > > > > >> > >> > > > > > >> > > <snip>
> > > > > >> > >> > > > > > >> > > > The rest report 404.
> > > > > >> > >> > > > > > >> > > >
> > > > > >> > >> > > > > > >> > > > On Mon, Aug 5, 2013 at 8:38 PM, Roman
> > Chyla
> > > <
> > > > > >> > >> > > > > > roman.chyla@gmail.com>
> > > > > >> > >> > > > > > >> > > wrote:
> > > > > >> > >> > > > > > >> > > >
> > > > > >> > >> > > > > > >> > > >> Hi Dmitry,
> > > > > >> > >> > > > > > >> > > >> So I think the admin pages are
> different
> > on
> > > > > your
> > > > > >> > >> version
> > > > > >> > >> > of
> > > > > >> > >> > > > > solr,
> > > > > >> > >> > > > > > >> what
> > > > > >> > >> > > > > > >> > > do
> > > > > >> > >> > > > > > >> > > >> you see when you request... ?
> > > > > >> > >> > > > > > >> > > >>
> > > > > >> > >> > > > > > >> > > >>
> > > > > http://localhost:8983/solr/admin/system?wt=json
> > > > > >> > >> > > > > > >> > > >>
> > > > > http://localhost:8983/solr/admin/mbeans?wt=json
> > > > > >> > >> > > > > > >> > > >>
> > > > http://localhost:8983/solr/admin/cores?wt=json
> > > > > >> > >> > > > > > >> > >
> > > > > >> > >> > > > > > >> > > Unless you have a valid defaultCoreName
> set
> > in
> > > > > your
> > > > > >> > >> > > (old-style)
> > > > > >> > >> > > > > > >> > > solr.xml, the first two URLs won't work,
> as
> > > > you've
> > > > > >> > >> > discovered.
> > > > > >> > >> > > > > > >> Without
> > > > > >> > >> > > > > > >> > > that valid defaultCoreName (or if you
> wanted
> > > > info
> > > > > >> from
> > > > > >> > a
> > > > > >> > >> > > > different
> > > > > >> > >> > > > > > >> > > core), you'd need to add a core name to
> the
> > > URL
> > > > > for
> > > > > >> > them
> > > > > >> > >> to
> > > > > >> > >> > > > work.
> > > > > >> > >> > > > > > >> > >
> > > > > >> > >> > > > > > >> > > The third one, which works for you, is a
> > > global
> > > > > >> handler
> > > > > >> > >> for
> > > > > >> > >> > > > > > >> manipulating
> > > > > >> > >> > > > > > >> > > cores, so naturally it doesn't need a core
> > > name
> > > > to
> > > > > >> > >> function.
> > > > > >> > >> > > > The
> > > > > >> > >> > > > > > URL
> > > > > >> > >> > > > > > >> > > path for this handler is defined by
> > solr.xml.
> > > > > >> > >> > > > > > >> > >
> > > > > >> > >> > > > > > >> > > Thanks,
> > > > > >> > >> > > > > > >> > > Shawn
> > > > > >> > >> > > > > > >> > >
> > > > > >> > >> > > > > > >> > >
> > > > > >> > >> > > > > > >> >
> > > > > >> > >> > > > > > >>
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > > >
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > >
> > > > > >> > >> > > >
> > > > > >> > >> > >
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Measuring SOLR performance
Posted by Dmitry Kan <so...@gmail.com>.
Hi Roman,
Thanks, the --additionalSolrParams was just what I wanted and works fine.
BTW, if you have some special "bug tracking forum" for the tool, I'm happy
to submit questions / bug reports there. Otherwise, this email list is ok
(for me at least).
One other thing I have noticed in the err logs was a series of messages of
this sort upon generating the perf test report. Seems to be jmeter related
(the err messages disappear, if extra lib dir is present under ext
directory).
java.lang.Throwable: Could not access
/home/dmitry/projects/lab/solrjmeter7/solrjmeter/jmeter/lib/ext/lib
at
kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
at
kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
at
kg.apc.cmd.UniversalRunner.buildUpdatedClassPath(UniversalRunner.java:109)
at kg.apc.cmd.UniversalRunner.<clinit>(UniversalRunner.java:55)
On Tue, Sep 3, 2013 at 2:50 AM, Roman Chyla <ro...@gmail.com> wrote:
> Hi Dmitry,
>
> If it is something you want to pass with every request (which is my use
> case), you can pass it as additional solr params, eg.
>
> python solrjmeter
>
> --additionalSolrParams="fq=other_field:bar+facet=true+facet.field=facet_field_name"
> ....
>
> the string should be url encoded.
>
> If it is something that changes with every request, you should modify the
> jmeter test. If you open/load it with jmeter GUI, in the HTTP request
> processor you can define other additional fields to pass with the request.
> These values can come from the CSV file, you'll see an example how to use
> that when you open the test difinition file.
>
> Cheers,
>
> roman
>
>
>
>
> On Mon, Sep 2, 2013 at 3:12 PM, Dmitry Kan <so...@gmail.com> wrote:
>
> > Hi Erick,
> >
> > Agree, this is perfectly fine to mix them in solr. But my question is
> about
> > solrjmeter input query format. Just couldn't find a suitable example on
> the
> > solrjmeter's github.
> >
> > Dmitry
> >
> >
> >
> > On Mon, Sep 2, 2013 at 5:40 PM, Erick Erickson <erickerickson@gmail.com
> > >wrote:
> >
> > > filter and facet queries can be freely intermixed, it's not a problem.
> > > What problem are you seeing when you try this?
> > >
> > > Best,
> > > Erick
> > >
> > >
> > > On Mon, Sep 2, 2013 at 7:46 AM, Dmitry Kan <so...@gmail.com>
> wrote:
> > >
> > > > Hi Roman,
> > > >
> > > > What's the format for running the facet+filter queries?
> > > >
> > > > Would something like this work:
> > > >
> > > > field:foo >=50 fq=other_field:bar facet=true
> > > facet.field=facet_field_name
> > > >
> > > >
> > > > Thanks,
> > > > Dmitry
> > > >
> > > >
> > > >
> > > > On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <so...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Roman,
> > > > >
> > > > > With adminPath="/admin" or adminPath="/admin/cores", no.
> > Interestingly
> > > > > enough, though, I can access
> > > > > http://localhost:8983/solr/statements/admin/system
> > > > >
> > > > > But I can access http://localhost:8983/solr/admin/cores, only when
> > > with
> > > > > adminPath="/admin/cores" (which suggests that this is the right
> value
> > > to
> > > > be
> > > > > used for cores), and not with adminPath="/admin".
> > > > >
> > > > > Bottom line, these core configuration is not self-evident.
> > > > >
> > > > > Dmitry
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <
> roman.chyla@gmail.com
> > > > >wrote:
> > > > >
> > > > >> Hi Dmitry,
> > > > >> So it seems solrjmeter should not assume the adminPath - and
> perhaps
> > > > needs
> > > > >> to be passed as an argument. When you set the adminPath, are you
> > able
> > > to
> > > > >> access localhost:8983/solr/statements/admin/cores ?
> > > > >>
> > > > >> roman
> > > > >>
> > > > >>
> > > > >> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <solrexpert@gmail.com
> >
> > > > wrote:
> > > > >>
> > > > >> > Hi Roman,
> > > > >> >
> > > > >> > I have noticed a difference with different solr.xml config
> > contents.
> > > > It
> > > > >> is
> > > > >> > probably legit, but thought to let you know (tests run on fresh
> > > > >> checkout as
> > > > >> > of today).
> > > > >> >
> > > > >> > As mentioned before, I have two cores configured in solr.xml. If
> > the
> > > > >> file
> > > > >> > is:
> > > > >> >
> > > > >> > [code]
> > > > >> > <solr persistent="false">
> > > > >> >
> > > > >> > <!--
> > > > >> > adminPath: RequestHandler path to manage cores.
> > > > >> > If 'null' (or absent), cores will not be manageable via
> > request
> > > > >> handler
> > > > >> > -->
> > > > >> > <cores adminPath="/admin/cores" host="${host:}"
> > > > >> > hostPort="${jetty.port:8983}" hostContext="${hostContext:solr}">
> > > > >> > <core name="metadata" instanceDir="metadata" />
> > > > >> > <core name="statements" instanceDir="statements" />
> > > > >> > </cores>
> > > > >> > </solr>
> > > > >> > [/code]
> > > > >> >
> > > > >> > then the instruction:
> > > > >> >
> > > > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > --durationInSecs
> > > > 60
> > > > >> -R
> > > > >> > cms -t /solr/statements -e statements -U 100
> > > > >> >
> > > > >> > works just fine. If however the solr.xml has adminPath set to
> > > "/admin"
> > > > >> > solrjmeter produces an error:
> > > > >> >
> > > > >> > [error]
> > > > >> > **ERROR**
> > > > >> > File "solrjmeter.py", line 1386, in <module>
> > > > >> > main(sys.argv)
> > > > >> > File "solrjmeter.py", line 1278, in main
> > > > >> > check_prerequisities(options)
> > > > >> > File "solrjmeter.py", line 375, in check_prerequisities
> > > > >> > error('Cannot find admin pages: %s, please report a bug' %
> > > apath)
> > > > >> > File "solrjmeter.py", line 66, in error
> > > > >> > traceback.print_stack()
> > > > >> > Cannot find admin pages: http://localhost:8983/solr/admin,
> please
> > > > >> report a
> > > > >> > bug
> > > > >> > [/error]
> > > > >> >
> > > > >> > With both solr.xml configs the following url returns just fine:
> > > > >> >
> > > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > > >> >
> > > > >> > Regards,
> > > > >> >
> > > > >> > Dmitry
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <
> solrexpert@gmail.com
> > >
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi Roman,
> > > > >> > >
> > > > >> > > This looks much better, thanks! The ordinary non-comarison
> mode
> > > > works.
> > > > >> > > I'll post here, if there are other findings.
> > > > >> > >
> > > > >> > > Thanks for quick turnarounds,
> > > > >> > >
> > > > >> > > Dmitry
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <
> > > roman.chyla@gmail.com
> > > > >> > >wrote:
> > > > >> > >
> > > > >> > >> Hi Dmitry, oh yes, late night fixes... :) The latest commit
> > > should
> > > > >> make
> > > > >> > it
> > > > >> > >> work for you.
> > > > >> > >> Thanks!
> > > > >> > >>
> > > > >> > >> roman
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <
> > > solrexpert@gmail.com>
> > > > >> > wrote:
> > > > >> > >>
> > > > >> > >> > Hi Roman,
> > > > >> > >> >
> > > > >> > >> > Something bad happened in fresh checkout:
> > > > >> > >> >
> > > > >> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > > >> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > > >> --durationInSecs
> > > > >> > 60
> > > > >> > >> -R
> > > > >> > >> > cms -t /solr/statements -e statements -U 100
> > > > >> > >> >
> > > > >> > >> > Traceback (most recent call last):
> > > > >> > >> > File "solrjmeter.py", line 1392, in <module>
> > > > >> > >> > main(sys.argv)
> > > > >> > >> > File "solrjmeter.py", line 1347, in main
> > > > >> > >> > save_into_file('before-test.json',
> > > > >> simplejson.dumps(before_test))
> > > > >> > >> > File
> > > "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
> > > > >> line
> > > > >> > >> 286,
> > > > >> > >> > in dumps
> > > > >> > >> > return _default_encoder.encode(obj)
> > > > >> > >> > File
> > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > >> line
> > > > >> > >> 226,
> > > > >> > >> > in encode
> > > > >> > >> > chunks = self.iterencode(o, _one_shot=True)
> > > > >> > >> > File
> > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > >> line
> > > > >> > >> 296,
> > > > >> > >> > in iterencode
> > > > >> > >> > return _iterencode(o, 0)
> > > > >> > >> > File
> > > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > > >> line
> > > > >> > >> 202,
> > > > >> > >> > in default
> > > > >> > >> > raise TypeError(repr(o) + " is not JSON serializable")
> > > > >> > >> > TypeError: <__main__.ForgivingValue object at
> 0x7fc6d4040fd0>
> > > is
> > > > >> not
> > > > >> > >> JSON
> > > > >> > >> > serializable
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > Regards,
> > > > >> > >> >
> > > > >> > >> > D.
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
> > > > >> roman.chyla@gmail.com>
> > > > >> > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > > Hi Dmitry,
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
> > > > >> solrexpert@gmail.com>
> > > > >> > >> > wrote:
> > > > >> > >> > >
> > > > >> > >> > > > Hi Roman,
> > > > >> > >> > > >
> > > > >> > >> > > > Good point. I managed to run the command with -C and
> > double
> > > > >> > quotes:
> > > > >> > >> > > >
> > > > >> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour -x
> > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > >> > >> > > >
> > > > >> > >> > > > As a result got several files (html, css, js, csv) in
> the
> > > > >> running
> > > > >> > >> > > directory
> > > > >> > >> > > > (any way to specify where the output should be stored
> in
> > > this
> > > > >> > case?)
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> > > i know it is confusing, i plan to change it - but later,
> > now
> > > it
> > > > >> is
> > > > >> > too
> > > > >> > >> > busy
> > > > >> > >> > > here...
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > >
> > > > >> > >> > > > When I look onto the comparison dashboard, I see this:
> > > > >> > >> > > >
> > > > >> > >> > > > http://pbrd.co/17IRI0b
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> > > two things: the tests probably took more than one hour to
> > > > >> finish, so
> > > > >> > >> they
> > > > >> > >> > > are not aligned - try generating the comparison with '-c
> > > > 14400'
> > > > >> > (ie.
> > > > >> > >> > > 4x3600 secs)
> > > > >> > >> > >
> > > > >> > >> > > the other thing: if you have only two datapoints, the
> > dygraph
> > > > >> will
> > > > >> > not
> > > > >> > >> > show
> > > > >> > >> > > anything - there must be more datapoints/measurements
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > >
> > > > >> > >> > > > One more thing: all the previous tests were run with
> > > > softCommit
> > > > >> > >> > disabled.
> > > > >> > >> > > > After enabling it, the tests started to fail:
> > > > >> > >> > > >
> > > > >> > >> > > > $ python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > > >> > >> > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > > >> > >> --durationInSecs 60
> > > > >> > >> > > -R
> > > > >> > >> > > > g1 -t /solr/statements -e statements -U 100
> > > > >> > >> > > > $ cd g1
> > > > >> > >> > > > Reading results of the previous test
> > > > >> > >> > > > $ cd 2013.08.12.16.32.48
> > > > >> > >> > > > $ cd
> /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > > >> > >> > > > $ mkdir 2013.08.12.16.33.02
> > > > >> > >> > > > $ cd 2013.08.12.16.33.02
> > > > >> > >> > > > $ cd
> /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > > >> > >> > > > Traceback (most recent call last):
> > > > >> > >> > > > File "solrjmeter.py", line 1427, in <module>
> > > > >> > >> > > > main(sys.argv)
> > > > >> > >> > > > File "solrjmeter.py", line 1381, in main
> > > > >> > >> > > > before_test =
> > harvest_details_about_montysolr(options)
> > > > >> > >> > > > File "solrjmeter.py", line 562, in
> > > > >> > harvest_details_about_montysolr
> > > > >> > >> > > > indexLstModified =
> > > > >> > >> > cores_data['status'][cn]['index']['lastModified'],
> > > > >> > >> > > > KeyError: 'lastModified'
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> > > Thanks for letting me know, that info is probably not
> > > available
> > > > >> in
> > > > >> > >> this
> > > > >> > >> > > situation - i've cooked st quick to fix it, please try
> the
> > > > latest
> > > > >> > >> commit
> > > > >> > >> > > (hope it doesn't do more harm, i should get some sleep
> > ..;))
> > > > >> > >> > >
> > > > >> > >> > > roman
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > >
> > > > >> > >> > > > In case it matters: Python 2.7.3, ubuntu, solr 4.3.1.
> > > > >> > >> > > >
> > > > >> > >> > > > Thanks,
> > > > >> > >> > > >
> > > > >> > >> > > > Dmitry
> > > > >> > >> > > >
> > > > >> > >> > > >
> > > > >> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla <
> > > > >> > roman.chyla@gmail.com>
> > > > >> > >> > > wrote:
> > > > >> > >> > > >
> > > > >> > >> > > > > Hi Dmitry,
> > > > >> > >> > > > > The command seems good. Are you sure your shell is
> not
> > > > doing
> > > > >> > >> > something
> > > > >> > >> > > > > funny with the params? You could try:
> > > > >> > >> > > > >
> > > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > >> > >> > -a
> > > > >> > >> > > > >
> > > > >> > >> > > > > where g1 and foo are results of the individual runs,
> > ie.
> > > > >> > something
> > > > >> > >> > that
> > > > >> > >> > > > was
> > > > >> > >> > > > > started and saved with '-R g1' and '-R foo'
> > respectively
> > > > >> > >> > > > >
> > > > >> > >> > > > > so, for example, i have these comparisons inside
> > > > >> > >> > > > >
> > '/var/lib/montysolr/different-java-settings/solrjmeter',
> > > so
> > > > >> I am
> > > > >> > >> > > > generating
> > > > >> > >> > > > > the comparison by:
> > > > >> > >> > > > >
> > > > >> > >> > > > > export
> > > > >> > >> > > > >
> > > > >> > >>
> > > > SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
> > > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > >> > >> > -a
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > > roman
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry Kan <
> > > > >> > solrexpert@gmail.com
> > > > >> > >> >
> > > > >> > >> > > > wrote:
> > > > >> > >> > > > >
> > > > >> > >> > > > > > Hi Roman,
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > One more question. I tried to compare different
> runs
> > > (g1
> > > > vs
> > > > >> > cms)
> > > > >> > >> > > using
> > > > >> > >> > > > > the
> > > > >> > >> > > > > > command below, but get an error. Should I attach
> some
> > > > other
> > > > >> > >> > param(s)?
> > > > >> > >> > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > python solrjmeter.py -C g1,foo -c hour -x
> > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > >> > >> > > > > > **ERROR**
> > > > >> > >> > > > > > File "solrjmeter.py", line 1427, in <module>
> > > > >> > >> > > > > > main(sys.argv)
> > > > >> > >> > > > > > File "solrjmeter.py", line 1303, in main
> > > > >> > >> > > > > > check_options(options, args)
> > > > >> > >> > > > > > File "solrjmeter.py", line 185, in check_options
> > > > >> > >> > > > > > error("The folder '%s' does not exist" % rf)
> > > > >> > >> > > > > > File "solrjmeter.py", line 66, in error
> > > > >> > >> > > > > > traceback.print_stack()
> > > > >> > >> > > > > > The folder '0' does not exist
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > Dmitry
> > > > >> > >> > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM, Dmitry Kan <
> > > > >> > >> solrexpert@gmail.com>
> > > > >> > >> > > > wrote:
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > > Hi Roman,
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > Finally, this has worked! Thanks for quick
> support.
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > The graphs look awesome. At least on the index
> > sample
> > > > :)
> > > > >> It
> > > > >> > is
> > > > >> > >> > > quite
> > > > >> > >> > > > > easy
> > > > >> > >> > > > > > > to setup and run + possible to run directly on
> the
> > > > shard
> > > > >> > >> server
> > > > >> > >> > in
> > > > >> > >> > > > > > > background mode.
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > my test run was:
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > python solrjmeter.py -a -x
> ./jmx/SolrQueryTest.jmx
> > -q
> > > > >> > >> > > > > > > ./queries/demo/demo.queries -s localhost -p 8983
> -a
> > > > >> > >> > > --durationInSecs
> > > > >> > >> > > > 60
> > > > >> > >> > > > > > -R
> > > > >> > >> > > > > > > foo -t /solr/statements -e statements
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > Thanks!
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > Dmitry
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54 AM, Roman Chyla <
> > > > >> > >> > roman.chyla@gmail.com
> > > > >> > >> > > >
> > > > >> > >> > > > > > wrote:
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > >> Hi Dmitry,
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >> I've modified the solrjmeter to retrieve data
> from
> > > > under
> > > > >> > the
> > > > >> > >> > core
> > > > >> > >> > > > (the
> > > > >> > >> > > > > > -t
> > > > >> > >> > > > > > >> parameter) and the rest from the /solr/admin - I
> > > could
> > > > >> test
> > > > >> > >> it
> > > > >> > >> > > only
> > > > >> > >> > > > > > >> against
> > > > >> > >> > > > > > >> 4.0, but it is there the same as 4.3 - it
> > seems...so
> > > > you
> > > > >> > can
> > > > >> > >> try
> > > > >> > >> > > the
> > > > >> > >> > > > > > fresh
> > > > >> > >> > > > > > >> checkout
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >> my test was: python solrjmeter.py -a -x
> > > > >> > >> ./jmx/SolrQueryTest.jmx
> > > > >> > >> > -t
> > > > >> > >> > > > > > >> /solr/collection1 -R foo -q ./queries/demo/* -p
> > 9002
> > > > -s
> > > > >> > >> adsate
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >> Thanks!
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >> roman
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >> On Tue, Aug 6, 2013 at 9:46 AM, Dmitry Kan <
> > > > >> > >> > solrexpert@gmail.com>
> > > > >> > >> > > > > > wrote:
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >> > Hi,
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > Thanks for the clarification, Shawn!
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > So with this in mind, the following work:
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> >
> > > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > > >> > >> > > > > > >> >
> > > > >> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > not copying their output to save space.
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > Roman:
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > is this something that should be set via -t
> > > > parameter
> > > > >> as
> > > > >> > >> well?
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > Dmitry
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > On Tue, Aug 6, 2013 at 4:34 PM, Shawn Heisey <
> > > > >> > >> > solr@elyograg.org
> > > > >> > >> > > >
> > > > >> > >> > > > > > wrote:
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >> > > On 8/6/2013 6:17 AM, Dmitry Kan wrote:
> > > > >> > >> > > > > > >> > > > Of three URLs you asked for, only the 3rd
> > one
> > > > gave
> > > > >> > >> > response:
> > > > >> > >> > > > > > >> > > <snip>
> > > > >> > >> > > > > > >> > > > The rest report 404.
> > > > >> > >> > > > > > >> > > >
> > > > >> > >> > > > > > >> > > > On Mon, Aug 5, 2013 at 8:38 PM, Roman
> Chyla
> > <
> > > > >> > >> > > > > > roman.chyla@gmail.com>
> > > > >> > >> > > > > > >> > > wrote:
> > > > >> > >> > > > > > >> > > >
> > > > >> > >> > > > > > >> > > >> Hi Dmitry,
> > > > >> > >> > > > > > >> > > >> So I think the admin pages are different
> on
> > > > your
> > > > >> > >> version
> > > > >> > >> > of
> > > > >> > >> > > > > solr,
> > > > >> > >> > > > > > >> what
> > > > >> > >> > > > > > >> > > do
> > > > >> > >> > > > > > >> > > >> you see when you request... ?
> > > > >> > >> > > > > > >> > > >>
> > > > >> > >> > > > > > >> > > >>
> > > > http://localhost:8983/solr/admin/system?wt=json
> > > > >> > >> > > > > > >> > > >>
> > > > http://localhost:8983/solr/admin/mbeans?wt=json
> > > > >> > >> > > > > > >> > > >>
> > > http://localhost:8983/solr/admin/cores?wt=json
> > > > >> > >> > > > > > >> > >
> > > > >> > >> > > > > > >> > > Unless you have a valid defaultCoreName set
> in
> > > > your
> > > > >> > >> > > (old-style)
> > > > >> > >> > > > > > >> > > solr.xml, the first two URLs won't work, as
> > > you've
> > > > >> > >> > discovered.
> > > > >> > >> > > > > > >> Without
> > > > >> > >> > > > > > >> > > that valid defaultCoreName (or if you wanted
> > > info
> > > > >> from
> > > > >> > a
> > > > >> > >> > > > different
> > > > >> > >> > > > > > >> > > core), you'd need to add a core name to the
> > URL
> > > > for
> > > > >> > them
> > > > >> > >> to
> > > > >> > >> > > > work.
> > > > >> > >> > > > > > >> > >
> > > > >> > >> > > > > > >> > > The third one, which works for you, is a
> > global
> > > > >> handler
> > > > >> > >> for
> > > > >> > >> > > > > > >> manipulating
> > > > >> > >> > > > > > >> > > cores, so naturally it doesn't need a core
> > name
> > > to
> > > > >> > >> function.
> > > > >> > >> > > > The
> > > > >> > >> > > > > > URL
> > > > >> > >> > > > > > >> > > path for this handler is defined by
> solr.xml.
> > > > >> > >> > > > > > >> > >
> > > > >> > >> > > > > > >> > > Thanks,
> > > > >> > >> > > > > > >> > > Shawn
> > > > >> > >> > > > > > >> > >
> > > > >> > >> > > > > > >> > >
> > > > >> > >> > > > > > >> >
> > > > >> > >> > > > > > >>
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Measuring SOLR performance
Posted by Roman Chyla <ro...@gmail.com>.
Hi Dmitry,
If it is something you want to pass with every request (which is my use
case), you can pass it as additional solr params, eg.
python solrjmeter
--additionalSolrParams="fq=other_field:bar+facet=true+facet.field=facet_field_name"
....
the string should be url encoded.
If it is something that changes with every request, you should modify the
jmeter test. If you open/load it with jmeter GUI, in the HTTP request
processor you can define other additional fields to pass with the request.
These values can come from the CSV file, you'll see an example how to use
that when you open the test difinition file.
Cheers,
roman
On Mon, Sep 2, 2013 at 3:12 PM, Dmitry Kan <so...@gmail.com> wrote:
> Hi Erick,
>
> Agree, this is perfectly fine to mix them in solr. But my question is about
> solrjmeter input query format. Just couldn't find a suitable example on the
> solrjmeter's github.
>
> Dmitry
>
>
>
> On Mon, Sep 2, 2013 at 5:40 PM, Erick Erickson <erickerickson@gmail.com
> >wrote:
>
> > filter and facet queries can be freely intermixed, it's not a problem.
> > What problem are you seeing when you try this?
> >
> > Best,
> > Erick
> >
> >
> > On Mon, Sep 2, 2013 at 7:46 AM, Dmitry Kan <so...@gmail.com> wrote:
> >
> > > Hi Roman,
> > >
> > > What's the format for running the facet+filter queries?
> > >
> > > Would something like this work:
> > >
> > > field:foo >=50 fq=other_field:bar facet=true
> > facet.field=facet_field_name
> > >
> > >
> > > Thanks,
> > > Dmitry
> > >
> > >
> > >
> > > On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <so...@gmail.com>
> > wrote:
> > >
> > > > Hi Roman,
> > > >
> > > > With adminPath="/admin" or adminPath="/admin/cores", no.
> Interestingly
> > > > enough, though, I can access
> > > > http://localhost:8983/solr/statements/admin/system
> > > >
> > > > But I can access http://localhost:8983/solr/admin/cores, only when
> > with
> > > > adminPath="/admin/cores" (which suggests that this is the right value
> > to
> > > be
> > > > used for cores), and not with adminPath="/admin".
> > > >
> > > > Bottom line, these core configuration is not self-evident.
> > > >
> > > > Dmitry
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <roman.chyla@gmail.com
> > > >wrote:
> > > >
> > > >> Hi Dmitry,
> > > >> So it seems solrjmeter should not assume the adminPath - and perhaps
> > > needs
> > > >> to be passed as an argument. When you set the adminPath, are you
> able
> > to
> > > >> access localhost:8983/solr/statements/admin/cores ?
> > > >>
> > > >> roman
> > > >>
> > > >>
> > > >> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <so...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi Roman,
> > > >> >
> > > >> > I have noticed a difference with different solr.xml config
> contents.
> > > It
> > > >> is
> > > >> > probably legit, but thought to let you know (tests run on fresh
> > > >> checkout as
> > > >> > of today).
> > > >> >
> > > >> > As mentioned before, I have two cores configured in solr.xml. If
> the
> > > >> file
> > > >> > is:
> > > >> >
> > > >> > [code]
> > > >> > <solr persistent="false">
> > > >> >
> > > >> > <!--
> > > >> > adminPath: RequestHandler path to manage cores.
> > > >> > If 'null' (or absent), cores will not be manageable via
> request
> > > >> handler
> > > >> > -->
> > > >> > <cores adminPath="/admin/cores" host="${host:}"
> > > >> > hostPort="${jetty.port:8983}" hostContext="${hostContext:solr}">
> > > >> > <core name="metadata" instanceDir="metadata" />
> > > >> > <core name="statements" instanceDir="statements" />
> > > >> > </cores>
> > > >> > </solr>
> > > >> > [/code]
> > > >> >
> > > >> > then the instruction:
> > > >> >
> > > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> --durationInSecs
> > > 60
> > > >> -R
> > > >> > cms -t /solr/statements -e statements -U 100
> > > >> >
> > > >> > works just fine. If however the solr.xml has adminPath set to
> > "/admin"
> > > >> > solrjmeter produces an error:
> > > >> >
> > > >> > [error]
> > > >> > **ERROR**
> > > >> > File "solrjmeter.py", line 1386, in <module>
> > > >> > main(sys.argv)
> > > >> > File "solrjmeter.py", line 1278, in main
> > > >> > check_prerequisities(options)
> > > >> > File "solrjmeter.py", line 375, in check_prerequisities
> > > >> > error('Cannot find admin pages: %s, please report a bug' %
> > apath)
> > > >> > File "solrjmeter.py", line 66, in error
> > > >> > traceback.print_stack()
> > > >> > Cannot find admin pages: http://localhost:8983/solr/admin, please
> > > >> report a
> > > >> > bug
> > > >> > [/error]
> > > >> >
> > > >> > With both solr.xml configs the following url returns just fine:
> > > >> >
> > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > >> >
> > > >> > Regards,
> > > >> >
> > > >> > Dmitry
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <solrexpert@gmail.com
> >
> > > >> wrote:
> > > >> >
> > > >> > > Hi Roman,
> > > >> > >
> > > >> > > This looks much better, thanks! The ordinary non-comarison mode
> > > works.
> > > >> > > I'll post here, if there are other findings.
> > > >> > >
> > > >> > > Thanks for quick turnarounds,
> > > >> > >
> > > >> > > Dmitry
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <
> > roman.chyla@gmail.com
> > > >> > >wrote:
> > > >> > >
> > > >> > >> Hi Dmitry, oh yes, late night fixes... :) The latest commit
> > should
> > > >> make
> > > >> > it
> > > >> > >> work for you.
> > > >> > >> Thanks!
> > > >> > >>
> > > >> > >> roman
> > > >> > >>
> > > >> > >>
> > > >> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <
> > solrexpert@gmail.com>
> > > >> > wrote:
> > > >> > >>
> > > >> > >> > Hi Roman,
> > > >> > >> >
> > > >> > >> > Something bad happened in fresh checkout:
> > > >> > >> >
> > > >> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > >> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > >> --durationInSecs
> > > >> > 60
> > > >> > >> -R
> > > >> > >> > cms -t /solr/statements -e statements -U 100
> > > >> > >> >
> > > >> > >> > Traceback (most recent call last):
> > > >> > >> > File "solrjmeter.py", line 1392, in <module>
> > > >> > >> > main(sys.argv)
> > > >> > >> > File "solrjmeter.py", line 1347, in main
> > > >> > >> > save_into_file('before-test.json',
> > > >> simplejson.dumps(before_test))
> > > >> > >> > File
> > "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
> > > >> line
> > > >> > >> 286,
> > > >> > >> > in dumps
> > > >> > >> > return _default_encoder.encode(obj)
> > > >> > >> > File
> > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > >> line
> > > >> > >> 226,
> > > >> > >> > in encode
> > > >> > >> > chunks = self.iterencode(o, _one_shot=True)
> > > >> > >> > File
> > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > >> line
> > > >> > >> 296,
> > > >> > >> > in iterencode
> > > >> > >> > return _iterencode(o, 0)
> > > >> > >> > File
> > "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > > >> line
> > > >> > >> 202,
> > > >> > >> > in default
> > > >> > >> > raise TypeError(repr(o) + " is not JSON serializable")
> > > >> > >> > TypeError: <__main__.ForgivingValue object at 0x7fc6d4040fd0>
> > is
> > > >> not
> > > >> > >> JSON
> > > >> > >> > serializable
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > Regards,
> > > >> > >> >
> > > >> > >> > D.
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
> > > >> roman.chyla@gmail.com>
> > > >> > >> > wrote:
> > > >> > >> >
> > > >> > >> > > Hi Dmitry,
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
> > > >> solrexpert@gmail.com>
> > > >> > >> > wrote:
> > > >> > >> > >
> > > >> > >> > > > Hi Roman,
> > > >> > >> > > >
> > > >> > >> > > > Good point. I managed to run the command with -C and
> double
> > > >> > quotes:
> > > >> > >> > > >
> > > >> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour -x
> > > >> > >> ./jmx/SolrQueryTest.jmx
> > > >> > >> > > >
> > > >> > >> > > > As a result got several files (html, css, js, csv) in the
> > > >> running
> > > >> > >> > > directory
> > > >> > >> > > > (any way to specify where the output should be stored in
> > this
> > > >> > case?)
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> > > i know it is confusing, i plan to change it - but later,
> now
> > it
> > > >> is
> > > >> > too
> > > >> > >> > busy
> > > >> > >> > > here...
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > >
> > > >> > >> > > > When I look onto the comparison dashboard, I see this:
> > > >> > >> > > >
> > > >> > >> > > > http://pbrd.co/17IRI0b
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> > > two things: the tests probably took more than one hour to
> > > >> finish, so
> > > >> > >> they
> > > >> > >> > > are not aligned - try generating the comparison with '-c
> > > 14400'
> > > >> > (ie.
> > > >> > >> > > 4x3600 secs)
> > > >> > >> > >
> > > >> > >> > > the other thing: if you have only two datapoints, the
> dygraph
> > > >> will
> > > >> > not
> > > >> > >> > show
> > > >> > >> > > anything - there must be more datapoints/measurements
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > >
> > > >> > >> > > > One more thing: all the previous tests were run with
> > > softCommit
> > > >> > >> > disabled.
> > > >> > >> > > > After enabling it, the tests started to fail:
> > > >> > >> > > >
> > > >> > >> > > > $ python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > > >> > >> > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > >> > >> --durationInSecs 60
> > > >> > >> > > -R
> > > >> > >> > > > g1 -t /solr/statements -e statements -U 100
> > > >> > >> > > > $ cd g1
> > > >> > >> > > > Reading results of the previous test
> > > >> > >> > > > $ cd 2013.08.12.16.32.48
> > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > >> > >> > > > $ mkdir 2013.08.12.16.33.02
> > > >> > >> > > > $ cd 2013.08.12.16.33.02
> > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > > >> > >> > > > Traceback (most recent call last):
> > > >> > >> > > > File "solrjmeter.py", line 1427, in <module>
> > > >> > >> > > > main(sys.argv)
> > > >> > >> > > > File "solrjmeter.py", line 1381, in main
> > > >> > >> > > > before_test =
> harvest_details_about_montysolr(options)
> > > >> > >> > > > File "solrjmeter.py", line 562, in
> > > >> > harvest_details_about_montysolr
> > > >> > >> > > > indexLstModified =
> > > >> > >> > cores_data['status'][cn]['index']['lastModified'],
> > > >> > >> > > > KeyError: 'lastModified'
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> > > Thanks for letting me know, that info is probably not
> > available
> > > >> in
> > > >> > >> this
> > > >> > >> > > situation - i've cooked st quick to fix it, please try the
> > > latest
> > > >> > >> commit
> > > >> > >> > > (hope it doesn't do more harm, i should get some sleep
> ..;))
> > > >> > >> > >
> > > >> > >> > > roman
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > >
> > > >> > >> > > > In case it matters: Python 2.7.3, ubuntu, solr 4.3.1.
> > > >> > >> > > >
> > > >> > >> > > > Thanks,
> > > >> > >> > > >
> > > >> > >> > > > Dmitry
> > > >> > >> > > >
> > > >> > >> > > >
> > > >> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla <
> > > >> > roman.chyla@gmail.com>
> > > >> > >> > > wrote:
> > > >> > >> > > >
> > > >> > >> > > > > Hi Dmitry,
> > > >> > >> > > > > The command seems good. Are you sure your shell is not
> > > doing
> > > >> > >> > something
> > > >> > >> > > > > funny with the params? You could try:
> > > >> > >> > > > >
> > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > >> > >> ./jmx/SolrQueryTest.jmx
> > > >> > >> > -a
> > > >> > >> > > > >
> > > >> > >> > > > > where g1 and foo are results of the individual runs,
> ie.
> > > >> > something
> > > >> > >> > that
> > > >> > >> > > > was
> > > >> > >> > > > > started and saved with '-R g1' and '-R foo'
> respectively
> > > >> > >> > > > >
> > > >> > >> > > > > so, for example, i have these comparisons inside
> > > >> > >> > > > >
> '/var/lib/montysolr/different-java-settings/solrjmeter',
> > so
> > > >> I am
> > > >> > >> > > > generating
> > > >> > >> > > > > the comparison by:
> > > >> > >> > > > >
> > > >> > >> > > > > export
> > > >> > >> > > > >
> > > >> > >>
> > > SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
> > > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > > >> > >> ./jmx/SolrQueryTest.jmx
> > > >> > >> > -a
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > > roman
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry Kan <
> > > >> > solrexpert@gmail.com
> > > >> > >> >
> > > >> > >> > > > wrote:
> > > >> > >> > > > >
> > > >> > >> > > > > > Hi Roman,
> > > >> > >> > > > > >
> > > >> > >> > > > > > One more question. I tried to compare different runs
> > (g1
> > > vs
> > > >> > cms)
> > > >> > >> > > using
> > > >> > >> > > > > the
> > > >> > >> > > > > > command below, but get an error. Should I attach some
> > > other
> > > >> > >> > param(s)?
> > > >> > >> > > > > >
> > > >> > >> > > > > >
> > > >> > >> > > > > > python solrjmeter.py -C g1,foo -c hour -x
> > > >> > >> ./jmx/SolrQueryTest.jmx
> > > >> > >> > > > > > **ERROR**
> > > >> > >> > > > > > File "solrjmeter.py", line 1427, in <module>
> > > >> > >> > > > > > main(sys.argv)
> > > >> > >> > > > > > File "solrjmeter.py", line 1303, in main
> > > >> > >> > > > > > check_options(options, args)
> > > >> > >> > > > > > File "solrjmeter.py", line 185, in check_options
> > > >> > >> > > > > > error("The folder '%s' does not exist" % rf)
> > > >> > >> > > > > > File "solrjmeter.py", line 66, in error
> > > >> > >> > > > > > traceback.print_stack()
> > > >> > >> > > > > > The folder '0' does not exist
> > > >> > >> > > > > >
> > > >> > >> > > > > > Dmitry
> > > >> > >> > > > > >
> > > >> > >> > > > > >
> > > >> > >> > > > > >
> > > >> > >> > > > > >
> > > >> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM, Dmitry Kan <
> > > >> > >> solrexpert@gmail.com>
> > > >> > >> > > > wrote:
> > > >> > >> > > > > >
> > > >> > >> > > > > > > Hi Roman,
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > Finally, this has worked! Thanks for quick support.
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > The graphs look awesome. At least on the index
> sample
> > > :)
> > > >> It
> > > >> > is
> > > >> > >> > > quite
> > > >> > >> > > > > easy
> > > >> > >> > > > > > > to setup and run + possible to run directly on the
> > > shard
> > > >> > >> server
> > > >> > >> > in
> > > >> > >> > > > > > > background mode.
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > my test run was:
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx
> -q
> > > >> > >> > > > > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > > >> > >> > > --durationInSecs
> > > >> > >> > > > 60
> > > >> > >> > > > > > -R
> > > >> > >> > > > > > > foo -t /solr/statements -e statements
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > Thanks!
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > Dmitry
> > > >> > >> > > > > > >
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54 AM, Roman Chyla <
> > > >> > >> > roman.chyla@gmail.com
> > > >> > >> > > >
> > > >> > >> > > > > > wrote:
> > > >> > >> > > > > > >
> > > >> > >> > > > > > >> Hi Dmitry,
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >> I've modified the solrjmeter to retrieve data from
> > > under
> > > >> > the
> > > >> > >> > core
> > > >> > >> > > > (the
> > > >> > >> > > > > > -t
> > > >> > >> > > > > > >> parameter) and the rest from the /solr/admin - I
> > could
> > > >> test
> > > >> > >> it
> > > >> > >> > > only
> > > >> > >> > > > > > >> against
> > > >> > >> > > > > > >> 4.0, but it is there the same as 4.3 - it
> seems...so
> > > you
> > > >> > can
> > > >> > >> try
> > > >> > >> > > the
> > > >> > >> > > > > > fresh
> > > >> > >> > > > > > >> checkout
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >> my test was: python solrjmeter.py -a -x
> > > >> > >> ./jmx/SolrQueryTest.jmx
> > > >> > >> > -t
> > > >> > >> > > > > > >> /solr/collection1 -R foo -q ./queries/demo/* -p
> 9002
> > > -s
> > > >> > >> adsate
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >> Thanks!
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >> roman
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >> On Tue, Aug 6, 2013 at 9:46 AM, Dmitry Kan <
> > > >> > >> > solrexpert@gmail.com>
> > > >> > >> > > > > > wrote:
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >> > Hi,
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > Thanks for the clarification, Shawn!
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > So with this in mind, the following work:
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> >
> > > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > > >> > >> > > > > > >> >
> > > >> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > not copying their output to save space.
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > Roman:
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > is this something that should be set via -t
> > > parameter
> > > >> as
> > > >> > >> well?
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > Dmitry
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > On Tue, Aug 6, 2013 at 4:34 PM, Shawn Heisey <
> > > >> > >> > solr@elyograg.org
> > > >> > >> > > >
> > > >> > >> > > > > > wrote:
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >> > > On 8/6/2013 6:17 AM, Dmitry Kan wrote:
> > > >> > >> > > > > > >> > > > Of three URLs you asked for, only the 3rd
> one
> > > gave
> > > >> > >> > response:
> > > >> > >> > > > > > >> > > <snip>
> > > >> > >> > > > > > >> > > > The rest report 404.
> > > >> > >> > > > > > >> > > >
> > > >> > >> > > > > > >> > > > On Mon, Aug 5, 2013 at 8:38 PM, Roman Chyla
> <
> > > >> > >> > > > > > roman.chyla@gmail.com>
> > > >> > >> > > > > > >> > > wrote:
> > > >> > >> > > > > > >> > > >
> > > >> > >> > > > > > >> > > >> Hi Dmitry,
> > > >> > >> > > > > > >> > > >> So I think the admin pages are different on
> > > your
> > > >> > >> version
> > > >> > >> > of
> > > >> > >> > > > > solr,
> > > >> > >> > > > > > >> what
> > > >> > >> > > > > > >> > > do
> > > >> > >> > > > > > >> > > >> you see when you request... ?
> > > >> > >> > > > > > >> > > >>
> > > >> > >> > > > > > >> > > >>
> > > http://localhost:8983/solr/admin/system?wt=json
> > > >> > >> > > > > > >> > > >>
> > > http://localhost:8983/solr/admin/mbeans?wt=json
> > > >> > >> > > > > > >> > > >>
> > http://localhost:8983/solr/admin/cores?wt=json
> > > >> > >> > > > > > >> > >
> > > >> > >> > > > > > >> > > Unless you have a valid defaultCoreName set in
> > > your
> > > >> > >> > > (old-style)
> > > >> > >> > > > > > >> > > solr.xml, the first two URLs won't work, as
> > you've
> > > >> > >> > discovered.
> > > >> > >> > > > > > >> Without
> > > >> > >> > > > > > >> > > that valid defaultCoreName (or if you wanted
> > info
> > > >> from
> > > >> > a
> > > >> > >> > > > different
> > > >> > >> > > > > > >> > > core), you'd need to add a core name to the
> URL
> > > for
> > > >> > them
> > > >> > >> to
> > > >> > >> > > > work.
> > > >> > >> > > > > > >> > >
> > > >> > >> > > > > > >> > > The third one, which works for you, is a
> global
> > > >> handler
> > > >> > >> for
> > > >> > >> > > > > > >> manipulating
> > > >> > >> > > > > > >> > > cores, so naturally it doesn't need a core
> name
> > to
> > > >> > >> function.
> > > >> > >> > > > The
> > > >> > >> > > > > > URL
> > > >> > >> > > > > > >> > > path for this handler is defined by solr.xml.
> > > >> > >> > > > > > >> > >
> > > >> > >> > > > > > >> > > Thanks,
> > > >> > >> > > > > > >> > > Shawn
> > > >> > >> > > > > > >> > >
> > > >> > >> > > > > > >> > >
> > > >> > >> > > > > > >> >
> > > >> > >> > > > > > >>
> > > >> > >> > > > > > >
> > > >> > >> > > > > > >
> > > >> > >> > > > > >
> > > >> > >> > > > >
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Measuring SOLR performance
Posted by Dmitry Kan <so...@gmail.com>.
Hi Erick,
Agree, this is perfectly fine to mix them in solr. But my question is about
solrjmeter input query format. Just couldn't find a suitable example on the
solrjmeter's github.
Dmitry
On Mon, Sep 2, 2013 at 5:40 PM, Erick Erickson <er...@gmail.com>wrote:
> filter and facet queries can be freely intermixed, it's not a problem.
> What problem are you seeing when you try this?
>
> Best,
> Erick
>
>
> On Mon, Sep 2, 2013 at 7:46 AM, Dmitry Kan <so...@gmail.com> wrote:
>
> > Hi Roman,
> >
> > What's the format for running the facet+filter queries?
> >
> > Would something like this work:
> >
> > field:foo >=50 fq=other_field:bar facet=true
> facet.field=facet_field_name
> >
> >
> > Thanks,
> > Dmitry
> >
> >
> >
> > On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <so...@gmail.com>
> wrote:
> >
> > > Hi Roman,
> > >
> > > With adminPath="/admin" or adminPath="/admin/cores", no. Interestingly
> > > enough, though, I can access
> > > http://localhost:8983/solr/statements/admin/system
> > >
> > > But I can access http://localhost:8983/solr/admin/cores, only when
> with
> > > adminPath="/admin/cores" (which suggests that this is the right value
> to
> > be
> > > used for cores), and not with adminPath="/admin".
> > >
> > > Bottom line, these core configuration is not self-evident.
> > >
> > > Dmitry
> > >
> > >
> > >
> > >
> > > On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <roman.chyla@gmail.com
> > >wrote:
> > >
> > >> Hi Dmitry,
> > >> So it seems solrjmeter should not assume the adminPath - and perhaps
> > needs
> > >> to be passed as an argument. When you set the adminPath, are you able
> to
> > >> access localhost:8983/solr/statements/admin/cores ?
> > >>
> > >> roman
> > >>
> > >>
> > >> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <so...@gmail.com>
> > wrote:
> > >>
> > >> > Hi Roman,
> > >> >
> > >> > I have noticed a difference with different solr.xml config contents.
> > It
> > >> is
> > >> > probably legit, but thought to let you know (tests run on fresh
> > >> checkout as
> > >> > of today).
> > >> >
> > >> > As mentioned before, I have two cores configured in solr.xml. If the
> > >> file
> > >> > is:
> > >> >
> > >> > [code]
> > >> > <solr persistent="false">
> > >> >
> > >> > <!--
> > >> > adminPath: RequestHandler path to manage cores.
> > >> > If 'null' (or absent), cores will not be manageable via request
> > >> handler
> > >> > -->
> > >> > <cores adminPath="/admin/cores" host="${host:}"
> > >> > hostPort="${jetty.port:8983}" hostContext="${hostContext:solr}">
> > >> > <core name="metadata" instanceDir="metadata" />
> > >> > <core name="statements" instanceDir="statements" />
> > >> > </cores>
> > >> > </solr>
> > >> > [/code]
> > >> >
> > >> > then the instruction:
> > >> >
> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a --durationInSecs
> > 60
> > >> -R
> > >> > cms -t /solr/statements -e statements -U 100
> > >> >
> > >> > works just fine. If however the solr.xml has adminPath set to
> "/admin"
> > >> > solrjmeter produces an error:
> > >> >
> > >> > [error]
> > >> > **ERROR**
> > >> > File "solrjmeter.py", line 1386, in <module>
> > >> > main(sys.argv)
> > >> > File "solrjmeter.py", line 1278, in main
> > >> > check_prerequisities(options)
> > >> > File "solrjmeter.py", line 375, in check_prerequisities
> > >> > error('Cannot find admin pages: %s, please report a bug' %
> apath)
> > >> > File "solrjmeter.py", line 66, in error
> > >> > traceback.print_stack()
> > >> > Cannot find admin pages: http://localhost:8983/solr/admin, please
> > >> report a
> > >> > bug
> > >> > [/error]
> > >> >
> > >> > With both solr.xml configs the following url returns just fine:
> > >> >
> > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > >> >
> > >> > Regards,
> > >> >
> > >> > Dmitry
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <so...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi Roman,
> > >> > >
> > >> > > This looks much better, thanks! The ordinary non-comarison mode
> > works.
> > >> > > I'll post here, if there are other findings.
> > >> > >
> > >> > > Thanks for quick turnarounds,
> > >> > >
> > >> > > Dmitry
> > >> > >
> > >> > >
> > >> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <
> roman.chyla@gmail.com
> > >> > >wrote:
> > >> > >
> > >> > >> Hi Dmitry, oh yes, late night fixes... :) The latest commit
> should
> > >> make
> > >> > it
> > >> > >> work for you.
> > >> > >> Thanks!
> > >> > >>
> > >> > >> roman
> > >> > >>
> > >> > >>
> > >> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <
> solrexpert@gmail.com>
> > >> > wrote:
> > >> > >>
> > >> > >> > Hi Roman,
> > >> > >> >
> > >> > >> > Something bad happened in fresh checkout:
> > >> > >> >
> > >> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > >> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > >> --durationInSecs
> > >> > 60
> > >> > >> -R
> > >> > >> > cms -t /solr/statements -e statements -U 100
> > >> > >> >
> > >> > >> > Traceback (most recent call last):
> > >> > >> > File "solrjmeter.py", line 1392, in <module>
> > >> > >> > main(sys.argv)
> > >> > >> > File "solrjmeter.py", line 1347, in main
> > >> > >> > save_into_file('before-test.json',
> > >> simplejson.dumps(before_test))
> > >> > >> > File
> "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
> > >> line
> > >> > >> 286,
> > >> > >> > in dumps
> > >> > >> > return _default_encoder.encode(obj)
> > >> > >> > File
> "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > >> line
> > >> > >> 226,
> > >> > >> > in encode
> > >> > >> > chunks = self.iterencode(o, _one_shot=True)
> > >> > >> > File
> "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > >> line
> > >> > >> 296,
> > >> > >> > in iterencode
> > >> > >> > return _iterencode(o, 0)
> > >> > >> > File
> "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> > >> line
> > >> > >> 202,
> > >> > >> > in default
> > >> > >> > raise TypeError(repr(o) + " is not JSON serializable")
> > >> > >> > TypeError: <__main__.ForgivingValue object at 0x7fc6d4040fd0>
> is
> > >> not
> > >> > >> JSON
> > >> > >> > serializable
> > >> > >> >
> > >> > >> >
> > >> > >> > Regards,
> > >> > >> >
> > >> > >> > D.
> > >> > >> >
> > >> > >> >
> > >> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
> > >> roman.chyla@gmail.com>
> > >> > >> > wrote:
> > >> > >> >
> > >> > >> > > Hi Dmitry,
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
> > >> solrexpert@gmail.com>
> > >> > >> > wrote:
> > >> > >> > >
> > >> > >> > > > Hi Roman,
> > >> > >> > > >
> > >> > >> > > > Good point. I managed to run the command with -C and double
> > >> > quotes:
> > >> > >> > > >
> > >> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour -x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > > >
> > >> > >> > > > As a result got several files (html, css, js, csv) in the
> > >> running
> > >> > >> > > directory
> > >> > >> > > > (any way to specify where the output should be stored in
> this
> > >> > case?)
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > > i know it is confusing, i plan to change it - but later, now
> it
> > >> is
> > >> > too
> > >> > >> > busy
> > >> > >> > > here...
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > When I look onto the comparison dashboard, I see this:
> > >> > >> > > >
> > >> > >> > > > http://pbrd.co/17IRI0b
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > > two things: the tests probably took more than one hour to
> > >> finish, so
> > >> > >> they
> > >> > >> > > are not aligned - try generating the comparison with '-c
> > 14400'
> > >> > (ie.
> > >> > >> > > 4x3600 secs)
> > >> > >> > >
> > >> > >> > > the other thing: if you have only two datapoints, the dygraph
> > >> will
> > >> > not
> > >> > >> > show
> > >> > >> > > anything - there must be more datapoints/measurements
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > One more thing: all the previous tests were run with
> > softCommit
> > >> > >> > disabled.
> > >> > >> > > > After enabling it, the tests started to fail:
> > >> > >> > > >
> > >> > >> > > > $ python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > >> > >> > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > >> > >> --durationInSecs 60
> > >> > >> > > -R
> > >> > >> > > > g1 -t /solr/statements -e statements -U 100
> > >> > >> > > > $ cd g1
> > >> > >> > > > Reading results of the previous test
> > >> > >> > > > $ cd 2013.08.12.16.32.48
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > >> > >> > > > $ mkdir 2013.08.12.16.33.02
> > >> > >> > > > $ cd 2013.08.12.16.33.02
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> > >> > >> > > > Traceback (most recent call last):
> > >> > >> > > > File "solrjmeter.py", line 1427, in <module>
> > >> > >> > > > main(sys.argv)
> > >> > >> > > > File "solrjmeter.py", line 1381, in main
> > >> > >> > > > before_test = harvest_details_about_montysolr(options)
> > >> > >> > > > File "solrjmeter.py", line 562, in
> > >> > harvest_details_about_montysolr
> > >> > >> > > > indexLstModified =
> > >> > >> > cores_data['status'][cn]['index']['lastModified'],
> > >> > >> > > > KeyError: 'lastModified'
> > >> > >> > > >
> > >> > >> > >
> > >> > >> > > Thanks for letting me know, that info is probably not
> available
> > >> in
> > >> > >> this
> > >> > >> > > situation - i've cooked st quick to fix it, please try the
> > latest
> > >> > >> commit
> > >> > >> > > (hope it doesn't do more harm, i should get some sleep ..;))
> > >> > >> > >
> > >> > >> > > roman
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > >
> > >> > >> > > > In case it matters: Python 2.7.3, ubuntu, solr 4.3.1.
> > >> > >> > > >
> > >> > >> > > > Thanks,
> > >> > >> > > >
> > >> > >> > > > Dmitry
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla <
> > >> > roman.chyla@gmail.com>
> > >> > >> > > wrote:
> > >> > >> > > >
> > >> > >> > > > > Hi Dmitry,
> > >> > >> > > > > The command seems good. Are you sure your shell is not
> > doing
> > >> > >> > something
> > >> > >> > > > > funny with the params? You could try:
> > >> > >> > > > >
> > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > -a
> > >> > >> > > > >
> > >> > >> > > > > where g1 and foo are results of the individual runs, ie.
> > >> > something
> > >> > >> > that
> > >> > >> > > > was
> > >> > >> > > > > started and saved with '-R g1' and '-R foo' respectively
> > >> > >> > > > >
> > >> > >> > > > > so, for example, i have these comparisons inside
> > >> > >> > > > > '/var/lib/montysolr/different-java-settings/solrjmeter',
> so
> > >> I am
> > >> > >> > > > generating
> > >> > >> > > > > the comparison by:
> > >> > >> > > > >
> > >> > >> > > > > export
> > >> > >> > > > >
> > >> > >>
> > SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
> > >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > -a
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > roman
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry Kan <
> > >> > solrexpert@gmail.com
> > >> > >> >
> > >> > >> > > > wrote:
> > >> > >> > > > >
> > >> > >> > > > > > Hi Roman,
> > >> > >> > > > > >
> > >> > >> > > > > > One more question. I tried to compare different runs
> (g1
> > vs
> > >> > cms)
> > >> > >> > > using
> > >> > >> > > > > the
> > >> > >> > > > > > command below, but get an error. Should I attach some
> > other
> > >> > >> > param(s)?
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > python solrjmeter.py -C g1,foo -c hour -x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > > > > > **ERROR**
> > >> > >> > > > > > File "solrjmeter.py", line 1427, in <module>
> > >> > >> > > > > > main(sys.argv)
> > >> > >> > > > > > File "solrjmeter.py", line 1303, in main
> > >> > >> > > > > > check_options(options, args)
> > >> > >> > > > > > File "solrjmeter.py", line 185, in check_options
> > >> > >> > > > > > error("The folder '%s' does not exist" % rf)
> > >> > >> > > > > > File "solrjmeter.py", line 66, in error
> > >> > >> > > > > > traceback.print_stack()
> > >> > >> > > > > > The folder '0' does not exist
> > >> > >> > > > > >
> > >> > >> > > > > > Dmitry
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM, Dmitry Kan <
> > >> > >> solrexpert@gmail.com>
> > >> > >> > > > wrote:
> > >> > >> > > > > >
> > >> > >> > > > > > > Hi Roman,
> > >> > >> > > > > > >
> > >> > >> > > > > > > Finally, this has worked! Thanks for quick support.
> > >> > >> > > > > > >
> > >> > >> > > > > > > The graphs look awesome. At least on the index sample
> > :)
> > >> It
> > >> > is
> > >> > >> > > quite
> > >> > >> > > > > easy
> > >> > >> > > > > > > to setup and run + possible to run directly on the
> > shard
> > >> > >> server
> > >> > >> > in
> > >> > >> > > > > > > background mode.
> > >> > >> > > > > > >
> > >> > >> > > > > > > my test run was:
> > >> > >> > > > > > >
> > >> > >> > > > > > > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> > >> > >> > > > > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> > >> > >> > > --durationInSecs
> > >> > >> > > > 60
> > >> > >> > > > > > -R
> > >> > >> > > > > > > foo -t /solr/statements -e statements
> > >> > >> > > > > > >
> > >> > >> > > > > > > Thanks!
> > >> > >> > > > > > >
> > >> > >> > > > > > > Dmitry
> > >> > >> > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54 AM, Roman Chyla <
> > >> > >> > roman.chyla@gmail.com
> > >> > >> > > >
> > >> > >> > > > > > wrote:
> > >> > >> > > > > > >
> > >> > >> > > > > > >> Hi Dmitry,
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> I've modified the solrjmeter to retrieve data from
> > under
> > >> > the
> > >> > >> > core
> > >> > >> > > > (the
> > >> > >> > > > > > -t
> > >> > >> > > > > > >> parameter) and the rest from the /solr/admin - I
> could
> > >> test
> > >> > >> it
> > >> > >> > > only
> > >> > >> > > > > > >> against
> > >> > >> > > > > > >> 4.0, but it is there the same as 4.3 - it seems...so
> > you
> > >> > can
> > >> > >> try
> > >> > >> > > the
> > >> > >> > > > > > fresh
> > >> > >> > > > > > >> checkout
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> my test was: python solrjmeter.py -a -x
> > >> > >> ./jmx/SolrQueryTest.jmx
> > >> > >> > -t
> > >> > >> > > > > > >> /solr/collection1 -R foo -q ./queries/demo/* -p 9002
> > -s
> > >> > >> adsate
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> Thanks!
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> roman
> > >> > >> > > > > > >>
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> On Tue, Aug 6, 2013 at 9:46 AM, Dmitry Kan <
> > >> > >> > solrexpert@gmail.com>
> > >> > >> > > > > > wrote:
> > >> > >> > > > > > >>
> > >> > >> > > > > > >> > Hi,
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > Thanks for the clarification, Shawn!
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > So with this in mind, the following work:
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> >
> > >> > http://localhost:8983/solr/statements/admin/system?wt=json
> > >> > >> > > > > > >> >
> > >> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > not copying their output to save space.
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > Roman:
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > is this something that should be set via -t
> > parameter
> > >> as
> > >> > >> well?
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > Dmitry
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > On Tue, Aug 6, 2013 at 4:34 PM, Shawn Heisey <
> > >> > >> > solr@elyograg.org
> > >> > >> > > >
> > >> > >> > > > > > wrote:
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >> > > On 8/6/2013 6:17 AM, Dmitry Kan wrote:
> > >> > >> > > > > > >> > > > Of three URLs you asked for, only the 3rd one
> > gave
> > >> > >> > response:
> > >> > >> > > > > > >> > > <snip>
> > >> > >> > > > > > >> > > > The rest report 404.
> > >> > >> > > > > > >> > > >
> > >> > >> > > > > > >> > > > On Mon, Aug 5, 2013 at 8:38 PM, Roman Chyla <
> > >> > >> > > > > > roman.chyla@gmail.com>
> > >> > >> > > > > > >> > > wrote:
> > >> > >> > > > > > >> > > >
> > >> > >> > > > > > >> > > >> Hi Dmitry,
> > >> > >> > > > > > >> > > >> So I think the admin pages are different on
> > your
> > >> > >> version
> > >> > >> > of
> > >> > >> > > > > solr,
> > >> > >> > > > > > >> what
> > >> > >> > > > > > >> > > do
> > >> > >> > > > > > >> > > >> you see when you request... ?
> > >> > >> > > > > > >> > > >>
> > >> > >> > > > > > >> > > >>
> > http://localhost:8983/solr/admin/system?wt=json
> > >> > >> > > > > > >> > > >>
> > http://localhost:8983/solr/admin/mbeans?wt=json
> > >> > >> > > > > > >> > > >>
> http://localhost:8983/solr/admin/cores?wt=json
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > > Unless you have a valid defaultCoreName set in
> > your
> > >> > >> > > (old-style)
> > >> > >> > > > > > >> > > solr.xml, the first two URLs won't work, as
> you've
> > >> > >> > discovered.
> > >> > >> > > > > > >> Without
> > >> > >> > > > > > >> > > that valid defaultCoreName (or if you wanted
> info
> > >> from
> > >> > a
> > >> > >> > > > different
> > >> > >> > > > > > >> > > core), you'd need to add a core name to the URL
> > for
> > >> > them
> > >> > >> to
> > >> > >> > > > work.
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > > The third one, which works for you, is a global
> > >> handler
> > >> > >> for
> > >> > >> > > > > > >> manipulating
> > >> > >> > > > > > >> > > cores, so naturally it doesn't need a core name
> to
> > >> > >> function.
> > >> > >> > > > The
> > >> > >> > > > > > URL
> > >> > >> > > > > > >> > > path for this handler is defined by solr.xml.
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > > Thanks,
> > >> > >> > > > > > >> > > Shawn
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> > >
> > >> > >> > > > > > >> >
> > >> > >> > > > > > >>
> > >> > >> > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > >
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Measuring SOLR performance
Posted by Erick Erickson <er...@gmail.com>.
filter and facet queries can be freely intermixed, it's not a problem.
What problem are you seeing when you try this?
Best,
Erick
On Mon, Sep 2, 2013 at 7:46 AM, Dmitry Kan <so...@gmail.com> wrote:
> Hi Roman,
>
> What's the format for running the facet+filter queries?
>
> Would something like this work:
>
> field:foo >=50 fq=other_field:bar facet=true facet.field=facet_field_name
>
>
> Thanks,
> Dmitry
>
>
>
> On Fri, Aug 23, 2013 at 2:34 PM, Dmitry Kan <so...@gmail.com> wrote:
>
> > Hi Roman,
> >
> > With adminPath="/admin" or adminPath="/admin/cores", no. Interestingly
> > enough, though, I can access
> > http://localhost:8983/solr/statements/admin/system
> >
> > But I can access http://localhost:8983/solr/admin/cores, only when with
> > adminPath="/admin/cores" (which suggests that this is the right value to
> be
> > used for cores), and not with adminPath="/admin".
> >
> > Bottom line, these core configuration is not self-evident.
> >
> > Dmitry
> >
> >
> >
> >
> > On Fri, Aug 23, 2013 at 4:18 AM, Roman Chyla <roman.chyla@gmail.com
> >wrote:
> >
> >> Hi Dmitry,
> >> So it seems solrjmeter should not assume the adminPath - and perhaps
> needs
> >> to be passed as an argument. When you set the adminPath, are you able to
> >> access localhost:8983/solr/statements/admin/cores ?
> >>
> >> roman
> >>
> >>
> >> On Wed, Aug 21, 2013 at 7:36 AM, Dmitry Kan <so...@gmail.com>
> wrote:
> >>
> >> > Hi Roman,
> >> >
> >> > I have noticed a difference with different solr.xml config contents.
> It
> >> is
> >> > probably legit, but thought to let you know (tests run on fresh
> >> checkout as
> >> > of today).
> >> >
> >> > As mentioned before, I have two cores configured in solr.xml. If the
> >> file
> >> > is:
> >> >
> >> > [code]
> >> > <solr persistent="false">
> >> >
> >> > <!--
> >> > adminPath: RequestHandler path to manage cores.
> >> > If 'null' (or absent), cores will not be manageable via request
> >> handler
> >> > -->
> >> > <cores adminPath="/admin/cores" host="${host:}"
> >> > hostPort="${jetty.port:8983}" hostContext="${hostContext:solr}">
> >> > <core name="metadata" instanceDir="metadata" />
> >> > <core name="statements" instanceDir="statements" />
> >> > </cores>
> >> > </solr>
> >> > [/code]
> >> >
> >> > then the instruction:
> >> >
> >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> >> > ./queries/demo/demo.queries -s localhost -p 8983 -a --durationInSecs
> 60
> >> -R
> >> > cms -t /solr/statements -e statements -U 100
> >> >
> >> > works just fine. If however the solr.xml has adminPath set to "/admin"
> >> > solrjmeter produces an error:
> >> >
> >> > [error]
> >> > **ERROR**
> >> > File "solrjmeter.py", line 1386, in <module>
> >> > main(sys.argv)
> >> > File "solrjmeter.py", line 1278, in main
> >> > check_prerequisities(options)
> >> > File "solrjmeter.py", line 375, in check_prerequisities
> >> > error('Cannot find admin pages: %s, please report a bug' % apath)
> >> > File "solrjmeter.py", line 66, in error
> >> > traceback.print_stack()
> >> > Cannot find admin pages: http://localhost:8983/solr/admin, please
> >> report a
> >> > bug
> >> > [/error]
> >> >
> >> > With both solr.xml configs the following url returns just fine:
> >> >
> >> > http://localhost:8983/solr/statements/admin/system?wt=json
> >> >
> >> > Regards,
> >> >
> >> > Dmitry
> >> >
> >> >
> >> >
> >> > On Wed, Aug 14, 2013 at 2:03 PM, Dmitry Kan <so...@gmail.com>
> >> wrote:
> >> >
> >> > > Hi Roman,
> >> > >
> >> > > This looks much better, thanks! The ordinary non-comarison mode
> works.
> >> > > I'll post here, if there are other findings.
> >> > >
> >> > > Thanks for quick turnarounds,
> >> > >
> >> > > Dmitry
> >> > >
> >> > >
> >> > > On Wed, Aug 14, 2013 at 1:32 AM, Roman Chyla <roman.chyla@gmail.com
> >> > >wrote:
> >> > >
> >> > >> Hi Dmitry, oh yes, late night fixes... :) The latest commit should
> >> make
> >> > it
> >> > >> work for you.
> >> > >> Thanks!
> >> > >>
> >> > >> roman
> >> > >>
> >> > >>
> >> > >> On Tue, Aug 13, 2013 at 3:37 AM, Dmitry Kan <so...@gmail.com>
> >> > wrote:
> >> > >>
> >> > >> > Hi Roman,
> >> > >> >
> >> > >> > Something bad happened in fresh checkout:
> >> > >> >
> >> > >> > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> >> > >> > ./queries/demo/demo.queries -s localhost -p 8983 -a
> >> --durationInSecs
> >> > 60
> >> > >> -R
> >> > >> > cms -t /solr/statements -e statements -U 100
> >> > >> >
> >> > >> > Traceback (most recent call last):
> >> > >> > File "solrjmeter.py", line 1392, in <module>
> >> > >> > main(sys.argv)
> >> > >> > File "solrjmeter.py", line 1347, in main
> >> > >> > save_into_file('before-test.json',
> >> simplejson.dumps(before_test))
> >> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py",
> >> line
> >> > >> 286,
> >> > >> > in dumps
> >> > >> > return _default_encoder.encode(obj)
> >> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> >> line
> >> > >> 226,
> >> > >> > in encode
> >> > >> > chunks = self.iterencode(o, _one_shot=True)
> >> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> >> line
> >> > >> 296,
> >> > >> > in iterencode
> >> > >> > return _iterencode(o, 0)
> >> > >> > File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py",
> >> line
> >> > >> 202,
> >> > >> > in default
> >> > >> > raise TypeError(repr(o) + " is not JSON serializable")
> >> > >> > TypeError: <__main__.ForgivingValue object at 0x7fc6d4040fd0> is
> >> not
> >> > >> JSON
> >> > >> > serializable
> >> > >> >
> >> > >> >
> >> > >> > Regards,
> >> > >> >
> >> > >> > D.
> >> > >> >
> >> > >> >
> >> > >> > On Tue, Aug 13, 2013 at 8:10 AM, Roman Chyla <
> >> roman.chyla@gmail.com>
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Hi Dmitry,
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > On Mon, Aug 12, 2013 at 9:36 AM, Dmitry Kan <
> >> solrexpert@gmail.com>
> >> > >> > wrote:
> >> > >> > >
> >> > >> > > > Hi Roman,
> >> > >> > > >
> >> > >> > > > Good point. I managed to run the command with -C and double
> >> > quotes:
> >> > >> > > >
> >> > >> > > > python solrjmeter.py -a -C "g1,cms" -c hour -x
> >> > >> ./jmx/SolrQueryTest.jmx
> >> > >> > > >
> >> > >> > > > As a result got several files (html, css, js, csv) in the
> >> running
> >> > >> > > directory
> >> > >> > > > (any way to specify where the output should be stored in this
> >> > case?)
> >> > >> > > >
> >> > >> > >
> >> > >> > > i know it is confusing, i plan to change it - but later, now it
> >> is
> >> > too
> >> > >> > busy
> >> > >> > > here...
> >> > >> > >
> >> > >> > >
> >> > >> > > >
> >> > >> > > > When I look onto the comparison dashboard, I see this:
> >> > >> > > >
> >> > >> > > > http://pbrd.co/17IRI0b
> >> > >> > > >
> >> > >> > >
> >> > >> > > two things: the tests probably took more than one hour to
> >> finish, so
> >> > >> they
> >> > >> > > are not aligned - try generating the comparison with '-c
> 14400'
> >> > (ie.
> >> > >> > > 4x3600 secs)
> >> > >> > >
> >> > >> > > the other thing: if you have only two datapoints, the dygraph
> >> will
> >> > not
> >> > >> > show
> >> > >> > > anything - there must be more datapoints/measurements
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > >
> >> > >> > > > One more thing: all the previous tests were run with
> softCommit
> >> > >> > disabled.
> >> > >> > > > After enabling it, the tests started to fail:
> >> > >> > > >
> >> > >> > > > $ python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> >> > >> > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> >> > >> --durationInSecs 60
> >> > >> > > -R
> >> > >> > > > g1 -t /solr/statements -e statements -U 100
> >> > >> > > > $ cd g1
> >> > >> > > > Reading results of the previous test
> >> > >> > > > $ cd 2013.08.12.16.32.48
> >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> >> > >> > > > $ mkdir 2013.08.12.16.33.02
> >> > >> > > > $ cd 2013.08.12.16.33.02
> >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter/g1
> >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> >> > >> > > > $ cd /home/dmitry/projects/lab/solrjmeter4/solrjmeter
> >> > >> > > > Traceback (most recent call last):
> >> > >> > > > File "solrjmeter.py", line 1427, in <module>
> >> > >> > > > main(sys.argv)
> >> > >> > > > File "solrjmeter.py", line 1381, in main
> >> > >> > > > before_test = harvest_details_about_montysolr(options)
> >> > >> > > > File "solrjmeter.py", line 562, in
> >> > harvest_details_about_montysolr
> >> > >> > > > indexLstModified =
> >> > >> > cores_data['status'][cn]['index']['lastModified'],
> >> > >> > > > KeyError: 'lastModified'
> >> > >> > > >
> >> > >> > >
> >> > >> > > Thanks for letting me know, that info is probably not available
> >> in
> >> > >> this
> >> > >> > > situation - i've cooked st quick to fix it, please try the
> latest
> >> > >> commit
> >> > >> > > (hope it doesn't do more harm, i should get some sleep ..;))
> >> > >> > >
> >> > >> > > roman
> >> > >> > >
> >> > >> > >
> >> > >> > > >
> >> > >> > > > In case it matters: Python 2.7.3, ubuntu, solr 4.3.1.
> >> > >> > > >
> >> > >> > > > Thanks,
> >> > >> > > >
> >> > >> > > > Dmitry
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Thu, Aug 8, 2013 at 2:22 AM, Roman Chyla <
> >> > roman.chyla@gmail.com>
> >> > >> > > wrote:
> >> > >> > > >
> >> > >> > > > > Hi Dmitry,
> >> > >> > > > > The command seems good. Are you sure your shell is not
> doing
> >> > >> > something
> >> > >> > > > > funny with the params? You could try:
> >> > >> > > > >
> >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> >> > >> ./jmx/SolrQueryTest.jmx
> >> > >> > -a
> >> > >> > > > >
> >> > >> > > > > where g1 and foo are results of the individual runs, ie.
> >> > something
> >> > >> > that
> >> > >> > > > was
> >> > >> > > > > started and saved with '-R g1' and '-R foo' respectively
> >> > >> > > > >
> >> > >> > > > > so, for example, i have these comparisons inside
> >> > >> > > > > '/var/lib/montysolr/different-java-settings/solrjmeter', so
> >> I am
> >> > >> > > > generating
> >> > >> > > > > the comparison by:
> >> > >> > > > >
> >> > >> > > > > export
> >> > >> > > > >
> >> > >>
> SOLRJMETER_HOME=/var/lib/montysolr/different-java-settings/solrjmeter
> >> > >> > > > > python solrjmeter.py -C "g1,foo" -c hour -x
> >> > >> ./jmx/SolrQueryTest.jmx
> >> > >> > -a
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > roman
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > On Wed, Aug 7, 2013 at 10:03 AM, Dmitry Kan <
> >> > solrexpert@gmail.com
> >> > >> >
> >> > >> > > > wrote:
> >> > >> > > > >
> >> > >> > > > > > Hi Roman,
> >> > >> > > > > >
> >> > >> > > > > > One more question. I tried to compare different runs (g1
> vs
> >> > cms)
> >> > >> > > using
> >> > >> > > > > the
> >> > >> > > > > > command below, but get an error. Should I attach some
> other
> >> > >> > param(s)?
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > python solrjmeter.py -C g1,foo -c hour -x
> >> > >> ./jmx/SolrQueryTest.jmx
> >> > >> > > > > > **ERROR**
> >> > >> > > > > > File "solrjmeter.py", line 1427, in <module>
> >> > >> > > > > > main(sys.argv)
> >> > >> > > > > > File "solrjmeter.py", line 1303, in main
> >> > >> > > > > > check_options(options, args)
> >> > >> > > > > > File "solrjmeter.py", line 185, in check_options
> >> > >> > > > > > error("The folder '%s' does not exist" % rf)
> >> > >> > > > > > File "solrjmeter.py", line 66, in error
> >> > >> > > > > > traceback.print_stack()
> >> > >> > > > > > The folder '0' does not exist
> >> > >> > > > > >
> >> > >> > > > > > Dmitry
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > On Wed, Aug 7, 2013 at 4:13 PM, Dmitry Kan <
> >> > >> solrexpert@gmail.com>
> >> > >> > > > wrote:
> >> > >> > > > > >
> >> > >> > > > > > > Hi Roman,
> >> > >> > > > > > >
> >> > >> > > > > > > Finally, this has worked! Thanks for quick support.
> >> > >> > > > > > >
> >> > >> > > > > > > The graphs look awesome. At least on the index sample
> :)
> >> It
> >> > is
> >> > >> > > quite
> >> > >> > > > > easy
> >> > >> > > > > > > to setup and run + possible to run directly on the
> shard
> >> > >> server
> >> > >> > in
> >> > >> > > > > > > background mode.
> >> > >> > > > > > >
> >> > >> > > > > > > my test run was:
> >> > >> > > > > > >
> >> > >> > > > > > > python solrjmeter.py -a -x ./jmx/SolrQueryTest.jmx -q
> >> > >> > > > > > > ./queries/demo/demo.queries -s localhost -p 8983 -a
> >> > >> > > --durationInSecs
> >> > >> > > > 60
> >> > >> > > > > > -R
> >> > >> > > > > > > foo -t /solr/statements -e statements
> >> > >> > > > > > >
> >> > >> > > > > > > Thanks!
> >> > >> > > > > > >
> >> > >> > > > > > > Dmitry
> >> > >> > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > > > On Wed, Aug 7, 2013 at 6:54 AM, Roman Chyla <
> >> > >> > roman.chyla@gmail.com
> >> > >> > > >
> >> > >> > > > > > wrote:
> >> > >> > > > > > >
> >> > >> > > > > > >> Hi Dmitry,
> >> > >> > > > > > >>
> >> > >> > > > > > >> I've modified the solrjmeter to retrieve data from
> under
> >> > the
> >> > >> > core
> >> > >> > > > (the
> >> > >> > > > > > -t
> >> > >> > > > > > >> parameter) and the rest from the /solr/admin - I could
> >> test
> >> > >> it
> >> > >> > > only
> >> > >> > > > > > >> against
> >> > >> > > > > > >> 4.0, but it is there the same as 4.3 - it seems...so
> you
> >> > can
> >> > >> try
> >> > >> > > the
> >> > >> > > > > > fresh
> >> > >> > > > > > >> checkout
> >> > >> > > > > > >>
> >> > >> > > > > > >> my test was: python solrjmeter.py -a -x
> >> > >> ./jmx/SolrQueryTest.jmx
> >> > >> > -t
> >> > >> > > > > > >> /solr/collection1 -R foo -q ./queries/demo/* -p 9002
> -s
> >> > >> adsate
> >> > >> > > > > > >>
> >> > >> > > > > > >> Thanks!
> >> > >> > > > > > >>
> >> > >> > > > > > >> roman
> >> > >> > > > > > >>
> >> > >> > > > > > >>
> >> > >> > > > > > >> On Tue, Aug 6, 2013 at 9:46 AM, Dmitry Kan <
> >> > >> > solrexpert@gmail.com>
> >> > >> > > > > > wrote:
> >> > >> > > > > > >>
> >> > >> > > > > > >> > Hi,
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > Thanks for the clarification, Shawn!
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > So with this in mind, the following work:
> >> > >> > > > > > >> >
> >> > >> > > > > > >> >
> >> > http://localhost:8983/solr/statements/admin/system?wt=json
> >> > >> > > > > > >> >
> >> > http://localhost:8983/solr/statements/admin/mbeans?wt=json
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > not copying their output to save space.
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > Roman:
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > is this something that should be set via -t
> parameter
> >> as
> >> > >> well?
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > Dmitry
> >> > >> > > > > > >> >
> >> > >> > > > > > >> >
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > On Tue, Aug 6, 2013 at 4:34 PM, Shawn Heisey <
> >> > >> > solr@elyograg.org
> >> > >> > > >
> >> > >> > > > > > wrote:
> >> > >> > > > > > >> >
> >> > >> > > > > > >> > > On 8/6/2013 6:17 AM, Dmitry Kan wrote:
> >> > >> > > > > > >> > > > Of three URLs you asked for, only the 3rd one
> gave
> >> > >> > response:
> >> > >> > > > > > >> > > <snip>
> >> > >> > > > > > >> > > > The rest report 404.
> >> > >> > > > > > >> > > >
> >> > >> > > > > > >> > > > On Mon, Aug 5, 2013 at 8:38 PM, Roman Chyla <
> >> > >> > > > > > roman.chyla@gmail.com>
> >> > >> > > > > > >> > > wrote:
> >> > >> > > > > > >> > > >
> >> > >> > > > > > >> > > >> Hi Dmitry,
> >> > >> > > > > > >> > > >> So I think the admin pages are different on
> your
> >> > >> version
> >> > >> > of
> >> > >> > > > > solr,
> >> > >> > > > > > >> what
> >> > >> > > > > > >> > > do
> >> > >> > > > > > >> > > >> you see when you request... ?
> >> > >> > > > > > >> > > >>
> >> > >> > > > > > >> > > >>
> http://localhost:8983/solr/admin/system?wt=json
> >> > >> > > > > > >> > > >>
> http://localhost:8983/solr/admin/mbeans?wt=json
> >> > >> > > > > > >> > > >> http://localhost:8983/solr/admin/cores?wt=json
> >> > >> > > > > > >> > >
> >> > >> > > > > > >> > > Unless you have a valid defaultCoreName set in
> your
> >> > >> > > (old-style)
> >> > >> > > > > > >> > > solr.xml, the first two URLs won't work, as you've
> >> > >> > discovered.
> >> > >> > > > > > >> Without
> >> > >> > > > > > >> > > that valid defaultCoreName (or if you wanted info
> >> from
> >> > a
> >> > >> > > > different
> >> > >> > > > > > >> > > core), you'd need to add a core name to the URL
> for
> >> > them
> >> > >> to
> >> > >> > > > work.
> >> > >> > > > > > >> > >
> >> > >> > > > > > >> > > The third one, which works for you, is a global
> >> handler
> >> > >> for
> >> > >> > > > > > >> manipulating
> >> > >> > > > > > >> > > cores, so naturally it doesn't need a core name to
> >> > >> function.
> >> > >> > > > The
> >> > >> > > > > > URL
> >> > >> > > > > > >> > > path for this handler is defined by solr.xml.
> >> > >> > > > > > >> > >
> >> > >> > > > > > >> > > Thanks,
> >> > >> > > > > > >> > > Shawn
> >> > >> > > > > > >> > >
> >> > >> > > > > > >> > >
> >> > >> > > > > > >> >
> >> > >> > > > > > >>
> >> > >> > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>