You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by con <co...@gmail.com> on 2008/10/10 14:02:46 UTC
Solr indexing not taking all values from DB.
A simple question:
I performed the following steps to index data from a oracle db to solr index
and then search:
a) I have the configurations for indexing data from a oracle db
b) started the server.
c) Done a full-import:
http://localhost:8983/solr/dataimport?command=full-import
But when I do a search using http://localhost:8983/solr/select/?q=....
Not all the result sets that matches the search string are displayed.
1) Is the above steps enough for getting db values to solr index?
My configurations (data-config.xml and schema.xml )are quite correct because
I am getting SOME of the result sets as search result(not all).
2) Is there some value in sorconfig.xml, or some other files that limits the
number of items being indexed? [For the time being I have only a few
hundreds of records in my db. ]
The query that I am specifying in data-config yields around 25 results if i
execute it in a oracle client, where as the status of full-import is
something like:
<str name="status">idle</str>
<str name="importResponse">Configuration Re-loaded sucessfully</str>
<lst name="statusMessages">
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">10</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-10-10 17:29:03</str>
<str name="Time taken ">0:0:0.513</str>
</lst>
--
View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr indexing not taking all values from DB.
Posted by con <co...@gmail.com>.
Hi Jon
I have my unique key set to false.
<uniqueKey required="false">EMP_ID</uniqueKey>.
I tried with enabling the unique key. Still the same problem exists.
Thanks
con
jonbaer wrote:
>
> What is your <uniqueKey> set to? Could it be you have duplicates in
> your uniqueKey setup (thus producing only 10 rows in index)?
>
> - Jon
>
> On Oct 12, 2008, at 1:30 PM, con wrote:
>
>>
>> I wrote a jdbc program to implement the same query. But it is
>> returning all
>> the responses, 25 nos.
>> But the solr is still indexing only 10 rows.
>> Is there any optimization settings by default in the solrconfig.xml
>> that
>> restricts the responses to 10 ?
>> thanks
>> con.
>>
>>
>>
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> template transformer does not eat up rows.
>>>
>>> I am almost sure that the query returns only 10 rows in that case.
>>> could you write a quick jdbc program and verify that (not the oralce
>>> client)
>>>
>>> everything else looks fine
>>>
>>> On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>>>>
>>>> Hi Noble
>>>> Thanks for your reply
>>>>
>>>> In my data-config.xml I have;
>>>>
>>>> <entity name="employees" transformer="TemplateTransformer"
>>>> query="Select
>>>> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>>> EMPLOYEES">
>>>> <field column="rowtype" template="employees" />
>>>> <field column="EMP_ID" name="EMP_ID" />
>>>> <field column="EMP_NAME" name="EMP_NAME" />
>>>> <field column="COMMENT" name="COMMENT" />
>>>> </entity>
>>>>
>>>> <entity name="customers" transformer="TemplateTransformer"
>>>> query="Select
>>>> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>>> CUSTOMERS">
>>>> <field column="rowtype" template="customers" />
>>>> <field column="CUST_ID" name="CUST_ID />
>>>> <field column="CUST_NAME" name="CUST_NAME" />
>>>> <field column="COMMENT" name="COMMENT" />
>>>> </entity>
>>>>
>>>> Whether this, TemplateTransformer, is the one that is restricting
>>>> the
>>>> resultset count to 10?
>>>> Where can I find it out?
>>>> I need this TemplateTransformer because I want to query the
>>>> responses of
>>>> either one of these at a time using the URL like,
>>>>
>>>> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>>>>
>>>> I tried in the debug mode:
>>>> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on
>>>> )
>>>> , But it is not all mentioning anything after the 10th document.
>>>>
>>>>
>>>> Thanks and regards
>>>> con
>>>>
>>>>
>>>>
>>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>>
>>>>> The DIH status says 10 rows which means only 10 rows got fetched
>>>>> for
>>>>> that query. Do you have any custom transformers which eats up rows?
>>>>>
>>>>> Try the debug page of DIH and see what is happening to the rest
>>>>> of the
>>>>> rows.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>>>>
>>>>>> A simple question:
>>>>>> I performed the following steps to index data from a oracle db
>>>>>> to solr
>>>>>> index
>>>>>> and then search:
>>>>>> a) I have the configurations for indexing data from a oracle db
>>>>>> b) started the server.
>>>>>> c) Done a full-import:
>>>>>> http://localhost:8983/solr/dataimport?command=full-import
>>>>>>
>>>>>> But when I do a search using http://localhost:8983/solr/select/?
>>>>>> q=....
>>>>>> Not all the result sets that matches the search string are
>>>>>> displayed.
>>>>>>
>>>>>> 1) Is the above steps enough for getting db values to solr index?
>>>>>> My configurations (data-config.xml and schema.xml )are quite
>>>>>> correct
>>>>>> because
>>>>>> I am getting SOME of the result sets as search result(not all).
>>>>>> 2) Is there some value in sorconfig.xml, or some other files that
>>>>>> limits
>>>>>> the
>>>>>> number of items being indexed? [For the time being I have only a
>>>>>> few
>>>>>> hundreds of records in my db. ]
>>>>>> The query that I am specifying in data-config yields around 25
>>>>>> results
>>>>>> if
>>>>>> i
>>>>>> execute it in a oracle client, where as the status of full-
>>>>>> import is
>>>>>> something like:
>>>>>> <str name="status">idle</str>
>>>>>> <str name="importResponse">Configuration Re-loaded sucessfully</
>>>>>> str>
>>>>>> <lst name="statusMessages">
>>>>>> <str name="Total Requests made to DataSource">1</str>
>>>>>> <str name="Total Rows Fetched">10</str>
>>>>>> <str name="Total Documents Skipped">0</str>
>>>>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>>>>> <str name="Time taken ">0:0:0.513</str>
>>>>>> </lst>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19943817.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19948508.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr indexing not taking all values from DB.
Posted by Jon Baer <jo...@gmail.com>.
What is your <uniqueKey> set to? Could it be you have duplicates in
your uniqueKey setup (thus producing only 10 rows in index)?
- Jon
On Oct 12, 2008, at 1:30 PM, con wrote:
>
> I wrote a jdbc program to implement the same query. But it is
> returning all
> the responses, 25 nos.
> But the solr is still indexing only 10 rows.
> Is there any optimization settings by default in the solrconfig.xml
> that
> restricts the responses to 10 ?
> thanks
> con.
>
>
>
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> template transformer does not eat up rows.
>>
>> I am almost sure that the query returns only 10 rows in that case.
>> could you write a quick jdbc program and verify that (not the oralce
>> client)
>>
>> everything else looks fine
>>
>> On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>>>
>>> Hi Noble
>>> Thanks for your reply
>>>
>>> In my data-config.xml I have;
>>>
>>> <entity name="employees" transformer="TemplateTransformer"
>>> query="Select
>>> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>> EMPLOYEES">
>>> <field column="rowtype" template="employees" />
>>> <field column="EMP_ID" name="EMP_ID" />
>>> <field column="EMP_NAME" name="EMP_NAME" />
>>> <field column="COMMENT" name="COMMENT" />
>>> </entity>
>>>
>>> <entity name="customers" transformer="TemplateTransformer"
>>> query="Select
>>> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>> CUSTOMERS">
>>> <field column="rowtype" template="customers" />
>>> <field column="CUST_ID" name="CUST_ID />
>>> <field column="CUST_NAME" name="CUST_NAME" />
>>> <field column="COMMENT" name="COMMENT" />
>>> </entity>
>>>
>>> Whether this, TemplateTransformer, is the one that is restricting
>>> the
>>> resultset count to 10?
>>> Where can I find it out?
>>> I need this TemplateTransformer because I want to query the
>>> responses of
>>> either one of these at a time using the URL like,
>>>
>>> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>>>
>>> I tried in the debug mode:
>>> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on
>>> )
>>> , But it is not all mentioning anything after the 10th document.
>>>
>>>
>>> Thanks and regards
>>> con
>>>
>>>
>>>
>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>
>>>> The DIH status says 10 rows which means only 10 rows got fetched
>>>> for
>>>> that query. Do you have any custom transformers which eats up rows?
>>>>
>>>> Try the debug page of DIH and see what is happening to the rest
>>>> of the
>>>> rows.
>>>>
>>>>
>>>>
>>>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>>>
>>>>> A simple question:
>>>>> I performed the following steps to index data from a oracle db
>>>>> to solr
>>>>> index
>>>>> and then search:
>>>>> a) I have the configurations for indexing data from a oracle db
>>>>> b) started the server.
>>>>> c) Done a full-import:
>>>>> http://localhost:8983/solr/dataimport?command=full-import
>>>>>
>>>>> But when I do a search using http://localhost:8983/solr/select/?
>>>>> q=....
>>>>> Not all the result sets that matches the search string are
>>>>> displayed.
>>>>>
>>>>> 1) Is the above steps enough for getting db values to solr index?
>>>>> My configurations (data-config.xml and schema.xml )are quite
>>>>> correct
>>>>> because
>>>>> I am getting SOME of the result sets as search result(not all).
>>>>> 2) Is there some value in sorconfig.xml, or some other files that
>>>>> limits
>>>>> the
>>>>> number of items being indexed? [For the time being I have only a
>>>>> few
>>>>> hundreds of records in my db. ]
>>>>> The query that I am specifying in data-config yields around 25
>>>>> results
>>>>> if
>>>>> i
>>>>> execute it in a oracle client, where as the status of full-
>>>>> import is
>>>>> something like:
>>>>> <str name="status">idle</str>
>>>>> <str name="importResponse">Configuration Re-loaded sucessfully</
>>>>> str>
>>>>> <lst name="statusMessages">
>>>>> <str name="Total Requests made to DataSource">1</str>
>>>>> <str name="Total Rows Fetched">10</str>
>>>>> <str name="Total Documents Skipped">0</str>
>>>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>>>> <str name="Time taken ">0:0:0.513</str>
>>>>> </lst>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19943817.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Solr indexing not taking all values from DB.
Posted by con <co...@gmail.com>.
Hi Noble,
Thanks for your reply
Sorry that I was not able to reply back in time.
I followed your suggestions and it is now indexing fine up to an extend.
1) if the query is taking values from a single table, the full import will
show the expected count itself.
And when i do
http://localhost:8983/solr/select/?q=(*:*%20AND%20rowtype:customers)&version=2.2&start=0&rows=100&indent=on&wt=json
it will display all the results.
2) if the query is taking values from multiple table(select * from EMPLOYEE,
CUSTOMER where EMPLOYEE.pro_id= CUSTOMER.prod_id) the full import will show
the expected count itself.
But, When I do
http://localhost:8983/solr/select/?q=(*:*%20AND%20rowtype:both)&version=2.2&start=0&rows=100&indent=on&wt=json
it is not displaying all the values.
Since for full import, <str name="Total Rows Fetched">**</str> is getting as
expected in both cases, i suppose the indexing is successfully done. Then
why my search doesn't yield all the values in the second case?
Thanks and regards
con
Noble Paul നോബിള് नोब्ळ् wrote:
>
> now just do a normal full-import do not enable debug . I guess it
> should be just fine
>
> On Mon, Oct 13, 2008 at 1:20 PM, con <co...@gmail.com> wrote:
>>
>> Thanks Nobble
>> I tried in the debug mode with rows=100 and it is accepting all the
>> result
>> sets.
>> So i suppose there is nothing wrong in the query.
>> But I am not able to update the index since this is available only in the
>> debug mode.
>>
>> Can you please give some suggestions based on this.
>>
>> thanks
>> con
>>
>>
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> in debug mode it writes only 10 because there is a rows parameter
>>> which is by default set to 10
>>> make it 100 or so and you should be seeing all docs. But in non-debug
>>> mode there is no such parameter
>>>
>>> On Sun, Oct 12, 2008 at 11:00 PM, con <co...@gmail.com> wrote:
>>>>
>>>> I wrote a jdbc program to implement the same query. But it is returning
>>>> all
>>>> the responses, 25 nos.
>>>> But the solr is still indexing only 10 rows.
>>>> Is there any optimization settings by default in the solrconfig.xml
>>>> that
>>>> restricts the responses to 10 ?
>>>> thanks
>>>> con.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>>
>>>>> template transformer does not eat up rows.
>>>>>
>>>>> I am almost sure that the query returns only 10 rows in that case.
>>>>> could you write a quick jdbc program and verify that (not the oralce
>>>>> client)
>>>>>
>>>>> everything else looks fine
>>>>>
>>>>> On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Noble
>>>>>> Thanks for your reply
>>>>>>
>>>>>> In my data-config.xml I have;
>>>>>>
>>>>>> <entity name="employees" transformer="TemplateTransformer"
>>>>>> query="Select
>>>>>> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>>>>> EMPLOYEES">
>>>>>> <field column="rowtype" template="employees" />
>>>>>> <field column="EMP_ID" name="EMP_ID" />
>>>>>> <field column="EMP_NAME" name="EMP_NAME" />
>>>>>> <field column="COMMENT" name="COMMENT" />
>>>>>> </entity>
>>>>>>
>>>>>> <entity name="customers" transformer="TemplateTransformer"
>>>>>> query="Select
>>>>>> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>>>>> CUSTOMERS">
>>>>>> <field column="rowtype" template="customers" />
>>>>>> <field column="CUST_ID" name="CUST_ID />
>>>>>> <field column="CUST_NAME" name="CUST_NAME" />
>>>>>> <field column="COMMENT" name="COMMENT" />
>>>>>> </entity>
>>>>>>
>>>>>> Whether this, TemplateTransformer, is the one that is restricting the
>>>>>> resultset count to 10?
>>>>>> Where can I find it out?
>>>>>> I need this TemplateTransformer because I want to query the responses
>>>>>> of
>>>>>> either one of these at a time using the URL like,
>>>>>>
>>>>>> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>>>>>>
>>>>>> I tried in the debug mode:
>>>>>> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on)
>>>>>> , But it is not all mentioning anything after the 10th document.
>>>>>>
>>>>>>
>>>>>> Thanks and regards
>>>>>> con
>>>>>>
>>>>>>
>>>>>>
>>>>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>>>>
>>>>>>> The DIH status says 10 rows which means only 10 rows got fetched for
>>>>>>> that query. Do you have any custom transformers which eats up rows?
>>>>>>>
>>>>>>> Try the debug page of DIH and see what is happening to the rest of
>>>>>>> the
>>>>>>> rows.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> A simple question:
>>>>>>>> I performed the following steps to index data from a oracle db to
>>>>>>>> solr
>>>>>>>> index
>>>>>>>> and then search:
>>>>>>>> a) I have the configurations for indexing data from a oracle db
>>>>>>>> b) started the server.
>>>>>>>> c) Done a full-import:
>>>>>>>> http://localhost:8983/solr/dataimport?command=full-import
>>>>>>>>
>>>>>>>> But when I do a search using
>>>>>>>> http://localhost:8983/solr/select/?q=....
>>>>>>>> Not all the result sets that matches the search string are
>>>>>>>> displayed.
>>>>>>>>
>>>>>>>> 1) Is the above steps enough for getting db values to solr index?
>>>>>>>> My configurations (data-config.xml and schema.xml )are quite
>>>>>>>> correct
>>>>>>>> because
>>>>>>>> I am getting SOME of the result sets as search result(not all).
>>>>>>>> 2) Is there some value in sorconfig.xml, or some other files that
>>>>>>>> limits
>>>>>>>> the
>>>>>>>> number of items being indexed? [For the time being I have only a
>>>>>>>> few
>>>>>>>> hundreds of records in my db. ]
>>>>>>>> The query that I am specifying in data-config yields around 25
>>>>>>>> results
>>>>>>>> if
>>>>>>>> i
>>>>>>>> execute it in a oracle client, where as the status of full-import
>>>>>>>> is
>>>>>>>> something like:
>>>>>>>> <str name="status">idle</str>
>>>>>>>> <str name="importResponse">Configuration Re-loaded
>>>>>>>> sucessfully</str>
>>>>>>>> <lst name="statusMessages">
>>>>>>>> <str name="Total Requests made to DataSource">1</str>
>>>>>>>> <str name="Total Rows Fetched">10</str>
>>>>>>>> <str name="Total Documents Skipped">0</str>
>>>>>>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>>>>>>> <str name="Time taken ">0:0:0.513</str>
>>>>>>>> </lst>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --Noble Paul
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19943817.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19950250.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19988190.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr indexing not taking all values from DB.
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
now just do a normal full-import do not enable debug . I guess it
should be just fine
On Mon, Oct 13, 2008 at 1:20 PM, con <co...@gmail.com> wrote:
>
> Thanks Nobble
> I tried in the debug mode with rows=100 and it is accepting all the result
> sets.
> So i suppose there is nothing wrong in the query.
> But I am not able to update the index since this is available only in the
> debug mode.
>
> Can you please give some suggestions based on this.
>
> thanks
> con
>
>
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> in debug mode it writes only 10 because there is a rows parameter
>> which is by default set to 10
>> make it 100 or so and you should be seeing all docs. But in non-debug
>> mode there is no such parameter
>>
>> On Sun, Oct 12, 2008 at 11:00 PM, con <co...@gmail.com> wrote:
>>>
>>> I wrote a jdbc program to implement the same query. But it is returning
>>> all
>>> the responses, 25 nos.
>>> But the solr is still indexing only 10 rows.
>>> Is there any optimization settings by default in the solrconfig.xml that
>>> restricts the responses to 10 ?
>>> thanks
>>> con.
>>>
>>>
>>>
>>>
>>>
>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>
>>>> template transformer does not eat up rows.
>>>>
>>>> I am almost sure that the query returns only 10 rows in that case.
>>>> could you write a quick jdbc program and verify that (not the oralce
>>>> client)
>>>>
>>>> everything else looks fine
>>>>
>>>> On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>>>>>
>>>>> Hi Noble
>>>>> Thanks for your reply
>>>>>
>>>>> In my data-config.xml I have;
>>>>>
>>>>> <entity name="employees" transformer="TemplateTransformer"
>>>>> query="Select
>>>>> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from EMPLOYEES">
>>>>> <field column="rowtype" template="employees" />
>>>>> <field column="EMP_ID" name="EMP_ID" />
>>>>> <field column="EMP_NAME" name="EMP_NAME" />
>>>>> <field column="COMMENT" name="COMMENT" />
>>>>> </entity>
>>>>>
>>>>> <entity name="customers" transformer="TemplateTransformer"
>>>>> query="Select
>>>>> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>>>> CUSTOMERS">
>>>>> <field column="rowtype" template="customers" />
>>>>> <field column="CUST_ID" name="CUST_ID />
>>>>> <field column="CUST_NAME" name="CUST_NAME" />
>>>>> <field column="COMMENT" name="COMMENT" />
>>>>> </entity>
>>>>>
>>>>> Whether this, TemplateTransformer, is the one that is restricting the
>>>>> resultset count to 10?
>>>>> Where can I find it out?
>>>>> I need this TemplateTransformer because I want to query the responses
>>>>> of
>>>>> either one of these at a time using the URL like,
>>>>>
>>>>> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>>>>>
>>>>> I tried in the debug mode:
>>>>> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on)
>>>>> , But it is not all mentioning anything after the 10th document.
>>>>>
>>>>>
>>>>> Thanks and regards
>>>>> con
>>>>>
>>>>>
>>>>>
>>>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>>>
>>>>>> The DIH status says 10 rows which means only 10 rows got fetched for
>>>>>> that query. Do you have any custom transformers which eats up rows?
>>>>>>
>>>>>> Try the debug page of DIH and see what is happening to the rest of the
>>>>>> rows.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>>>>>
>>>>>>> A simple question:
>>>>>>> I performed the following steps to index data from a oracle db to
>>>>>>> solr
>>>>>>> index
>>>>>>> and then search:
>>>>>>> a) I have the configurations for indexing data from a oracle db
>>>>>>> b) started the server.
>>>>>>> c) Done a full-import:
>>>>>>> http://localhost:8983/solr/dataimport?command=full-import
>>>>>>>
>>>>>>> But when I do a search using
>>>>>>> http://localhost:8983/solr/select/?q=....
>>>>>>> Not all the result sets that matches the search string are displayed.
>>>>>>>
>>>>>>> 1) Is the above steps enough for getting db values to solr index?
>>>>>>> My configurations (data-config.xml and schema.xml )are quite correct
>>>>>>> because
>>>>>>> I am getting SOME of the result sets as search result(not all).
>>>>>>> 2) Is there some value in sorconfig.xml, or some other files that
>>>>>>> limits
>>>>>>> the
>>>>>>> number of items being indexed? [For the time being I have only a few
>>>>>>> hundreds of records in my db. ]
>>>>>>> The query that I am specifying in data-config yields around 25
>>>>>>> results
>>>>>>> if
>>>>>>> i
>>>>>>> execute it in a oracle client, where as the status of full-import is
>>>>>>> something like:
>>>>>>> <str name="status">idle</str>
>>>>>>> <str name="importResponse">Configuration Re-loaded sucessfully</str>
>>>>>>> <lst name="statusMessages">
>>>>>>> <str name="Total Requests made to DataSource">1</str>
>>>>>>> <str name="Total Rows Fetched">10</str>
>>>>>>> <str name="Total Documents Skipped">0</str>
>>>>>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>>>>>> <str name="Time taken ">0:0:0.513</str>
>>>>>>> </lst>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Noble Paul
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19943817.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19950250.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Solr indexing not taking all values from DB.
Posted by con <co...@gmail.com>.
Thanks Nobble
I tried in the debug mode with rows=100 and it is accepting all the result
sets.
So i suppose there is nothing wrong in the query.
But I am not able to update the index since this is available only in the
debug mode.
Can you please give some suggestions based on this.
thanks
con
Noble Paul നോബിള് नोब्ळ् wrote:
>
> in debug mode it writes only 10 because there is a rows parameter
> which is by default set to 10
> make it 100 or so and you should be seeing all docs. But in non-debug
> mode there is no such parameter
>
> On Sun, Oct 12, 2008 at 11:00 PM, con <co...@gmail.com> wrote:
>>
>> I wrote a jdbc program to implement the same query. But it is returning
>> all
>> the responses, 25 nos.
>> But the solr is still indexing only 10 rows.
>> Is there any optimization settings by default in the solrconfig.xml that
>> restricts the responses to 10 ?
>> thanks
>> con.
>>
>>
>>
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> template transformer does not eat up rows.
>>>
>>> I am almost sure that the query returns only 10 rows in that case.
>>> could you write a quick jdbc program and verify that (not the oralce
>>> client)
>>>
>>> everything else looks fine
>>>
>>> On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>>>>
>>>> Hi Noble
>>>> Thanks for your reply
>>>>
>>>> In my data-config.xml I have;
>>>>
>>>> <entity name="employees" transformer="TemplateTransformer"
>>>> query="Select
>>>> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from EMPLOYEES">
>>>> <field column="rowtype" template="employees" />
>>>> <field column="EMP_ID" name="EMP_ID" />
>>>> <field column="EMP_NAME" name="EMP_NAME" />
>>>> <field column="COMMENT" name="COMMENT" />
>>>> </entity>
>>>>
>>>> <entity name="customers" transformer="TemplateTransformer"
>>>> query="Select
>>>> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from
>>>> CUSTOMERS">
>>>> <field column="rowtype" template="customers" />
>>>> <field column="CUST_ID" name="CUST_ID />
>>>> <field column="CUST_NAME" name="CUST_NAME" />
>>>> <field column="COMMENT" name="COMMENT" />
>>>> </entity>
>>>>
>>>> Whether this, TemplateTransformer, is the one that is restricting the
>>>> resultset count to 10?
>>>> Where can I find it out?
>>>> I need this TemplateTransformer because I want to query the responses
>>>> of
>>>> either one of these at a time using the URL like,
>>>>
>>>> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>>>>
>>>> I tried in the debug mode:
>>>> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on)
>>>> , But it is not all mentioning anything after the 10th document.
>>>>
>>>>
>>>> Thanks and regards
>>>> con
>>>>
>>>>
>>>>
>>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>>
>>>>> The DIH status says 10 rows which means only 10 rows got fetched for
>>>>> that query. Do you have any custom transformers which eats up rows?
>>>>>
>>>>> Try the debug page of DIH and see what is happening to the rest of the
>>>>> rows.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>>>>
>>>>>> A simple question:
>>>>>> I performed the following steps to index data from a oracle db to
>>>>>> solr
>>>>>> index
>>>>>> and then search:
>>>>>> a) I have the configurations for indexing data from a oracle db
>>>>>> b) started the server.
>>>>>> c) Done a full-import:
>>>>>> http://localhost:8983/solr/dataimport?command=full-import
>>>>>>
>>>>>> But when I do a search using
>>>>>> http://localhost:8983/solr/select/?q=....
>>>>>> Not all the result sets that matches the search string are displayed.
>>>>>>
>>>>>> 1) Is the above steps enough for getting db values to solr index?
>>>>>> My configurations (data-config.xml and schema.xml )are quite correct
>>>>>> because
>>>>>> I am getting SOME of the result sets as search result(not all).
>>>>>> 2) Is there some value in sorconfig.xml, or some other files that
>>>>>> limits
>>>>>> the
>>>>>> number of items being indexed? [For the time being I have only a few
>>>>>> hundreds of records in my db. ]
>>>>>> The query that I am specifying in data-config yields around 25
>>>>>> results
>>>>>> if
>>>>>> i
>>>>>> execute it in a oracle client, where as the status of full-import is
>>>>>> something like:
>>>>>> <str name="status">idle</str>
>>>>>> <str name="importResponse">Configuration Re-loaded sucessfully</str>
>>>>>> <lst name="statusMessages">
>>>>>> <str name="Total Requests made to DataSource">1</str>
>>>>>> <str name="Total Rows Fetched">10</str>
>>>>>> <str name="Total Documents Skipped">0</str>
>>>>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>>>>> <str name="Time taken ">0:0:0.513</str>
>>>>>> </lst>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19943817.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19950250.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr indexing not taking all values from DB.
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
in debug mode it writes only 10 because there is a rows parameter
which is by default set to 10
make it 100 or so and you should be seeing all docs. But in non-debug
mode there is no such parameter
On Sun, Oct 12, 2008 at 11:00 PM, con <co...@gmail.com> wrote:
>
> I wrote a jdbc program to implement the same query. But it is returning all
> the responses, 25 nos.
> But the solr is still indexing only 10 rows.
> Is there any optimization settings by default in the solrconfig.xml that
> restricts the responses to 10 ?
> thanks
> con.
>
>
>
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> template transformer does not eat up rows.
>>
>> I am almost sure that the query returns only 10 rows in that case.
>> could you write a quick jdbc program and verify that (not the oralce
>> client)
>>
>> everything else looks fine
>>
>> On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>>>
>>> Hi Noble
>>> Thanks for your reply
>>>
>>> In my data-config.xml I have;
>>>
>>> <entity name="employees" transformer="TemplateTransformer"
>>> query="Select
>>> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from EMPLOYEES">
>>> <field column="rowtype" template="employees" />
>>> <field column="EMP_ID" name="EMP_ID" />
>>> <field column="EMP_NAME" name="EMP_NAME" />
>>> <field column="COMMENT" name="COMMENT" />
>>> </entity>
>>>
>>> <entity name="customers" transformer="TemplateTransformer"
>>> query="Select
>>> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from CUSTOMERS">
>>> <field column="rowtype" template="customers" />
>>> <field column="CUST_ID" name="CUST_ID />
>>> <field column="CUST_NAME" name="CUST_NAME" />
>>> <field column="COMMENT" name="COMMENT" />
>>> </entity>
>>>
>>> Whether this, TemplateTransformer, is the one that is restricting the
>>> resultset count to 10?
>>> Where can I find it out?
>>> I need this TemplateTransformer because I want to query the responses of
>>> either one of these at a time using the URL like,
>>>
>>> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>>>
>>> I tried in the debug mode:
>>> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on)
>>> , But it is not all mentioning anything after the 10th document.
>>>
>>>
>>> Thanks and regards
>>> con
>>>
>>>
>>>
>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>>
>>>> The DIH status says 10 rows which means only 10 rows got fetched for
>>>> that query. Do you have any custom transformers which eats up rows?
>>>>
>>>> Try the debug page of DIH and see what is happening to the rest of the
>>>> rows.
>>>>
>>>>
>>>>
>>>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>>>
>>>>> A simple question:
>>>>> I performed the following steps to index data from a oracle db to solr
>>>>> index
>>>>> and then search:
>>>>> a) I have the configurations for indexing data from a oracle db
>>>>> b) started the server.
>>>>> c) Done a full-import:
>>>>> http://localhost:8983/solr/dataimport?command=full-import
>>>>>
>>>>> But when I do a search using http://localhost:8983/solr/select/?q=....
>>>>> Not all the result sets that matches the search string are displayed.
>>>>>
>>>>> 1) Is the above steps enough for getting db values to solr index?
>>>>> My configurations (data-config.xml and schema.xml )are quite correct
>>>>> because
>>>>> I am getting SOME of the result sets as search result(not all).
>>>>> 2) Is there some value in sorconfig.xml, or some other files that
>>>>> limits
>>>>> the
>>>>> number of items being indexed? [For the time being I have only a few
>>>>> hundreds of records in my db. ]
>>>>> The query that I am specifying in data-config yields around 25 results
>>>>> if
>>>>> i
>>>>> execute it in a oracle client, where as the status of full-import is
>>>>> something like:
>>>>> <str name="status">idle</str>
>>>>> <str name="importResponse">Configuration Re-loaded sucessfully</str>
>>>>> <lst name="statusMessages">
>>>>> <str name="Total Requests made to DataSource">1</str>
>>>>> <str name="Total Rows Fetched">10</str>
>>>>> <str name="Total Documents Skipped">0</str>
>>>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>>>> <str name="Time taken ">0:0:0.513</str>
>>>>> </lst>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19943817.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Solr indexing not taking all values from DB.
Posted by con <co...@gmail.com>.
I wrote a jdbc program to implement the same query. But it is returning all
the responses, 25 nos.
But the solr is still indexing only 10 rows.
Is there any optimization settings by default in the solrconfig.xml that
restricts the responses to 10 ?
thanks
con.
Noble Paul നോബിള് नोब्ळ् wrote:
>
> template transformer does not eat up rows.
>
> I am almost sure that the query returns only 10 rows in that case.
> could you write a quick jdbc program and verify that (not the oralce
> client)
>
> everything else looks fine
>
> On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>>
>> Hi Noble
>> Thanks for your reply
>>
>> In my data-config.xml I have;
>>
>> <entity name="employees" transformer="TemplateTransformer"
>> query="Select
>> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from EMPLOYEES">
>> <field column="rowtype" template="employees" />
>> <field column="EMP_ID" name="EMP_ID" />
>> <field column="EMP_NAME" name="EMP_NAME" />
>> <field column="COMMENT" name="COMMENT" />
>> </entity>
>>
>> <entity name="customers" transformer="TemplateTransformer"
>> query="Select
>> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from CUSTOMERS">
>> <field column="rowtype" template="customers" />
>> <field column="CUST_ID" name="CUST_ID />
>> <field column="CUST_NAME" name="CUST_NAME" />
>> <field column="COMMENT" name="COMMENT" />
>> </entity>
>>
>> Whether this, TemplateTransformer, is the one that is restricting the
>> resultset count to 10?
>> Where can I find it out?
>> I need this TemplateTransformer because I want to query the responses of
>> either one of these at a time using the URL like,
>>
>> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>>
>> I tried in the debug mode:
>> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on)
>> , But it is not all mentioning anything after the 10th document.
>>
>>
>> Thanks and regards
>> con
>>
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> The DIH status says 10 rows which means only 10 rows got fetched for
>>> that query. Do you have any custom transformers which eats up rows?
>>>
>>> Try the debug page of DIH and see what is happening to the rest of the
>>> rows.
>>>
>>>
>>>
>>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>>
>>>> A simple question:
>>>> I performed the following steps to index data from a oracle db to solr
>>>> index
>>>> and then search:
>>>> a) I have the configurations for indexing data from a oracle db
>>>> b) started the server.
>>>> c) Done a full-import:
>>>> http://localhost:8983/solr/dataimport?command=full-import
>>>>
>>>> But when I do a search using http://localhost:8983/solr/select/?q=....
>>>> Not all the result sets that matches the search string are displayed.
>>>>
>>>> 1) Is the above steps enough for getting db values to solr index?
>>>> My configurations (data-config.xml and schema.xml )are quite correct
>>>> because
>>>> I am getting SOME of the result sets as search result(not all).
>>>> 2) Is there some value in sorconfig.xml, or some other files that
>>>> limits
>>>> the
>>>> number of items being indexed? [For the time being I have only a few
>>>> hundreds of records in my db. ]
>>>> The query that I am specifying in data-config yields around 25 results
>>>> if
>>>> i
>>>> execute it in a oracle client, where as the status of full-import is
>>>> something like:
>>>> <str name="status">idle</str>
>>>> <str name="importResponse">Configuration Re-loaded sucessfully</str>
>>>> <lst name="statusMessages">
>>>> <str name="Total Requests made to DataSource">1</str>
>>>> <str name="Total Rows Fetched">10</str>
>>>> <str name="Total Documents Skipped">0</str>
>>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>>> <str name="Time taken ">0:0:0.513</str>
>>>> </lst>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19943817.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr indexing not taking all values from DB.
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
template transformer does not eat up rows.
I am almost sure that the query returns only 10 rows in that case.
could you write a quick jdbc program and verify that (not the oralce
client)
everything else looks fine
On Sat, Oct 11, 2008 at 4:52 PM, con <co...@gmail.com> wrote:
>
> Hi Noble
> Thanks for your reply
>
> In my data-config.xml I have;
>
> <entity name="employees" transformer="TemplateTransformer" query="Select
> EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from EMPLOYEES">
> <field column="rowtype" template="employees" />
> <field column="EMP_ID" name="EMP_ID" />
> <field column="EMP_NAME" name="EMP_NAME" />
> <field column="COMMENT" name="COMMENT" />
> </entity>
>
> <entity name="customers" transformer="TemplateTransformer" query="Select
> CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from CUSTOMERS">
> <field column="rowtype" template="customers" />
> <field column="CUST_ID" name="CUST_ID />
> <field column="CUST_NAME" name="CUST_NAME" />
> <field column="COMMENT" name="COMMENT" />
> </entity>
>
> Whether this, TemplateTransformer, is the one that is restricting the
> resultset count to 10?
> Where can I find it out?
> I need this TemplateTransformer because I want to query the responses of
> either one of these at a time using the URL like,
>
> http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
>
> I tried in the debug mode:
> (http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on)
> , But it is not all mentioning anything after the 10th document.
>
>
> Thanks and regards
> con
>
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> The DIH status says 10 rows which means only 10 rows got fetched for
>> that query. Do you have any custom transformers which eats up rows?
>>
>> Try the debug page of DIH and see what is happening to the rest of the
>> rows.
>>
>>
>>
>> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>>
>>> A simple question:
>>> I performed the following steps to index data from a oracle db to solr
>>> index
>>> and then search:
>>> a) I have the configurations for indexing data from a oracle db
>>> b) started the server.
>>> c) Done a full-import:
>>> http://localhost:8983/solr/dataimport?command=full-import
>>>
>>> But when I do a search using http://localhost:8983/solr/select/?q=....
>>> Not all the result sets that matches the search string are displayed.
>>>
>>> 1) Is the above steps enough for getting db values to solr index?
>>> My configurations (data-config.xml and schema.xml )are quite correct
>>> because
>>> I am getting SOME of the result sets as search result(not all).
>>> 2) Is there some value in sorconfig.xml, or some other files that limits
>>> the
>>> number of items being indexed? [For the time being I have only a few
>>> hundreds of records in my db. ]
>>> The query that I am specifying in data-config yields around 25 results if
>>> i
>>> execute it in a oracle client, where as the status of full-import is
>>> something like:
>>> <str name="status">idle</str>
>>> <str name="importResponse">Configuration Re-loaded sucessfully</str>
>>> <lst name="statusMessages">
>>> <str name="Total Requests made to DataSource">1</str>
>>> <str name="Total Rows Fetched">10</str>
>>> <str name="Total Documents Skipped">0</str>
>>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>>> <str name="Time taken ">0:0:0.513</str>
>>> </lst>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Solr indexing not taking all values from DB.
Posted by con <co...@gmail.com>.
Hi Noble
Thanks for your reply
In my data-config.xml I have;
<entity name="employees" transformer="TemplateTransformer" query="Select
EMP_ID , EMP_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from EMPLOYEES">
<field column="rowtype" template="employees" />
<field column="EMP_ID" name="EMP_ID" />
<field column="EMP_NAME" name="EMP_NAME" />
<field column="COMMENT" name="COMMENT" />
</entity>
<entity name="customers" transformer="TemplateTransformer" query="Select
CUST_ID , CUST_NAME , NVL (COMMENT ,'-Nil-') as COMMENT from CUSTOMERS">
<field column="rowtype" template="customers" />
<field column="CUST_ID" name="CUST_ID />
<field column="CUST_NAME" name="CUST_NAME" />
<field column="COMMENT" name="COMMENT" />
</entity>
Whether this, TemplateTransformer, is the one that is restricting the
resultset count to 10?
Where can I find it out?
I need this TemplateTransformer because I want to query the responses of
either one of these at a time using the URL like,
http://localhost:8983/solr/select/?q=(Bob%20AND%20rowtype:customers)&version=2.2&start=0&rows=10&indent=on&wt=json
I tried in the debug mode:
(http://localhost:8983/solr/dataimport?command=full-import&debug=on&verbose=on)
, But it is not all mentioning anything after the 10th document.
Thanks and regards
con
Noble Paul നോബിള് नोब्ळ् wrote:
>
> The DIH status says 10 rows which means only 10 rows got fetched for
> that query. Do you have any custom transformers which eats up rows?
>
> Try the debug page of DIH and see what is happening to the rest of the
> rows.
>
>
>
> On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>>
>> A simple question:
>> I performed the following steps to index data from a oracle db to solr
>> index
>> and then search:
>> a) I have the configurations for indexing data from a oracle db
>> b) started the server.
>> c) Done a full-import:
>> http://localhost:8983/solr/dataimport?command=full-import
>>
>> But when I do a search using http://localhost:8983/solr/select/?q=....
>> Not all the result sets that matches the search string are displayed.
>>
>> 1) Is the above steps enough for getting db values to solr index?
>> My configurations (data-config.xml and schema.xml )are quite correct
>> because
>> I am getting SOME of the result sets as search result(not all).
>> 2) Is there some value in sorconfig.xml, or some other files that limits
>> the
>> number of items being indexed? [For the time being I have only a few
>> hundreds of records in my db. ]
>> The query that I am specifying in data-config yields around 25 results if
>> i
>> execute it in a oracle client, where as the status of full-import is
>> something like:
>> <str name="status">idle</str>
>> <str name="importResponse">Configuration Re-loaded sucessfully</str>
>> <lst name="statusMessages">
>> <str name="Total Requests made to DataSource">1</str>
>> <str name="Total Rows Fetched">10</str>
>> <str name="Total Documents Skipped">0</str>
>> <str name="Full Dump Started">2008-10-10 17:29:03</str>
>> <str name="Time taken ">0:0:0.513</str>
>> </lst>
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19931736.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr indexing not taking all values from DB.
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
The DIH status says 10 rows which means only 10 rows got fetched for
that query. Do you have any custom transformers which eats up rows?
Try the debug page of DIH and see what is happening to the rest of the rows.
On Fri, Oct 10, 2008 at 5:32 PM, con <co...@gmail.com> wrote:
>
> A simple question:
> I performed the following steps to index data from a oracle db to solr index
> and then search:
> a) I have the configurations for indexing data from a oracle db
> b) started the server.
> c) Done a full-import:
> http://localhost:8983/solr/dataimport?command=full-import
>
> But when I do a search using http://localhost:8983/solr/select/?q=....
> Not all the result sets that matches the search string are displayed.
>
> 1) Is the above steps enough for getting db values to solr index?
> My configurations (data-config.xml and schema.xml )are quite correct because
> I am getting SOME of the result sets as search result(not all).
> 2) Is there some value in sorconfig.xml, or some other files that limits the
> number of items being indexed? [For the time being I have only a few
> hundreds of records in my db. ]
> The query that I am specifying in data-config yields around 25 results if i
> execute it in a oracle client, where as the status of full-import is
> something like:
> <str name="status">idle</str>
> <str name="importResponse">Configuration Re-loaded sucessfully</str>
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">1</str>
> <str name="Total Rows Fetched">10</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-10-10 17:29:03</str>
> <str name="Time taken ">0:0:0.513</str>
> </lst>
>
>
>
> --
> View this message in context: http://www.nabble.com/Solr-indexing-not-taking-all-values-from-DB.-tp19916938p19916938.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul