You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Sturlese <ma...@gmail.com> on 2009/11/05 13:48:04 UTC
DIH full-import with fetchSize(Integer.MIN_VALUE) taking long time
to start processing rows
I have been using fetchSize(Integer.MIN_VALUE) for a long time and it was
working perfect until now. I use MySQL, java 1.6,
mysql-connector-java-5.1.7-bin.jar and InoDB tables.
Since a month ago when the query is executed it will take a long time untill
it starts processing the results from the resultSet. The query matches about
2M rows. It use to take 10 min untill rows processing started. Now it’s
taking about 2 hours.
Parallelly I have another process wich is doing lots of inserts to that
table (I also had it before but with less number of inserts). Could this be
causing some bloking that makes the query take that long? In case not, and
advice what could make to take so long until I start see rows beeing
processed?
Thanks in advance.
--
View this message in context: http://old.nabble.com/DIH-full-import-with-fetchSize%28Integer.MIN_VALUE%29-taking-long-time-to-start-processing-rows-tp26213642p26213642.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH full-import with fetchSize(Integer.MIN_VALUE) taking long
time to start processing rows
Posted by Lance Norskog <go...@gmail.com>.
Right, a view will not help here. It is just and SQL query embedded as
a virtual table, and is used to lift SQL syntax out of the DIH.
InnoDB is row-level except for auto-increment operations. Ow. You
could drop the indexes on the table. Each insert batch has to
recalculate all indexes, so this will cut the amount of database
contention.
On Thu, Nov 5, 2009 at 6:47 AM, Marc Sturlese <ma...@gmail.com> wrote:
>
> Hey there,
> I need this funcionality because I have an indexer continuously updating my
> index with delta import from a table. This table is fed by another process
> that is constantly runing too.
> With delta-import tehre's no problem but sometimes I need to execute
> full-import.
> I don't see the benefits of using a view. I think provably the best solution
> would be to create a master-slave mysql structure. The process that inserts
> would attack the master and the query would attack the slave. This provably
> would speed up the rows processing, isn't it?
>
> Avlesh Singh wrote:
>>
>>>
>>> Parallelly I have another process wich is doing lots of inserts to that
>>> table (I also had it before but with less number of inserts). Could this
>>> be
>>> causing some bloking that makes the query take that long? In case not,
>>> and
>>> advice what could make to take so long until I start see rows beeing
>>> processed?
>>>
>> Sounds scary! With innodb engine you are causing a table level lock with
>> each insert (assuming your table has an auto-increment column). With
>> frequent inserts you are of-course delaying the read time.
>> Why would you want to do this kind of an operation in the very first
>> place?
>> Can't you use views for indexing?
>>
>> Cheers
>> Avlesh
>>
>> On Thu, Nov 5, 2009 at 6:18 PM, Marc Sturlese
>> <ma...@gmail.com>wrote:
>>
>>>
>>> I have been using fetchSize(Integer.MIN_VALUE) for a long time and it was
>>> working perfect until now. I use MySQL, java 1.6,
>>> mysql-connector-java-5.1.7-bin.jar and InoDB tables.
>>> Since a month ago when the query is executed it will take a long time
>>> untill
>>> it starts processing the results from the resultSet. The query matches
>>> about
>>> 2M rows. It use to take 10 min untill rows processing started. Now it’s
>>> taking about 2 hours.
>>> Parallelly I have another process wich is doing lots of inserts to that
>>> table (I also had it before but with less number of inserts). Could this
>>> be
>>> causing some bloking that makes the query take that long? In case not,
>>> and
>>> advice what could make to take so long until I start see rows beeing
>>> processed?
>>> Thanks in advance.
>>> --
>>> View this message in context:
>>> http://old.nabble.com/DIH-full-import-with-fetchSize%28Integer.MIN_VALUE%29-taking-long-time-to-start-processing-rows-tp26213642p26213642.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/DIH-full-import-with-fetchSize%28Integer.MIN_VALUE%29-taking-long-time-to-start-processing-rows-tp26213642p26215730.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Lance Norskog
goksron@gmail.com
Re: DIH full-import with fetchSize(Integer.MIN_VALUE) taking long
time to start processing rows
Posted by Marc Sturlese <ma...@gmail.com>.
Hey there,
I need this funcionality because I have an indexer continuously updating my
index with delta import from a table. This table is fed by another process
that is constantly runing too.
With delta-import tehre's no problem but sometimes I need to execute
full-import.
I don't see the benefits of using a view. I think provably the best solution
would be to create a master-slave mysql structure. The process that inserts
would attack the master and the query would attack the slave. This provably
would speed up the rows processing, isn't it?
Avlesh Singh wrote:
>
>>
>> Parallelly I have another process wich is doing lots of inserts to that
>> table (I also had it before but with less number of inserts). Could this
>> be
>> causing some bloking that makes the query take that long? In case not,
>> and
>> advice what could make to take so long until I start see rows beeing
>> processed?
>>
> Sounds scary! With innodb engine you are causing a table level lock with
> each insert (assuming your table has an auto-increment column). With
> frequent inserts you are of-course delaying the read time.
> Why would you want to do this kind of an operation in the very first
> place?
> Can't you use views for indexing?
>
> Cheers
> Avlesh
>
> On Thu, Nov 5, 2009 at 6:18 PM, Marc Sturlese
> <ma...@gmail.com>wrote:
>
>>
>> I have been using fetchSize(Integer.MIN_VALUE) for a long time and it was
>> working perfect until now. I use MySQL, java 1.6,
>> mysql-connector-java-5.1.7-bin.jar and InoDB tables.
>> Since a month ago when the query is executed it will take a long time
>> untill
>> it starts processing the results from the resultSet. The query matches
>> about
>> 2M rows. It use to take 10 min untill rows processing started. Now it’s
>> taking about 2 hours.
>> Parallelly I have another process wich is doing lots of inserts to that
>> table (I also had it before but with less number of inserts). Could this
>> be
>> causing some bloking that makes the query take that long? In case not,
>> and
>> advice what could make to take so long until I start see rows beeing
>> processed?
>> Thanks in advance.
>> --
>> View this message in context:
>> http://old.nabble.com/DIH-full-import-with-fetchSize%28Integer.MIN_VALUE%29-taking-long-time-to-start-processing-rows-tp26213642p26213642.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://old.nabble.com/DIH-full-import-with-fetchSize%28Integer.MIN_VALUE%29-taking-long-time-to-start-processing-rows-tp26213642p26215730.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH full-import with fetchSize(Integer.MIN_VALUE) taking long
time to start processing rows
Posted by Avlesh Singh <av...@gmail.com>.
>
> Parallelly I have another process wich is doing lots of inserts to that
> table (I also had it before but with less number of inserts). Could this be
> causing some bloking that makes the query take that long? In case not, and
> advice what could make to take so long until I start see rows beeing
> processed?
>
Sounds scary! With innodb engine you are causing a table level lock with
each insert (assuming your table has an auto-increment column). With
frequent inserts you are of-course delaying the read time.
Why would you want to do this kind of an operation in the very first place?
Can't you use views for indexing?
Cheers
Avlesh
On Thu, Nov 5, 2009 at 6:18 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> I have been using fetchSize(Integer.MIN_VALUE) for a long time and it was
> working perfect until now. I use MySQL, java 1.6,
> mysql-connector-java-5.1.7-bin.jar and InoDB tables.
> Since a month ago when the query is executed it will take a long time
> untill
> it starts processing the results from the resultSet. The query matches
> about
> 2M rows. It use to take 10 min untill rows processing started. Now it’s
> taking about 2 hours.
> Parallelly I have another process wich is doing lots of inserts to that
> table (I also had it before but with less number of inserts). Could this be
> causing some bloking that makes the query take that long? In case not, and
> advice what could make to take so long until I start see rows beeing
> processed?
> Thanks in advance.
> --
> View this message in context:
> http://old.nabble.com/DIH-full-import-with-fetchSize%28Integer.MIN_VALUE%29-taking-long-time-to-start-processing-rows-tp26213642p26213642.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>