You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rusli Ruslakall <ru...@gmail.com> on 2008/06/10 14:41:25 UTC
XML vs mysql import with DataImportHandler
I wanted to see what people think about using DataImportHandler and
importing directly from mysql rather than with XML.
I have already programmed mysql -> XML in PHP and configured Solr to
import it when I realized I could use DataImportHandler and skip the
XML stage. Are there any negatives of using DataImportHandler? I would
have to use the patch that complicates the process I guess when its
kind of tempting to try to use Solr as much vanilla as possible.
What is the general method when using database information?
Re: XML vs mysql import with DataImportHandler
Posted by Rusli Ruslakall <ru...@gmail.com>.
That sounds great!
I'm not launching for the next few of months so I'll wait and hope
that 1.3 comes out before that and just use XML while I develop the
rest of the site.
Thank you!
On Tue, Jun 10, 2008 at 12:57 PM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> Hi Rusli,
>
> Without going into specifics, I can tell you that we're using
> DataImportHandler successfully in production with many millions of documents
> being imported from databases. Try it out and let us know if you face any
> problems. DataImportHandler will be part of Solr 1.3 (the next release) and
> may be committed quite soon after which you may not need to fiddle with
> patches.
>
> Disclaimer: Since I'm one of the developers, I may be biased :)
>
> On Tue, Jun 10, 2008 at 6:11 PM, Rusli Ruslakall <ru...@gmail.com>
> wrote:
>
>> I wanted to see what people think about using DataImportHandler and
>> importing directly from mysql rather than with XML.
>>
>> I have already programmed mysql -> XML in PHP and configured Solr to
>> import it when I realized I could use DataImportHandler and skip the
>> XML stage. Are there any negatives of using DataImportHandler? I would
>> have to use the patch that complicates the process I guess when its
>> kind of tempting to try to use Solr as much vanilla as possible.
>>
>> What is the general method when using database information?
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: XML vs mysql import with DataImportHandler
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
The attribute name is batchSize="-1" (it is case sensitive) . Tjis
ensures that Mysql driver fetcches row by row
http://wiki.apache.org/solr/DataImportHandlerFaq
On Mon, Nov 3, 2008 at 9:17 PM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi Shalin,
> *
> I would like to know if you just used batchsize = -1.
> When I do that I use all Mysql's memory and it's a problem for the database
> and other process on it like update insert ...
> It will keep my database busy for 10hours, it's too much, is there a way to
> manage it differently ?
>
> Thanks a lot,
>
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Hi Rusli,
>>
>> Without going into specifics, I can tell you that we're using
>> DataImportHandler successfully in production with many millions of
>> documents
>> being imported from databases. Try it out and let us know if you face any
>> problems. DataImportHandler will be part of Solr 1.3 (the next release)
>> and
>> may be committed quite soon after which you may not need to fiddle with
>> patches.
>>
>> Disclaimer: Since I'm one of the developers, I may be biased :)
>>
>> On Tue, Jun 10, 2008 at 6:11 PM, Rusli Ruslakall
>> <ru...@gmail.com>
>> wrote:
>>
>>> I wanted to see what people think about using DataImportHandler and
>>> importing directly from mysql rather than with XML.
>>>
>>> I have already programmed mysql -> XML in PHP and configured Solr to
>>> import it when I realized I could use DataImportHandler and skip the
>>> XML stage. Are there any negatives of using DataImportHandler? I would
>>> have to use the patch that complicates the process I guess when its
>>> kind of tempting to try to use Solr as much vanilla as possible.
>>>
>>> What is the general method when using database information?
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
> --
> View this message in context: http://www.nabble.com/XML-vs-mysql-import-with-DataImportHandler-tp17754471p20305117.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: XML vs mysql import with DataImportHandler
Posted by sunnyfr <jo...@gmail.com>.
Hi Shalin,
*
I would like to know if you just used batchsize = -1.
When I do that I use all Mysql's memory and it's a problem for the database
and other process on it like update insert ...
It will keep my database busy for 10hours, it's too much, is there a way to
manage it differently ?
Thanks a lot,
Shalin Shekhar Mangar wrote:
>
> Hi Rusli,
>
> Without going into specifics, I can tell you that we're using
> DataImportHandler successfully in production with many millions of
> documents
> being imported from databases. Try it out and let us know if you face any
> problems. DataImportHandler will be part of Solr 1.3 (the next release)
> and
> may be committed quite soon after which you may not need to fiddle with
> patches.
>
> Disclaimer: Since I'm one of the developers, I may be biased :)
>
> On Tue, Jun 10, 2008 at 6:11 PM, Rusli Ruslakall
> <ru...@gmail.com>
> wrote:
>
>> I wanted to see what people think about using DataImportHandler and
>> importing directly from mysql rather than with XML.
>>
>> I have already programmed mysql -> XML in PHP and configured Solr to
>> import it when I realized I could use DataImportHandler and skip the
>> XML stage. Are there any negatives of using DataImportHandler? I would
>> have to use the patch that complicates the process I guess when its
>> kind of tempting to try to use Solr as much vanilla as possible.
>>
>> What is the general method when using database information?
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/XML-vs-mysql-import-with-DataImportHandler-tp17754471p20305117.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: XML vs mysql import with DataImportHandler
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
No. DataImportHandler runs inside of Solr itself so it does not need to use
HTTP. It reads databases using Jdbc and uses API calls to create and load
documents into Solr. Benefits over custom solutions would include speed (no
HTTP/XML parsing overhead) and ofcourse all benefits of open source software
apply :)
On Wed, Jun 11, 2008 at 2:23 AM, wojtekpia <wo...@hotmail.com> wrote:
>
> Does the DataImportHandler use HTTP to load documents into Solr?
>
> Is there any (run-time) performance benefit to using it rather than a
> custom
> MySQL -> XML solution?
>
>
>
> Shalin Shekhar Mangar wrote:
> >
> > Hi Rusli,
> >
> > Without going into specifics, I can tell you that we're using
> > DataImportHandler successfully in production with many millions of
> > documents
> > being imported from databases. Try it out and let us know if you face any
> > problems. DataImportHandler will be part of Solr 1.3 (the next release)
> > and
> > may be committed quite soon after which you may not need to fiddle with
> > patches.
> >
> > Disclaimer: Since I'm one of the developers, I may be biased :)
> >
> > On Tue, Jun 10, 2008 at 6:11 PM, Rusli Ruslakall
> > <ru...@gmail.com>
> > wrote:
> >
> >> I wanted to see what people think about using DataImportHandler and
> >> importing directly from mysql rather than with XML.
> >>
> >> I have already programmed mysql -> XML in PHP and configured Solr to
> >> import it when I realized I could use DataImportHandler and skip the
> >> XML stage. Are there any negatives of using DataImportHandler? I would
> >> have to use the patch that complicates the process I guess when its
> >> kind of tempting to try to use Solr as much vanilla as possible.
> >>
> >> What is the general method when using database information?
> >>
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/XML-vs-mysql-import-with-DataImportHandler-tp17754471p17764425.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: XML vs mysql import with DataImportHandler
Posted by wojtekpia <wo...@hotmail.com>.
Does the DataImportHandler use HTTP to load documents into Solr?
Is there any (run-time) performance benefit to using it rather than a custom
MySQL -> XML solution?
Shalin Shekhar Mangar wrote:
>
> Hi Rusli,
>
> Without going into specifics, I can tell you that we're using
> DataImportHandler successfully in production with many millions of
> documents
> being imported from databases. Try it out and let us know if you face any
> problems. DataImportHandler will be part of Solr 1.3 (the next release)
> and
> may be committed quite soon after which you may not need to fiddle with
> patches.
>
> Disclaimer: Since I'm one of the developers, I may be biased :)
>
> On Tue, Jun 10, 2008 at 6:11 PM, Rusli Ruslakall
> <ru...@gmail.com>
> wrote:
>
>> I wanted to see what people think about using DataImportHandler and
>> importing directly from mysql rather than with XML.
>>
>> I have already programmed mysql -> XML in PHP and configured Solr to
>> import it when I realized I could use DataImportHandler and skip the
>> XML stage. Are there any negatives of using DataImportHandler? I would
>> have to use the patch that complicates the process I guess when its
>> kind of tempting to try to use Solr as much vanilla as possible.
>>
>> What is the general method when using database information?
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/XML-vs-mysql-import-with-DataImportHandler-tp17754471p17764425.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: XML vs mysql import with DataImportHandler
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Rusli,
Without going into specifics, I can tell you that we're using
DataImportHandler successfully in production with many millions of documents
being imported from databases. Try it out and let us know if you face any
problems. DataImportHandler will be part of Solr 1.3 (the next release) and
may be committed quite soon after which you may not need to fiddle with
patches.
Disclaimer: Since I'm one of the developers, I may be biased :)
On Tue, Jun 10, 2008 at 6:11 PM, Rusli Ruslakall <ru...@gmail.com>
wrote:
> I wanted to see what people think about using DataImportHandler and
> importing directly from mysql rather than with XML.
>
> I have already programmed mysql -> XML in PHP and configured Solr to
> import it when I realized I could use DataImportHandler and skip the
> XML stage. Are there any negatives of using DataImportHandler? I would
> have to use the patch that complicates the process I guess when its
> kind of tempting to try to use Solr as much vanilla as possible.
>
> What is the general method when using database information?
>
--
Regards,
Shalin Shekhar Mangar.