You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Glen Newton <gl...@gmail.com> on 2009/07/02 05:06:13 UTC
Re: Is there any other way to load the index beside using "http"
connection?
You can directly load to the backend Lucene using LuSql[1]. It is
faster than Solr, sometimes as much as an order of magnitude faster.
Disclosure: I am the author of LuSql
-Glen
http://zzzoot.blogspot.com/
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
2009/7/1 Francis Yakin <fy...@liquid.com>:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Glen Newton <gl...@gmail.com>.
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Is this LuSql is free? Is that an open source.
LuSql is an Open Source project.
> Is that requires a separate machine with Solr Master
LuSql is a Java application that runs on the command line. It connects
to a the database using JDBC and creates a local Lucene index, based
on the configuration you supply to it.
> I forgot to tell you that we have Master/Slaves environment of Solr.
>
> The Database is running Oracle and it's separate machine that running in different network than Master and Slaves Solr(There is a firewall between Oracle machine and Solr Machines).
> If we have LuSql Machine, do you think it's better to put into the same network with DataBase machine or Solr machines?
LuSql is heavily multi-threaded, and can suck up the resources of all
cores (this is why it runs so fast), so you need to decide if this is
not appropriate for your database machine (i.e. if it is a production
machine). You can isolate LuSql to specific cores using something
like numactl http://www.linuxmanpages.com/man8/numactl.8.php
> Do I need to create a sql script to get the data from Oarcle and loading it using LuSql and convert it to Lucene index, and how solr master will get that data?
LuSql reads from Oracle and writes to a Lucene index. You just need to
give LuSql a configuration that has it generate the appropriate index
for Solr.
thanks,
Glen
http://zzzoot.blogspot.com/search?q=lucene
>
>
> Thanks
>
> Francis
>
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Thursday, July 02, 2009 8:22 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> LuSql can be found here:
> http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
> User Manual:
> http://cuvier.cisti.nrc.ca/~gnewton/lusql/v0.9/lusqlManual.pdf.html
>
> LuSql can communicate directly with Oracle and create a Lucene index for you.
> Of course - as mentioned by other posters - you need to make sure the
> versions of Lucene and Solr are compatible (use same jars), you use
> the same Analyzers, and you create the appropriate 'schema' that Solr
> understands.
>
> -glen
>
> 2009/7/2 Francis Yakin <fy...@liquid.com>:
>>
>> Glen,
>>
>> Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
>> SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
>>
>> My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
>>
>> You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
>>
>> Thanks
>>
>> Francis
>>
>> -----Original Message-----
>> From: Glen Newton [mailto:glen.newton@gmail.com]
>> Sent: Wednesday, July 01, 2009 8:06 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Is there any other way to load the index beside using "http" connection?
>>
>> You can directly load to the backend Lucene using LuSql[1]. It is
>> faster than Solr, sometimes as much as an order of magnitude faster.
>>
>> Disclosure: I am the author of LuSql
>>
>> -Glen
>> http://zzzoot.blogspot.com/
>>
>> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>>
>> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>>
>>> We have several thousands of xml files in database that we load it to solr master
>>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>>
>>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>>
>>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>>
>>> Any input or help will be much appreciated.
>>>
>>>
>>> Thanks
>>>
>>> Francis
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>>
>> -
>>
>
>
>
> --
>
> -
>
--
-
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Glen,
Is this LuSql is free? Is that an open source.
Is that requires a separate machine with Solr Master
I forgot to tell you that we have Master/Slaves environment of Solr.
The Database is running Oracle and it's separate machine that running in different network than Master and Slaves Solr(There is a firewall between Oracle machine and Solr Machines).
If we have LuSql Machine, do you think it's better to put into the same network with DataBase machine or Solr machines?
Do I need to create a sql script to get the data from Oarcle and loading it using LuSql and convert it to Lucene index, and how solr master will get that data?
Thanks
Francis
-----Original Message-----
From: Glen Newton [mailto:glen.newton@gmail.com]
Sent: Thursday, July 02, 2009 8:22 AM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
LuSql can be found here:
http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
User Manual:
http://cuvier.cisti.nrc.ca/~gnewton/lusql/v0.9/lusqlManual.pdf.html
LuSql can communicate directly with Oracle and create a Lucene index for you.
Of course - as mentioned by other posters - you need to make sure the
versions of Lucene and Solr are compatible (use same jars), you use
the same Analyzers, and you create the appropriate 'schema' that Solr
understands.
-glen
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
> SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
>
> My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
>
> You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
>
> Thanks
>
> Francis
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Wednesday, July 01, 2009 8:06 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> You can directly load to the backend Lucene using LuSql[1]. It is
> faster than Solr, sometimes as much as an order of magnitude faster.
>
> Disclosure: I am the author of LuSql
>
> -Glen
> http://zzzoot.blogspot.com/
>
> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>
> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>
>> We have several thousands of xml files in database that we load it to solr master
>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>
>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>
>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>
>> Any input or help will be much appreciated.
>>
>>
>> Thanks
>>
>> Francis
>>
>>
>>
>>
>
>
>
> --
>
> -
>
--
-
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Glen Newton <gl...@gmail.com>.
LuSql can be found here:
http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
User Manual:
http://cuvier.cisti.nrc.ca/~gnewton/lusql/v0.9/lusqlManual.pdf.html
LuSql can communicate directly with Oracle and create a Lucene index for you.
Of course - as mentioned by other posters - you need to make sure the
versions of Lucene and Solr are compatible (use same jars), you use
the same Analyzers, and you create the appropriate 'schema' that Solr
understands.
-glen
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
> SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
>
> My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
>
> You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
>
> Thanks
>
> Francis
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Wednesday, July 01, 2009 8:06 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> You can directly load to the backend Lucene using LuSql[1]. It is
> faster than Solr, sometimes as much as an order of magnitude faster.
>
> Disclosure: I am the author of LuSql
>
> -Glen
> http://zzzoot.blogspot.com/
>
> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>
> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>
>> We have several thousands of xml files in database that we load it to solr master
>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>
>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>
>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>
>> Any input or help will be much appreciated.
>>
>>
>> Thanks
>>
>> Francis
>>
>>
>>
>>
>
>
>
> --
>
> -
>
--
-
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Glen,
Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
Thanks
Francis
-----Original Message-----
From: Glen Newton [mailto:glen.newton@gmail.com]
Sent: Wednesday, July 01, 2009 8:06 PM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
You can directly load to the backend Lucene using LuSql[1]. It is
faster than Solr, sometimes as much as an order of magnitude faster.
Disclosure: I am the author of LuSql
-Glen
http://zzzoot.blogspot.com/
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
2009/7/1 Francis Yakin <fy...@liquid.com>:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Glen Newton <gl...@gmail.com>.
> Are you saying that we have to use LuSql replacing our Solr?
To load your data: Yes, it is an option
To search your data: No, LuSql is only a loading tool
-glen
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Are you saying that we have to use LuSql replacing our Solr?
>
> Francis
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Wednesday, July 01, 2009 8:06 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> You can directly load to the backend Lucene using LuSql[1]. It is
> faster than Solr, sometimes as much as an order of magnitude faster.
>
> Disclosure: I am the author of LuSql
>
> -Glen
> http://zzzoot.blogspot.com/
>
> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>
> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>
>> We have several thousands of xml files in database that we load it to solr master
>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>
>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>
>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>
>> Any input or help will be much appreciated.
>>
>>
>> Thanks
>>
>> Francis
>>
>>
>>
>>
>
>
>
> --
>
> -
>
--
-
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Glen,
Are you saying that we have to use LuSql replacing our Solr?
Francis
-----Original Message-----
From: Glen Newton [mailto:glen.newton@gmail.com]
Sent: Wednesday, July 01, 2009 8:06 PM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
You can directly load to the backend Lucene using LuSql[1]. It is
faster than Solr, sometimes as much as an order of magnitude faster.
Disclosure: I am the author of LuSql
-Glen
http://zzzoot.blogspot.com/
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
2009/7/1 Francis Yakin <fy...@liquid.com>:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-