You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Francis Yakin <fy...@liquid.com> on 2009/07/02 00:07:12 UTC
Is there any other way to load the index beside using "http"
connection?
We have several thousands of xml files in database that we load it to solr master
The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
We are experiencing issue with close/open connection in the firewall and very very slow.
Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
Any input or help will be much appreciated.
Thanks
Francis
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Jul 2, 2009 at 10:24 AM, Francis Yakin <fy...@liquid.com> wrote:
>
> This is only for version 1.3.0? We are running 1.2.0 currently.
>
Yes, DIH is available since 1.3 only.
--
Regards,
Shalin Shekhar Mangar.
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Thanks Noble!
This is only for version 1.3.0? We are running 1.2.0 currently.
Francis
-----Original Message-----
From: noble.paul@gmail.com [mailto:noble.paul@gmail.com] On Behalf Of Noble Paul ??????? ??????
Sent: Wednesday, July 01, 2009 9:43 PM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
did you explore DIH http://wiki.apache.org/solr/DataImportHandler
it has features to import from Db, xml files etc
On Thu, Jul 2, 2009 at 3:37 AM, Francis Yakin<fy...@liquid.com> wrote:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
did you explore DIH http://wiki.apache.org/solr/DataImportHandler
it has features to import from Db, xml files etc
On Thu, Jul 2, 2009 at 3:37 AM, Francis Yakin<fy...@liquid.com> wrote:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Otis,
Do you have the document how to do those things that you mentioned?
How about if I don't want use HHTP at all? Or we have no other option that we have to use HHTP to transfer the XML files to Solr master from Db box?
Thanks
Francis
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Wednesday, July 01, 2009 8:01 PM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
Francis,
There are a number of things you can do to make indexing over HTTP faster.
You can also import documents as csv data/file.
Finally, you can use EmbeddedSolrServer.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Francis Yakin <fy...@liquid.com>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Wednesday, July 1, 2009 6:07:12 PM
> Subject: Is there any other way to load the index beside using "http" connection?
>
>
> We have several thousands of xml files in database that we load it to solr
> master
> The Database uses "http" connection and transfer those files to solr master.
> Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very
> very slow.
>
> Is there any other way to load the data/index from Database to solr master
> beside using http connection, so it means we just scp/ftp the xml file from
> Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
Re: Is there any other way to load the index beside using "http" connection?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Francis,
I think both of these are on the Solr Wiki. You'll have to figure out how to export from DB yourself, and you'll probably write a script/tool to read the export and rewrite it in the csv format.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Francis Yakin <fy...@liquid.com>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Thursday, July 2, 2009 12:26:14 AM
> Subject: RE: Is there any other way to load the index beside using "http" connection?
>
>
> How you import the documents as csv data/file from Oracle Database to Sol
> master( they are two different machines)?
>
> And you have the doc for using EmbeddedSolrServer?
>
> Thanks Otis!
>
> Francis
>
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Sent: Wednesday, July 01, 2009 8:01 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http"
> connection?
>
>
> Francis,
>
> There are a number of things you can do to make indexing over HTTP faster.
> You can also import documents as csv data/file.
> Finally, you can use EmbeddedSolrServer.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: Francis Yakin
> > To: "solr-user@lucene.apache.org"
> > Sent: Wednesday, July 1, 2009 6:07:12 PM
> > Subject: Is there any other way to load the index beside using "http"
> connection?
> >
> >
> > We have several thousands of xml files in database that we load it to solr
> > master
> > The Database uses "http" connection and transfer those files to solr master.
> > Solr then translate xml files to their lindex.
> >
> > We are experiencing issue with close/open connection in the firewall and very
> > very slow.
> >
> > Is there any other way to load the data/index from Database to solr master
> > beside using http connection, so it means we just scp/ftp the xml file from
> > Database system to solr master and let solr convert those to lucene indexes?
> >
> > Any input or help will be much appreciated.
> >
> >
> > Thanks
> >
> > Francis
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
How you import the documents as csv data/file from Oracle Database to Sol master( they are two different machines)?
And you have the doc for using EmbeddedSolrServer?
Thanks Otis!
Francis
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Wednesday, July 01, 2009 8:01 PM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
Francis,
There are a number of things you can do to make indexing over HTTP faster.
You can also import documents as csv data/file.
Finally, you can use EmbeddedSolrServer.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Francis Yakin <fy...@liquid.com>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Wednesday, July 1, 2009 6:07:12 PM
> Subject: Is there any other way to load the index beside using "http" connection?
>
>
> We have several thousands of xml files in database that we load it to solr
> master
> The Database uses "http" connection and transfer those files to solr master.
> Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very
> very slow.
>
> Is there any other way to load the data/index from Database to solr master
> beside using http connection, so it means we just scp/ftp the xml file from
> Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
Re: Is there any other way to load the index beside using "http" connection?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Francis,
There are a number of things you can do to make indexing over HTTP faster.
You can also import documents as csv data/file.
Finally, you can use EmbeddedSolrServer.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Francis Yakin <fy...@liquid.com>
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Sent: Wednesday, July 1, 2009 6:07:12 PM
> Subject: Is there any other way to load the index beside using "http" connection?
>
>
> We have several thousands of xml files in database that we load it to solr
> master
> The Database uses "http" connection and transfer those files to solr master.
> Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very
> very slow.
>
> Is there any other way to load the data/index from Database to solr master
> beside using http connection, so it means we just scp/ftp the xml file from
> Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Norberto Meijome <nu...@gmail.com>.
On Tue, 7 Jul 2009 13:54:07 -0700
Francis Yakin <fy...@liquid.com> wrote:
[...]
> much on our setup.
>
> Like said we have file name "test.xml" which come from SQL output , we put it
> locally on the solr server under "/opt/test.xml"
>
> So, I need to execute the commands from solr system to add and update this to
> the solr data/indexes.
>
> What commands do I have to use, for example the xml file
> named" /opt/test.xml" ?
>
Francis,
as much as we can tell you the answer, have you tried reading the documentation in the wiki, and the example setup bundled with SOLR?
Most, if not ALL your questions, are answered there.
Good luck,
B
_________________________
{Beto|Norberto|Numard} Meijome
Computers are like air conditioners; they can't do their job properly if you open windows.
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Norberto,
You said last week:
"why not generate your SQL output directly into your oracle server as a file,
upload the file to your SOLR server? Then the data file is local to your SOLR
server , you will bypass any WAN and firewall you may be having. (or some
variation of it, sql -> SOLR server as file, etc..)"
I think this is the best solution that we are going to without changing too much on our setup.
Like said we have file name "test.xml" which come from SQL output , we put it locally on the solr server under "/opt/test.xml"
So, I need to execute the commands from solr system to add and update this to the solr data/indexes.
What commands do I have to use, for example the xml file named" /opt/test.xml" ?
Thanks
Francis
-----Original Message-----
From: Norberto Meijome [mailto:numardbsd@gmail.com]
Sent: Sunday, July 05, 2009 3:57 AM
To: Francis Yakin
Cc: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
On Thu, 2 Jul 2009 11:02:28 -0700
Francis Yakin <fy...@liquid.com> wrote:
> Norberto, Thanks for your input.
>
> What do you mean with "Have you tried connecting to SOLR over HTTP from
> localhost, therefore avoiding any firewall issues and network latency ? it
> should work a LOT faster than from a remote site." ?
>
>
> Here are how our servers lay out:
>
> 1) Database ( Oracle ) is running on separate machine
> 2) Solr master is running on separate machine by itself
> 3) 6 solr slaves ( these 6 pulll the index from master using rsync)
>
> We have a SQL(Oracle) script to post the data/index from Oracle Database
> machine to Solr Master over http. We wrote those script(Someone in Oracle
> Database administrator write it).
You said in your other email you are having issues with slow transfers between
1) and 2). Your subject relates to the data transfer between 1) and 2, - 2) and
3) is irrelevant to this part.
My question (what you quoted above) relates to the point you made about it
being slow ( WHY is it slow?), and issues with opening so many connections
through firewall. so, I'll rephrase my question (see below...)
[....]
>
> We can not do localhost since it's solr is not running on Oracle machine.
why not generate your SQL output directly into your oracle server as a file,
upload the file to your SOLR server? Then the data file is local to your SOLR
server , you will bypass any WAN and firewall you may be having. (or some
variation of it, sql -> SOLR server as file, etc..)
Any speed issues that are rooted in the fact that you are posting via
HTTP (vs embedded solr or DIH) aren't going to go away. But it's the simpler
approach without changing too much of your current setup.
> Another alternative that we think of is to transform XML into CSV and
> import/export it.
>
> How about if LUSQL, some mentioned about this? Is this apps free(open source)
> application? Do you have any experience with this apps?
Not i, sorry.
Have you looked into DIH? It's designed for this kind of work.
B
_________________________
{Beto|Norberto|Numard} Meijome
"Great spirits have often encountered violent opposition from mediocre minds."
Albert Einstein
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Tue, Jul 7, 2009 at 1:50 PM, Francis Yakin<fy...@liquid.com> wrote:
> yeah, It works now.
>
> How can I verify if the new CSV file get uploaded?
point your browser at
http://localhost:8983/solr/admin/stats.jsp
Check out the "UPDATE HANDLERS" section
-Yonik
http://www.lucidimagination.com
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
yeah, It works now.
How can I verify if the new CSV file get uploaded?
Thanks
Francis
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
Sent: Tuesday, July 07, 2009 10:49 AM
To: solr-user@lucene.apache.org
Cc: Norberto Meijome
Subject: Re: Is there any other way to load the index beside using "http" connection?
The double quotes around the ampersand don't belong there.
I think that UTF8 should also be the default, so the following should also work:
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv'
-Yonik
http://www.lucidimagination.com
On Tue, Jul 7, 2009 at 1:37 PM, Francis Yakin<fy...@liquid.com> wrote:
>
> I did try:
>
> curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv"&"stream.contentType=text/plain;charset=utf-8'
>
> It doesn't work
>
> Francis
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
> Sent: Tuesday, July 07, 2009 4:59 AM
> To: solr-user@lucene.apache.org
> Cc: Norberto Meijome
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> Look at the error - it's bash (your command line shell) complaining.
> The '&' terminates one command and puts it in the background.
> Surrounding the command with quotes will get you one step closer:
>
> curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8'
>
> -Yonik
> http://www.lucidimagination.com
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
The double quotes around the ampersand don't belong there.
I think that UTF8 should also be the default, so the following should also work:
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv'
-Yonik
http://www.lucidimagination.com
On Tue, Jul 7, 2009 at 1:37 PM, Francis Yakin<fy...@liquid.com> wrote:
>
> I did try:
>
> curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv"&"stream.contentType=text/plain;charset=utf-8'
>
> It doesn't work
>
> Francis
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
> Sent: Tuesday, July 07, 2009 4:59 AM
> To: solr-user@lucene.apache.org
> Cc: Norberto Meijome
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> Look at the error - it's bash (your command line shell) complaining.
> The '&' terminates one command and puts it in the background.
> Surrounding the command with quotes will get you one step closer:
>
> curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8'
>
> -Yonik
> http://www.lucidimagination.com
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
With
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8'
No errors now.
But , how can I verify if the update happening?
Thanks
Francis
-----Original Message-----
From: Francis Yakin [mailto:fyakin@liquid.com]
Sent: Tuesday, July 07, 2009 10:37 AM
To: 'solr-user@lucene.apache.org'; 'yonik@lucidimagination.com'
Cc: Norberto Meijome
Subject: RE: Is there any other way to load the index beside using "http" connection?
I did try:
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv"&"stream.contentType=text/plain;charset=utf-8'
It doesn't work
Francis
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
Sent: Tuesday, July 07, 2009 4:59 AM
To: solr-user@lucene.apache.org
Cc: Norberto Meijome
Subject: Re: Is there any other way to load the index beside using "http" connection?
Look at the error - it's bash (your command line shell) complaining.
The '&' terminates one command and puts it in the background.
Surrounding the command with quotes will get you one step closer:
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8'
-Yonik
http://www.lucidimagination.com
On Mon, Jul 6, 2009 at 2:11 PM, Francis Yakin<fy...@liquid.com> wrote:
>
> Ok, I have a CSV file(called it test.csv) from database.
>
> When I tried to upload this file to solr using this cmd, I got "stream.contentType=text/plain: No such file or directory" error
>
> curl http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8
>
> -bash: stream.contentType=text/plain: No such file or directory
> undefined field cat
>
> What did I do wrong?
>
> Francis
>
> -----Original Message-----
> From: Norberto Meijome [mailto:numardbsd@gmail.com]
> Sent: Monday, July 06, 2009 11:01 AM
> To: Francis Yakin
> Cc: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> On Mon, 6 Jul 2009 09:56:03 -0700
> Francis Yakin <fy...@liquid.com> wrote:
>
>> Norberto,
>>
>> Thanks, I think my questions is:
>>
>> >>why not generate your SQL output directly into your oracle server as a file
>>
>> What type of file is this?
>>
>>
>
> a file in a format that you can then import into SOLR.
>
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Gravity cannot be blamed for people falling in love."
> Albert Einstein
>
> I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
>
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
I did try:
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv"&"stream.contentType=text/plain;charset=utf-8'
It doesn't work
Francis
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
Sent: Tuesday, July 07, 2009 4:59 AM
To: solr-user@lucene.apache.org
Cc: Norberto Meijome
Subject: Re: Is there any other way to load the index beside using "http" connection?
Look at the error - it's bash (your command line shell) complaining.
The '&' terminates one command and puts it in the background.
Surrounding the command with quotes will get you one step closer:
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8'
-Yonik
http://www.lucidimagination.com
On Mon, Jul 6, 2009 at 2:11 PM, Francis Yakin<fy...@liquid.com> wrote:
>
> Ok, I have a CSV file(called it test.csv) from database.
>
> When I tried to upload this file to solr using this cmd, I got "stream.contentType=text/plain: No such file or directory" error
>
> curl http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8
>
> -bash: stream.contentType=text/plain: No such file or directory
> undefined field cat
>
> What did I do wrong?
>
> Francis
>
> -----Original Message-----
> From: Norberto Meijome [mailto:numardbsd@gmail.com]
> Sent: Monday, July 06, 2009 11:01 AM
> To: Francis Yakin
> Cc: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> On Mon, 6 Jul 2009 09:56:03 -0700
> Francis Yakin <fy...@liquid.com> wrote:
>
>> Norberto,
>>
>> Thanks, I think my questions is:
>>
>> >>why not generate your SQL output directly into your oracle server as a file
>>
>> What type of file is this?
>>
>>
>
> a file in a format that you can then import into SOLR.
>
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Gravity cannot be blamed for people falling in love."
> Albert Einstein
>
> I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
>
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
Look at the error - it's bash (your command line shell) complaining.
The '&' terminates one command and puts it in the background.
Surrounding the command with quotes will get you one step closer:
curl 'http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8'
-Yonik
http://www.lucidimagination.com
On Mon, Jul 6, 2009 at 2:11 PM, Francis Yakin<fy...@liquid.com> wrote:
>
> Ok, I have a CSV file(called it test.csv) from database.
>
> When I tried to upload this file to solr using this cmd, I got "stream.contentType=text/plain: No such file or directory" error
>
> curl http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8
>
> -bash: stream.contentType=text/plain: No such file or directory
> undefined field cat
>
> What did I do wrong?
>
> Francis
>
> -----Original Message-----
> From: Norberto Meijome [mailto:numardbsd@gmail.com]
> Sent: Monday, July 06, 2009 11:01 AM
> To: Francis Yakin
> Cc: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> On Mon, 6 Jul 2009 09:56:03 -0700
> Francis Yakin <fy...@liquid.com> wrote:
>
>> Norberto,
>>
>> Thanks, I think my questions is:
>>
>> >>why not generate your SQL output directly into your oracle server as a file
>>
>> What type of file is this?
>>
>>
>
> a file in a format that you can then import into SOLR.
>
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Gravity cannot be blamed for people falling in love."
> Albert Einstein
>
> I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
>
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Marcus Herou <ma...@tailsweep.com>.
Out of my head... but are you not supposed to active the stream-handler in
SOLR ? Think it is documented...
Cheers
//Marcus
On Mon, Jul 6, 2009 at 8:55 PM, Francis Yakin <fy...@liquid.com> wrote:
> Yes, I uploaded the CSV file that I get it from Database then I ran that
> cmd and I have the error.
>
> Any suggestions?
>
> Thanks
>
> Francis
>
> -----Original Message-----
> From: NitinMalik [mailto:malik.nitin@yahoo.com]
> Sent: Monday, July 06, 2009 11:32 AM
> To: solr-user@lucene.apache.org
> Subject: RE: Is there any other way to load the index beside using "http"
> connection?
>
>
> Hi Francis,
>
> I have experienced that update stream handler (for a xml file in my case)
> worked only for Solr running on the same machine. I also got same error
> when
> I tried to update the documents on a remote Solr instance.
>
> Regards
> Nitin
>
>
> Francis Yakin wrote:
> >
> >
> > Ok, I have a CSV file(called it test.csv) from database.
> >
> > When I tried to upload this file to solr using this cmd, I got
> > "stream.contentType=text/plain: No such file or directory" error
> >
> > curl
> >
> http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8
> >
> > -bash: stream.contentType=text/plain: No such file or directory
> > undefined field cat
> >
> > What did I do wrong?
> >
> > Francis
> >
> > -----Original Message-----
> > From: Norberto Meijome [mailto:numardbsd@gmail.com]
> > Sent: Monday, July 06, 2009 11:01 AM
> > To: Francis Yakin
> > Cc: solr-user@lucene.apache.org
> > Subject: Re: Is there any other way to load the index beside using "http"
> > connection?
> >
> > On Mon, 6 Jul 2009 09:56:03 -0700
> > Francis Yakin <fy...@liquid.com> wrote:
> >
> >> Norberto,
> >>
> >> Thanks, I think my questions is:
> >>
> >> >>why not generate your SQL output directly into your oracle server as a
> >> file
> >>
> >> What type of file is this?
> >>
> >>
> >
> > a file in a format that you can then import into SOLR.
> >
> > _________________________
> > {Beto|Norberto|Numard} Meijome
> >
> > "Gravity cannot be blamed for people falling in love."
> > Albert Einstein
> >
> > I speak for myself, not my employer. Contents may be hot. Slippery when
> > wet. Reading disclaimers makes you go blind. Writing them is worse. You
> > have been Warned.
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Is-there-any-other-way-to-load-the-index-beside-using-%22http%22-connection--tp24297934p24360603.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Yes, I uploaded the CSV file that I get it from Database then I ran that cmd and I have the error.
Any suggestions?
Thanks
Francis
-----Original Message-----
From: NitinMalik [mailto:malik.nitin@yahoo.com]
Sent: Monday, July 06, 2009 11:32 AM
To: solr-user@lucene.apache.org
Subject: RE: Is there any other way to load the index beside using "http" connection?
Hi Francis,
I have experienced that update stream handler (for a xml file in my case)
worked only for Solr running on the same machine. I also got same error when
I tried to update the documents on a remote Solr instance.
Regards
Nitin
Francis Yakin wrote:
>
>
> Ok, I have a CSV file(called it test.csv) from database.
>
> When I tried to upload this file to solr using this cmd, I got
> "stream.contentType=text/plain: No such file or directory" error
>
> curl
> http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8
>
> -bash: stream.contentType=text/plain: No such file or directory
> undefined field cat
>
> What did I do wrong?
>
> Francis
>
> -----Original Message-----
> From: Norberto Meijome [mailto:numardbsd@gmail.com]
> Sent: Monday, July 06, 2009 11:01 AM
> To: Francis Yakin
> Cc: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http"
> connection?
>
> On Mon, 6 Jul 2009 09:56:03 -0700
> Francis Yakin <fy...@liquid.com> wrote:
>
>> Norberto,
>>
>> Thanks, I think my questions is:
>>
>> >>why not generate your SQL output directly into your oracle server as a
>> file
>>
>> What type of file is this?
>>
>>
>
> a file in a format that you can then import into SOLR.
>
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Gravity cannot be blamed for people falling in love."
> Albert Einstein
>
> I speak for myself, not my employer. Contents may be hot. Slippery when
> wet. Reading disclaimers makes you go blind. Writing them is worse. You
> have been Warned.
>
>
--
View this message in context: http://www.nabble.com/Is-there-any-other-way-to-load-the-index-beside-using-%22http%22-connection--tp24297934p24360603.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Is there any other way to load the index beside using "http"
connection?
Posted by NitinMalik <ma...@yahoo.com>.
Hi Francis,
I have experienced that update stream handler (for a xml file in my case)
worked only for Solr running on the same machine. I also got same error when
I tried to update the documents on a remote Solr instance.
Regards
Nitin
Francis Yakin wrote:
>
>
> Ok, I have a CSV file(called it test.csv) from database.
>
> When I tried to upload this file to solr using this cmd, I got
> "stream.contentType=text/plain: No such file or directory" error
>
> curl
> http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8
>
> -bash: stream.contentType=text/plain: No such file or directory
> undefined field cat
>
> What did I do wrong?
>
> Francis
>
> -----Original Message-----
> From: Norberto Meijome [mailto:numardbsd@gmail.com]
> Sent: Monday, July 06, 2009 11:01 AM
> To: Francis Yakin
> Cc: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http"
> connection?
>
> On Mon, 6 Jul 2009 09:56:03 -0700
> Francis Yakin <fy...@liquid.com> wrote:
>
>> Norberto,
>>
>> Thanks, I think my questions is:
>>
>> >>why not generate your SQL output directly into your oracle server as a
>> file
>>
>> What type of file is this?
>>
>>
>
> a file in a format that you can then import into SOLR.
>
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Gravity cannot be blamed for people falling in love."
> Albert Einstein
>
> I speak for myself, not my employer. Contents may be hot. Slippery when
> wet. Reading disclaimers makes you go blind. Writing them is worse. You
> have been Warned.
>
>
--
View this message in context: http://www.nabble.com/Is-there-any-other-way-to-load-the-index-beside-using-%22http%22-connection--tp24297934p24360603.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Ok, I have a CSV file(called it test.csv) from database.
When I tried to upload this file to solr using this cmd, I got "stream.contentType=text/plain: No such file or directory" error
curl http://localhost:8983/solr/update/csv?stream.file=/opt/apache-1.2.0/example/exampledocs/test.csv&stream.contentType=text/plain;charset=utf-8
-bash: stream.contentType=text/plain: No such file or directory
undefined field cat
What did I do wrong?
Francis
-----Original Message-----
From: Norberto Meijome [mailto:numardbsd@gmail.com]
Sent: Monday, July 06, 2009 11:01 AM
To: Francis Yakin
Cc: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
On Mon, 6 Jul 2009 09:56:03 -0700
Francis Yakin <fy...@liquid.com> wrote:
> Norberto,
>
> Thanks, I think my questions is:
>
> >>why not generate your SQL output directly into your oracle server as a file
>
> What type of file is this?
>
>
a file in a format that you can then import into SOLR.
_________________________
{Beto|Norberto|Numard} Meijome
"Gravity cannot be blamed for people falling in love."
Albert Einstein
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Norberto Meijome <nu...@gmail.com>.
On Mon, 6 Jul 2009 09:56:03 -0700
Francis Yakin <fy...@liquid.com> wrote:
> Norberto,
>
> Thanks, I think my questions is:
>
> >>why not generate your SQL output directly into your oracle server as a file
>
> What type of file is this?
>
>
a file in a format that you can then import into SOLR.
_________________________
{Beto|Norberto|Numard} Meijome
"Gravity cannot be blamed for people falling in love."
Albert Einstein
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Norberto,
Thanks, I think my questions is:
>>why not generate your SQL output directly into your oracle server as a file
What type of file is this?
Thanks again for your help.
Francis
-----Original Message-----
From: Norberto Meijome [mailto:numardbsd@gmail.com]
Sent: Monday, July 06, 2009 4:33 AM
To: Francis Yakin
Cc: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
On Sun, 5 Jul 2009 10:28:16 -0700
Francis Yakin <fy...@liquid.com> wrote:
[...]>
> >upload the file to your SOLR server? Then the data file is local to your SOLR
> >server , you will bypass any WAN and firewall you may be having. (or some
> >variation of it, sql -> SOLR server as file, etc..)
>
> How we upload the file? Do we need to convert the data file to Lucene Index
> first? And Documentation how we do this?
pick your poison... rsync? ftp? scp ?
B
_________________________
{Beto|Norberto|Numard} Meijome
"The freethinking of one age is the common sense of the next."
Matthew Arnold
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Norberto Meijome <nu...@gmail.com>.
On Sun, 5 Jul 2009 10:28:16 -0700
Francis Yakin <fy...@liquid.com> wrote:
[...]>
> >upload the file to your SOLR server? Then the data file is local to your SOLR
> >server , you will bypass any WAN and firewall you may be having. (or some
> >variation of it, sql -> SOLR server as file, etc..)
>
> How we upload the file? Do we need to convert the data file to Lucene Index
> first? And Documentation how we do this?
pick your poison... rsync? ftp? scp ?
B
_________________________
{Beto|Norberto|Numard} Meijome
"The freethinking of one age is the common sense of the next."
Matthew Arnold
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Thanks Marcus,
I will give a try to a test machine first.
Francis
-----Original Message-----
From: Marcus Herou [mailto:marcus.herou@tailsweep.com]
Sent: Sunday, July 05, 2009 12:37 PM
To: solr-user@lucene.apache.org
Cc: Norberto Meijome
Subject: Re: Is there any other way to load the index beside using "http" connection?
Sharing some of our exports from DB to solr. Note: many of the statements
below might not work due to clip-clip.
$SOLR_HOME/conf/dataConfig.xml
<dataConfig>
<dataSource name="myfilereader" type="FileDataSource" />
<document>
<entity name="jc" rootEntity="false" dataSource="null"
processor="FileListEntityProcessor"
fileName="^.*\.xml$" recursive="false"
baseDir="$dumpdir"
>
<entity name="x"
pk="uid"
processor="XPathEntityProcessor"
url="${jc.fileAbsolutePath}"
forEach="/entries/entry"
transformer="DateFormatTransformer"
stream="true"
dataSource="myfilereader">
<field column="uid" xpath="/entries/entry/uid"
/>
<!-- sorry hiding the rest-->
</entity>
</entity>
</document>
</dataConfig>
# Then add this to solrconfig.xml
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</requestHandler>
restart solr
Issue a mysql dump
mysql --xml -uXXX -pXXX -hXXX -DXXX -e "select MD5(link) as
uid,DATE_FORMAT(publishedDate, \"%Y-%m-%dT%H:%i:%sZ\") as publishedDate from
X" > $dumpdir/dump.xml
# Warning: Note the clean command which will wipe your index...
GET "http://
$server:$port/$path/dataimport?command=full-import&clean=true&optimize=true"
Hope this helps out some.
Cheers
//Marcus
On Sun, Jul 5, 2009 at 7:28 PM, Francis Yakin <fy...@liquid.com> wrote:
> Norberto,
>
> Yes, DIH is one of the option we think to use, but it's required 1.3.0 and
> above and currently we are running Sol 1.2.0.
>
> I am thinking to use CSV file(Convert the XML to CSV format in Database
> machine( , then transport that CSV file to solr box.
> In Solr we run the update to convert the CSV file to Lucene index.
>
> Also , we think the one you suggested, note my question below:
>
> >
> >why not generate your SQL output directly into your oracle server as a
> file,
> question: What type of file is this(XML or CSV)?
>
> >upload the file to your SOLR server? Then the data file is local to your
> SOLR
> >server , you will bypass any WAN and firewall you may be having. (or some
> >variation of it, sql -> SOLR server as file, etc..)
>
> How we upload the file? Do we need to convert the data file to Lucene Index
> first?
> And Documentation how we do this?
>
> >Any speed issues that are rooted in the fact that you are posting via
> >HTTP (vs embedded solr or DIH) aren't going to go away. But it's the
> simpler
> >approach without changing too much of your current setup.
>
>
> -----Original Message-----
> From: Norberto Meijome [mailto:numardbsd@gmail.com]
> Sent: Sunday, July 05, 2009 3:57 AM
> To: Francis Yakin
> Cc: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http"
> connection?
>
> On Thu, 2 Jul 2009 11:02:28 -0700
> Francis Yakin <fy...@liquid.com> wrote:
>
> > Norberto, Thanks for your input.
> >
> > What do you mean with "Have you tried connecting to SOLR over HTTP from
> > localhost, therefore avoiding any firewall issues and network latency ?
> it
> > should work a LOT faster than from a remote site." ?
> >
> >
> > Here are how our servers lay out:
> >
> > 1) Database ( Oracle ) is running on separate machine
> > 2) Solr master is running on separate machine by itself
> > 3) 6 solr slaves ( these 6 pulll the index from master using rsync)
> >
> > We have a SQL(Oracle) script to post the data/index from Oracle Database
> > machine to Solr Master over http. We wrote those script(Someone in Oracle
> > Database administrator write it).
>
> You said in your other email you are having issues with slow transfers
> between
> 1) and 2). Your subject relates to the data transfer between 1) and 2, - 2)
> and
> 3) is irrelevant to this part.
>
> My question (what you quoted above) relates to the point you made about it
> being slow ( WHY is it slow?), and issues with opening so many connections
> through firewall. so, I'll rephrase my question (see below...)
>
> [....]
> >
> > We can not do localhost since it's solr is not running on Oracle machine.
>
> why not generate your SQL output directly into your oracle server as a
> file,
> upload the file to your SOLR server? Then the data file is local to your
> SOLR
> server , you will bypass any WAN and firewall you may be having. (or some
> variation of it, sql -> SOLR server as file, etc..)
>
> Any speed issues that are rooted in the fact that you are posting via
> HTTP (vs embedded solr or DIH) aren't going to go away. But it's the
> simpler
> approach without changing too much of your current setup.
>
>
> > Another alternative that we think of is to transform XML into CSV and
> > import/export it.
> >
> > How about if LUSQL, some mentioned about this? Is this apps free(open
> source)
> > application? Do you have any experience with this apps?
>
> Not i, sorry.
>
> Have you looked into DIH? It's designed for this kind of work.
>
> B
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Great spirits have often encountered violent opposition from mediocre
> minds."
> Albert Einstein
>
> I speak for myself, not my employer. Contents may be hot. Slippery when
> wet.
> Reading disclaimers makes you go blind. Writing them is worse. You have
> been
> Warned.
>
>
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Marcus Herou <ma...@tailsweep.com>.
Yes exactly just being friendly sharing a working routine. Took me some
hours to figure out DIH myself at the time.
//Marcus
On Mon, Jul 6, 2009 at 1:32 PM, Norberto Meijome <nu...@gmail.com>wrote:
> On Sun, 5 Jul 2009 21:36:35 +0200
> Marcus Herou <ma...@tailsweep.com> wrote:
>
> > Sharing some of our exports from DB to solr. Note: many of the statements
> > below might not work due to clip-clip.
>
> thx Marcus - but that's a DIH config right? :)
> b
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "I respect faith, but doubt is what gives you an education."
> Wilson Mizner
>
> I speak for myself, not my employer. Contents may be hot. Slippery when
> wet. Reading disclaimers makes you go blind. Writing them is worse. You have
> been Warned.
>
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Norberto Meijome <nu...@gmail.com>.
On Sun, 5 Jul 2009 21:36:35 +0200
Marcus Herou <ma...@tailsweep.com> wrote:
> Sharing some of our exports from DB to solr. Note: many of the statements
> below might not work due to clip-clip.
thx Marcus - but that's a DIH config right? :)
b
_________________________
{Beto|Norberto|Numard} Meijome
"I respect faith, but doubt is what gives you an education."
Wilson Mizner
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Marcus Herou <ma...@tailsweep.com>.
Sharing some of our exports from DB to solr. Note: many of the statements
below might not work due to clip-clip.
$SOLR_HOME/conf/dataConfig.xml
<dataConfig>
<dataSource name="myfilereader" type="FileDataSource" />
<document>
<entity name="jc" rootEntity="false" dataSource="null"
processor="FileListEntityProcessor"
fileName="^.*\.xml$" recursive="false"
baseDir="$dumpdir"
>
<entity name="x"
pk="uid"
processor="XPathEntityProcessor"
url="${jc.fileAbsolutePath}"
forEach="/entries/entry"
transformer="DateFormatTransformer"
stream="true"
dataSource="myfilereader">
<field column="uid" xpath="/entries/entry/uid"
/>
<!-- sorry hiding the rest-->
</entity>
</entity>
</document>
</dataConfig>
# Then add this to solrconfig.xml
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">data-config.xml</str>
</lst>
</requestHandler>
restart solr
Issue a mysql dump
mysql --xml -uXXX -pXXX -hXXX -DXXX -e "select MD5(link) as
uid,DATE_FORMAT(publishedDate, \"%Y-%m-%dT%H:%i:%sZ\") as publishedDate from
X" > $dumpdir/dump.xml
# Warning: Note the clean command which will wipe your index...
GET "http://
$server:$port/$path/dataimport?command=full-import&clean=true&optimize=true"
Hope this helps out some.
Cheers
//Marcus
On Sun, Jul 5, 2009 at 7:28 PM, Francis Yakin <fy...@liquid.com> wrote:
> Norberto,
>
> Yes, DIH is one of the option we think to use, but it's required 1.3.0 and
> above and currently we are running Sol 1.2.0.
>
> I am thinking to use CSV file(Convert the XML to CSV format in Database
> machine( , then transport that CSV file to solr box.
> In Solr we run the update to convert the CSV file to Lucene index.
>
> Also , we think the one you suggested, note my question below:
>
> >
> >why not generate your SQL output directly into your oracle server as a
> file,
> question: What type of file is this(XML or CSV)?
>
> >upload the file to your SOLR server? Then the data file is local to your
> SOLR
> >server , you will bypass any WAN and firewall you may be having. (or some
> >variation of it, sql -> SOLR server as file, etc..)
>
> How we upload the file? Do we need to convert the data file to Lucene Index
> first?
> And Documentation how we do this?
>
> >Any speed issues that are rooted in the fact that you are posting via
> >HTTP (vs embedded solr or DIH) aren't going to go away. But it's the
> simpler
> >approach without changing too much of your current setup.
>
>
> -----Original Message-----
> From: Norberto Meijome [mailto:numardbsd@gmail.com]
> Sent: Sunday, July 05, 2009 3:57 AM
> To: Francis Yakin
> Cc: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http"
> connection?
>
> On Thu, 2 Jul 2009 11:02:28 -0700
> Francis Yakin <fy...@liquid.com> wrote:
>
> > Norberto, Thanks for your input.
> >
> > What do you mean with "Have you tried connecting to SOLR over HTTP from
> > localhost, therefore avoiding any firewall issues and network latency ?
> it
> > should work a LOT faster than from a remote site." ?
> >
> >
> > Here are how our servers lay out:
> >
> > 1) Database ( Oracle ) is running on separate machine
> > 2) Solr master is running on separate machine by itself
> > 3) 6 solr slaves ( these 6 pulll the index from master using rsync)
> >
> > We have a SQL(Oracle) script to post the data/index from Oracle Database
> > machine to Solr Master over http. We wrote those script(Someone in Oracle
> > Database administrator write it).
>
> You said in your other email you are having issues with slow transfers
> between
> 1) and 2). Your subject relates to the data transfer between 1) and 2, - 2)
> and
> 3) is irrelevant to this part.
>
> My question (what you quoted above) relates to the point you made about it
> being slow ( WHY is it slow?), and issues with opening so many connections
> through firewall. so, I'll rephrase my question (see below...)
>
> [....]
> >
> > We can not do localhost since it's solr is not running on Oracle machine.
>
> why not generate your SQL output directly into your oracle server as a
> file,
> upload the file to your SOLR server? Then the data file is local to your
> SOLR
> server , you will bypass any WAN and firewall you may be having. (or some
> variation of it, sql -> SOLR server as file, etc..)
>
> Any speed issues that are rooted in the fact that you are posting via
> HTTP (vs embedded solr or DIH) aren't going to go away. But it's the
> simpler
> approach without changing too much of your current setup.
>
>
> > Another alternative that we think of is to transform XML into CSV and
> > import/export it.
> >
> > How about if LUSQL, some mentioned about this? Is this apps free(open
> source)
> > application? Do you have any experience with this apps?
>
> Not i, sorry.
>
> Have you looked into DIH? It's designed for this kind of work.
>
> B
> _________________________
> {Beto|Norberto|Numard} Meijome
>
> "Great spirits have often encountered violent opposition from mediocre
> minds."
> Albert Einstein
>
> I speak for myself, not my employer. Contents may be hot. Slippery when
> wet.
> Reading disclaimers makes you go blind. Writing them is worse. You have
> been
> Warned.
>
>
--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Norberto,
Yes, DIH is one of the option we think to use, but it's required 1.3.0 and above and currently we are running Sol 1.2.0.
I am thinking to use CSV file(Convert the XML to CSV format in Database machine( , then transport that CSV file to solr box.
In Solr we run the update to convert the CSV file to Lucene index.
Also , we think the one you suggested, note my question below:
>
>why not generate your SQL output directly into your oracle server as a file,
question: What type of file is this(XML or CSV)?
>upload the file to your SOLR server? Then the data file is local to your SOLR
>server , you will bypass any WAN and firewall you may be having. (or some
>variation of it, sql -> SOLR server as file, etc..)
How we upload the file? Do we need to convert the data file to Lucene Index first?
And Documentation how we do this?
>Any speed issues that are rooted in the fact that you are posting via
>HTTP (vs embedded solr or DIH) aren't going to go away. But it's the simpler
>approach without changing too much of your current setup.
-----Original Message-----
From: Norberto Meijome [mailto:numardbsd@gmail.com]
Sent: Sunday, July 05, 2009 3:57 AM
To: Francis Yakin
Cc: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
On Thu, 2 Jul 2009 11:02:28 -0700
Francis Yakin <fy...@liquid.com> wrote:
> Norberto, Thanks for your input.
>
> What do you mean with "Have you tried connecting to SOLR over HTTP from
> localhost, therefore avoiding any firewall issues and network latency ? it
> should work a LOT faster than from a remote site." ?
>
>
> Here are how our servers lay out:
>
> 1) Database ( Oracle ) is running on separate machine
> 2) Solr master is running on separate machine by itself
> 3) 6 solr slaves ( these 6 pulll the index from master using rsync)
>
> We have a SQL(Oracle) script to post the data/index from Oracle Database
> machine to Solr Master over http. We wrote those script(Someone in Oracle
> Database administrator write it).
You said in your other email you are having issues with slow transfers between
1) and 2). Your subject relates to the data transfer between 1) and 2, - 2) and
3) is irrelevant to this part.
My question (what you quoted above) relates to the point you made about it
being slow ( WHY is it slow?), and issues with opening so many connections
through firewall. so, I'll rephrase my question (see below...)
[....]
>
> We can not do localhost since it's solr is not running on Oracle machine.
why not generate your SQL output directly into your oracle server as a file,
upload the file to your SOLR server? Then the data file is local to your SOLR
server , you will bypass any WAN and firewall you may be having. (or some
variation of it, sql -> SOLR server as file, etc..)
Any speed issues that are rooted in the fact that you are posting via
HTTP (vs embedded solr or DIH) aren't going to go away. But it's the simpler
approach without changing too much of your current setup.
> Another alternative that we think of is to transform XML into CSV and
> import/export it.
>
> How about if LUSQL, some mentioned about this? Is this apps free(open source)
> application? Do you have any experience with this apps?
Not i, sorry.
Have you looked into DIH? It's designed for this kind of work.
B
_________________________
{Beto|Norberto|Numard} Meijome
"Great spirits have often encountered violent opposition from mediocre minds."
Albert Einstein
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Norberto Meijome <nu...@gmail.com>.
On Thu, 2 Jul 2009 11:02:28 -0700
Francis Yakin <fy...@liquid.com> wrote:
> Norberto, Thanks for your input.
>
> What do you mean with "Have you tried connecting to SOLR over HTTP from
> localhost, therefore avoiding any firewall issues and network latency ? it
> should work a LOT faster than from a remote site." ?
>
>
> Here are how our servers lay out:
>
> 1) Database ( Oracle ) is running on separate machine
> 2) Solr master is running on separate machine by itself
> 3) 6 solr slaves ( these 6 pulll the index from master using rsync)
>
> We have a SQL(Oracle) script to post the data/index from Oracle Database
> machine to Solr Master over http. We wrote those script(Someone in Oracle
> Database administrator write it).
You said in your other email you are having issues with slow transfers between
1) and 2). Your subject relates to the data transfer between 1) and 2, - 2) and
3) is irrelevant to this part.
My question (what you quoted above) relates to the point you made about it
being slow ( WHY is it slow?), and issues with opening so many connections
through firewall. so, I'll rephrase my question (see below...)
[....]
>
> We can not do localhost since it's solr is not running on Oracle machine.
why not generate your SQL output directly into your oracle server as a file,
upload the file to your SOLR server? Then the data file is local to your SOLR
server , you will bypass any WAN and firewall you may be having. (or some
variation of it, sql -> SOLR server as file, etc..)
Any speed issues that are rooted in the fact that you are posting via
HTTP (vs embedded solr or DIH) aren't going to go away. But it's the simpler
approach without changing too much of your current setup.
> Another alternative that we think of is to transform XML into CSV and
> import/export it.
>
> How about if LUSQL, some mentioned about this? Is this apps free(open source)
> application? Do you have any experience with this apps?
Not i, sorry.
Have you looked into DIH? It's designed for this kind of work.
B
_________________________
{Beto|Norberto|Numard} Meijome
"Great spirits have often encountered violent opposition from mediocre minds."
Albert Einstein
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Norberto, Thanks for your input.
What do you mean with "Have you tried connecting to SOLR over HTTP from localhost, therefore avoiding any firewall issues and network latency ? it should work a LOT faster than from a remote site." ?
Here are how our servers lay out:
1) Database ( Oracle ) is running on separate machine
2) Solr master is running on separate machine by itself
3) 6 solr slaves ( these 6 pulll the index from master using rsync)
We have a SQL(Oracle) script to post the data/index from Oracle Database machine to Solr Master over http.
We wrote those script(Someone in Oracle Database administrator write it).
In Solr master configuration we have scripts.conf that like this:
user=
solr_hostname=localhost
solr_port=7001
rsyncd_port=18983
data_dir=
webapp_name=solr
master_host=localhost
master_data_dir=solr/snapshot
master_status_dir=solr/status
So, basically from Oracle system we launch the Oracle/SQL script posting the data to Solr Master using
http://solrmaster/solr/update ( inside the SQL script we put this).
We can not do localhost since it's solr is not running on Oracle machine.
Another alternative that we think of is to transform XML into CSV and import/export it.
How about if LUSQL, some mentioned about this? Is this apps free(open source) application? Do you have any experience with this apps?
Thanks All for your valuable suggestions!
Francis
-----Original Message-----
From: Norberto Meijome [mailto:numardbsd@gmail.com]
Sent: Thursday, July 02, 2009 3:01 AM
To: solr-user@lucene.apache.org
Cc: Francis Yakin
Subject: Re: Is there any other way to load the index beside using "http" connection?
On Wed, 1 Jul 2009 15:07:12 -0700
Francis Yakin <fy...@liquid.com> wrote:
>
> We have several thousands of xml files in database that we load it to solr
> master The Database uses "http" connection and transfer those files to solr
> master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very
> very slow.
>
> Is there any other way to load the data/index from Database to solr master
> beside using http connection, so it means we just scp/ftp the xml file from
> Database system to solr master and let solr convert those to lucene indexes?
>
Francis,
after reading the whole thread, it seems you have :
- Data source : Oracle DB, on separate location to your SOLR.
- Data format : XML output.
definitely DIH is a great option, but since you are on 1.2, not available to you (you should look into upgrading if you can!).
Have you tried connecting to SOLR over HTTP from localhost, therefore avoiding any firewall issues and network latency ? it should work a LOT faster than from a remote site. Also make sure not to commit until you really needed.
Other alternatives are to transform the XML into csv and import it that way. Or write a simple app that will parse the xml and post it directly using the embedded solr method.
plenty of options, all of them documented @ solr's site.
good luck,
b
_________________________
{Beto|Norberto|Numard} Meijome
"People demand freedom of speech to make up for the freedom of thought which they avoid. "
Soren Aabye Kierkegaard
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Norberto Meijome <nu...@gmail.com>.
On Thu, 2 Jul 2009 11:28:51 -0700
Francis Yakin <fy...@liquid.com> wrote:
> Norberto,
>
Hi Francis,
Please reply to the list, or keep it in CC.
> You saying:
>
> "Other alternatives are to transform the XML into csv and import it that way"
>
> How do you transfer that CSV file to Solr?
>
http://wiki.apache.org/solr/UpdateCSV
There actually is a LOT of information in the wiki, as well as the mailing list archives.
good luck,
B
_________________________
{Beto|Norberto|Numard} Meijome
"The freethinking of one age is the common sense of the next."
Matthew Arnold
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Norberto Meijome <nu...@gmail.com>.
On Wed, 1 Jul 2009 15:07:12 -0700
Francis Yakin <fy...@liquid.com> wrote:
>
> We have several thousands of xml files in database that we load it to solr
> master The Database uses "http" connection and transfer those files to solr
> master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very
> very slow.
>
> Is there any other way to load the data/index from Database to solr master
> beside using http connection, so it means we just scp/ftp the xml file from
> Database system to solr master and let solr convert those to lucene indexes?
>
Francis,
after reading the whole thread, it seems you have :
- Data source : Oracle DB, on separate location to your SOLR.
- Data format : XML output.
definitely DIH is a great option, but since you are on 1.2, not available to you (you should look into upgrading if you can!).
Have you tried connecting to SOLR over HTTP from localhost, therefore avoiding any firewall issues and network latency ? it should work a LOT faster than from a remote site. Also make sure not to commit until you really needed.
Other alternatives are to transform the XML into csv and import it that way. Or write a simple app that will parse the xml and post it directly using the embedded solr method.
plenty of options, all of them documented @ solr's site.
good luck,
b
_________________________
{Beto|Norberto|Numard} Meijome
"People demand freedom of speech to make up for the freedom of thought which they avoid. "
Soren Aabye Kierkegaard
I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Glen Newton <gl...@gmail.com>.
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Is this LuSql is free? Is that an open source.
LuSql is an Open Source project.
> Is that requires a separate machine with Solr Master
LuSql is a Java application that runs on the command line. It connects
to a the database using JDBC and creates a local Lucene index, based
on the configuration you supply to it.
> I forgot to tell you that we have Master/Slaves environment of Solr.
>
> The Database is running Oracle and it's separate machine that running in different network than Master and Slaves Solr(There is a firewall between Oracle machine and Solr Machines).
> If we have LuSql Machine, do you think it's better to put into the same network with DataBase machine or Solr machines?
LuSql is heavily multi-threaded, and can suck up the resources of all
cores (this is why it runs so fast), so you need to decide if this is
not appropriate for your database machine (i.e. if it is a production
machine). You can isolate LuSql to specific cores using something
like numactl http://www.linuxmanpages.com/man8/numactl.8.php
> Do I need to create a sql script to get the data from Oarcle and loading it using LuSql and convert it to Lucene index, and how solr master will get that data?
LuSql reads from Oracle and writes to a Lucene index. You just need to
give LuSql a configuration that has it generate the appropriate index
for Solr.
thanks,
Glen
http://zzzoot.blogspot.com/search?q=lucene
>
>
> Thanks
>
> Francis
>
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Thursday, July 02, 2009 8:22 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> LuSql can be found here:
> http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
> User Manual:
> http://cuvier.cisti.nrc.ca/~gnewton/lusql/v0.9/lusqlManual.pdf.html
>
> LuSql can communicate directly with Oracle and create a Lucene index for you.
> Of course - as mentioned by other posters - you need to make sure the
> versions of Lucene and Solr are compatible (use same jars), you use
> the same Analyzers, and you create the appropriate 'schema' that Solr
> understands.
>
> -glen
>
> 2009/7/2 Francis Yakin <fy...@liquid.com>:
>>
>> Glen,
>>
>> Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
>> SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
>>
>> My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
>>
>> You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
>>
>> Thanks
>>
>> Francis
>>
>> -----Original Message-----
>> From: Glen Newton [mailto:glen.newton@gmail.com]
>> Sent: Wednesday, July 01, 2009 8:06 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Is there any other way to load the index beside using "http" connection?
>>
>> You can directly load to the backend Lucene using LuSql[1]. It is
>> faster than Solr, sometimes as much as an order of magnitude faster.
>>
>> Disclosure: I am the author of LuSql
>>
>> -Glen
>> http://zzzoot.blogspot.com/
>>
>> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>>
>> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>>
>>> We have several thousands of xml files in database that we load it to solr master
>>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>>
>>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>>
>>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>>
>>> Any input or help will be much appreciated.
>>>
>>>
>>> Thanks
>>>
>>> Francis
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>>
>> -
>>
>
>
>
> --
>
> -
>
--
-
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Glen,
Is this LuSql is free? Is that an open source.
Is that requires a separate machine with Solr Master
I forgot to tell you that we have Master/Slaves environment of Solr.
The Database is running Oracle and it's separate machine that running in different network than Master and Slaves Solr(There is a firewall between Oracle machine and Solr Machines).
If we have LuSql Machine, do you think it's better to put into the same network with DataBase machine or Solr machines?
Do I need to create a sql script to get the data from Oarcle and loading it using LuSql and convert it to Lucene index, and how solr master will get that data?
Thanks
Francis
-----Original Message-----
From: Glen Newton [mailto:glen.newton@gmail.com]
Sent: Thursday, July 02, 2009 8:22 AM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
LuSql can be found here:
http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
User Manual:
http://cuvier.cisti.nrc.ca/~gnewton/lusql/v0.9/lusqlManual.pdf.html
LuSql can communicate directly with Oracle and create a Lucene index for you.
Of course - as mentioned by other posters - you need to make sure the
versions of Lucene and Solr are compatible (use same jars), you use
the same Analyzers, and you create the appropriate 'schema' that Solr
understands.
-glen
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
> SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
>
> My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
>
> You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
>
> Thanks
>
> Francis
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Wednesday, July 01, 2009 8:06 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> You can directly load to the backend Lucene using LuSql[1]. It is
> faster than Solr, sometimes as much as an order of magnitude faster.
>
> Disclosure: I am the author of LuSql
>
> -Glen
> http://zzzoot.blogspot.com/
>
> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>
> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>
>> We have several thousands of xml files in database that we load it to solr master
>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>
>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>
>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>
>> Any input or help will be much appreciated.
>>
>>
>> Thanks
>>
>> Francis
>>
>>
>>
>>
>
>
>
> --
>
> -
>
--
-
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Glen Newton <gl...@gmail.com>.
LuSql can be found here:
http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
User Manual:
http://cuvier.cisti.nrc.ca/~gnewton/lusql/v0.9/lusqlManual.pdf.html
LuSql can communicate directly with Oracle and create a Lucene index for you.
Of course - as mentioned by other posters - you need to make sure the
versions of Lucene and Solr are compatible (use same jars), you use
the same Analyzers, and you create the appropriate 'schema' that Solr
understands.
-glen
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
> SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
>
> My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
>
> You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
>
> Thanks
>
> Francis
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Wednesday, July 01, 2009 8:06 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> You can directly load to the backend Lucene using LuSql[1]. It is
> faster than Solr, sometimes as much as an order of magnitude faster.
>
> Disclosure: I am the author of LuSql
>
> -Glen
> http://zzzoot.blogspot.com/
>
> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>
> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>
>> We have several thousands of xml files in database that we load it to solr master
>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>
>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>
>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>
>> Any input or help will be much appreciated.
>>
>>
>> Thanks
>>
>> Francis
>>
>>
>>
>>
>
>
>
> --
>
> -
>
--
-
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Glen,
Database we use is Oracle, I am not the database administrator, so I don't familiar with their script.
SO, basically we have the Oracle SQL script to load the XML files over HTTP connection to our Solr Master.
My question is there any other way instead of using HTTP connection to load the XML files to our SOLR Master?
You mentioned about LuSql, I am not familiar with that. Can you provide us the docs or something? Again I am not the database Guys, I am only the solr Guy. The database we have is a different box than Solr master and both are running linux(RedHat).
Thanks
Francis
-----Original Message-----
From: Glen Newton [mailto:glen.newton@gmail.com]
Sent: Wednesday, July 01, 2009 8:06 PM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
You can directly load to the backend Lucene using LuSql[1]. It is
faster than Solr, sometimes as much as an order of magnitude faster.
Disclosure: I am the author of LuSql
-Glen
http://zzzoot.blogspot.com/
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
2009/7/1 Francis Yakin <fy...@liquid.com>:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Glen Newton <gl...@gmail.com>.
> Are you saying that we have to use LuSql replacing our Solr?
To load your data: Yes, it is an option
To search your data: No, LuSql is only a loading tool
-glen
2009/7/2 Francis Yakin <fy...@liquid.com>:
>
> Glen,
>
> Are you saying that we have to use LuSql replacing our Solr?
>
> Francis
>
> -----Original Message-----
> From: Glen Newton [mailto:glen.newton@gmail.com]
> Sent: Wednesday, July 01, 2009 8:06 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Is there any other way to load the index beside using "http" connection?
>
> You can directly load to the backend Lucene using LuSql[1]. It is
> faster than Solr, sometimes as much as an order of magnitude faster.
>
> Disclosure: I am the author of LuSql
>
> -Glen
> http://zzzoot.blogspot.com/
>
> [1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
>
> 2009/7/1 Francis Yakin <fy...@liquid.com>:
>>
>> We have several thousands of xml files in database that we load it to solr master
>> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>>
>> We are experiencing issue with close/open connection in the firewall and very very slow.
>>
>> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>>
>> Any input or help will be much appreciated.
>>
>>
>> Thanks
>>
>> Francis
>>
>>
>>
>>
>
>
>
> --
>
> -
>
--
-
RE: Is there any other way to load the index beside using "http"
connection?
Posted by Francis Yakin <fy...@liquid.com>.
Glen,
Are you saying that we have to use LuSql replacing our Solr?
Francis
-----Original Message-----
From: Glen Newton [mailto:glen.newton@gmail.com]
Sent: Wednesday, July 01, 2009 8:06 PM
To: solr-user@lucene.apache.org
Subject: Re: Is there any other way to load the index beside using "http" connection?
You can directly load to the backend Lucene using LuSql[1]. It is
faster than Solr, sometimes as much as an order of magnitude faster.
Disclosure: I am the author of LuSql
-Glen
http://zzzoot.blogspot.com/
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
2009/7/1 Francis Yakin <fy...@liquid.com>:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-
Re: Is there any other way to load the index beside using "http"
connection?
Posted by Glen Newton <gl...@gmail.com>.
You can directly load to the backend Lucene using LuSql[1]. It is
faster than Solr, sometimes as much as an order of magnitude faster.
Disclosure: I am the author of LuSql
-Glen
http://zzzoot.blogspot.com/
[1]http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql
2009/7/1 Francis Yakin <fy...@liquid.com>:
>
> We have several thousands of xml files in database that we load it to solr master
> The Database uses "http" connection and transfer those files to solr master. Solr then translate xml files to their lindex.
>
> We are experiencing issue with close/open connection in the firewall and very very slow.
>
> Is there any other way to load the data/index from Database to solr master beside using http connection, so it means we just scp/ftp the xml file from Database system to solr master and let solr convert those to lucene indexes?
>
> Any input or help will be much appreciated.
>
>
> Thanks
>
> Francis
>
>
>
>
--
-