You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Stefan Maric <sm...@ntlworld.com> on 2010/04/15 22:19:51 UTC
XSD for Solrv1.4
Are there any plans to release an xsd (& preferably a set of JAXB classes)
so we can process the xml returned for a search request
Re: XSD for Solrv1.4
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: XSD for Solrv1.4
: In-Reply-To: <12...@n3.nabble.com>
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking
-Hoss
Re: Solr Index Lock Issue
Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Solr Index Lock Issue
: Date: Fri, 16 Apr 2010 22:43:57 +0530
: In-Reply-To: <CJ...@ntlworld.com>
: References: <12...@n3.nabble.com>
: <CJ...@ntlworld.com>
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking
-Hoss
RE: Solr Index Lock Issue
Posted by "Sethi, Parampreet" <pa...@corp.aol.com>.
Hi Otis,
Thanks. I cleared the data folder and restarted the Solr server.
We trigger solr ingestion through Java. We tried removing all
incremental solr.commit() after every batch, and added a commit command
using curl in the shell script file after the ingestion finishes.
Content of .sh file -
###########################################################
java mySolrIngestionClass
curl "http://localhost:9083/solr/update?commit=true"
###########################################################
mySolrIngestionClass adds documents to solr, and does not contain any
commit.
The observation is that the lock file still appears in the
Solr/data/index folder within 30 mins of Solr ingestion and after that
solr hangs. In threaddump.jsp of Solr, it shows some of the threads in
Blocked state. We checked the solr config, the autocommit is also
commented out.
We are using Solr 1.4 with tomcat 1.6.18. It used to run earlier without
any issues in 24 hrs time, but now even after 3 days its not completed.
Any pointers to the solution will be great. Thanks!
Regards,
Param
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Saturday, April 17, 2010 3:50 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr Index Lock Issue
Hi,
What you are doing sounds fine. You don't need to commit while
indexing, though, just commit/optimize at the end. I'm not saying this
will solve your problem, but give it a try.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Hadoop
ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: "Sethi, Parampreet" <pa...@corp.aol.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, April 16, 2010 1:13:57 PM
> Subject: Solr Index Lock Issue
>
> Hi All,
We are facing the issue with the Solr server in the DMOZ data
> migration.
The Solr has 0 records when the migration starts and the data is
> added
into Solr in the batches of 20000 records. The commit is called on
> Solr
after 20k records are processed.
While commiting the data into
> Solr, a lucene lock file is created in the
<Solr_Home>/data/index
> folder which is automatically released once the
successful commit happens.
> But after 4-5 batches, the lock file remains
there and Solr just hangs and
> does not add any new records. Some times
the whole migration goes through
> without any errors.
Kindly let me know in case some setting needs to be
> required on Solr
side, which ensures that until the Solr commits the index,
> the next set
of records should not be added.
Thanks,
Param
Re: Solr Index Lock Issue
Posted by Lance Norskog <go...@gmail.com>.
The commit call can return before Solr is completely finished with
everything it does after a commit. This tail processing can build up
with successive commits until Solr is basically stuck.
Commit should use waitFlush=true. This makes the commit call wait
until the disk manipulation is done. If you add 'waitSearch=true' and
'expungeDeletes=true', this will wait completely until Solr is ready
to do more indexing.
"Solr just hangs" is not enough to go on. Solr/Lucene have processing
where it takes increasing amounts of time to index data, Is the CPU
busy? Is the disk busy? Does 'jhat' or 'visualgc' show memory
activity?
On 4/16/10, Otis Gospodnetic <ot...@yahoo.com> wrote:
> Hi,
>
> What you are doing sounds fine. You don't need to commit while indexing,
> though, just commit/optimize at the end. I'm not saying this will solve
> your problem, but give it a try.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Hadoop ecosystem search :: http://search-hadoop.com/
>
>
>
> ----- Original Message ----
>> From: "Sethi, Parampreet" <pa...@corp.aol.com>
>> To: solr-user@lucene.apache.org
>> Sent: Fri, April 16, 2010 1:13:57 PM
>> Subject: Solr Index Lock Issue
>>
>> Hi All,
>
> We are facing the issue with the Solr server in the DMOZ data
>> migration.
> The Solr has 0 records when the migration starts and the data is
>> added
> into Solr in the batches of 20000 records. The commit is called on
>> Solr
> after 20k records are processed.
>
> While commiting the data into
>> Solr, a lucene lock file is created in the
> <Solr_Home>/data/index
>> folder which is automatically released once the
> successful commit happens.
>> But after 4-5 batches, the lock file remains
> there and Solr just hangs and
>> does not add any new records. Some times
> the whole migration goes through
>> without any errors.
>
> Kindly let me know in case some setting needs to be
>> required on Solr
> side, which ensures that until the Solr commits the index,
>> the next set
> of records should not be added.
>
> Thanks,
> Param
>
--
Lance Norskog
goksron@gmail.com
Re: Solr Index Lock Issue
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
What you are doing sounds fine. You don't need to commit while indexing, though, just commit/optimize at the end. I'm not saying this will solve your problem, but give it a try.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: "Sethi, Parampreet" <pa...@corp.aol.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, April 16, 2010 1:13:57 PM
> Subject: Solr Index Lock Issue
>
> Hi All,
We are facing the issue with the Solr server in the DMOZ data
> migration.
The Solr has 0 records when the migration starts and the data is
> added
into Solr in the batches of 20000 records. The commit is called on
> Solr
after 20k records are processed.
While commiting the data into
> Solr, a lucene lock file is created in the
<Solr_Home>/data/index
> folder which is automatically released once the
successful commit happens.
> But after 4-5 batches, the lock file remains
there and Solr just hangs and
> does not add any new records. Some times
the whole migration goes through
> without any errors.
Kindly let me know in case some setting needs to be
> required on Solr
side, which ensures that until the Solr commits the index,
> the next set
of records should not be added.
Thanks,
Param
Solr Index Lock Issue
Posted by "Sethi, Parampreet" <pa...@corp.aol.com>.
Hi All,
We are facing the issue with the Solr server in the DMOZ data migration.
The Solr has 0 records when the migration starts and the data is added
into Solr in the batches of 20000 records. The commit is called on Solr
after 20k records are processed.
While commiting the data into Solr, a lucene lock file is created in the
<Solr_Home>/data/index folder which is automatically released once the
successful commit happens. But after 4-5 batches, the lock file remains
there and Solr just hangs and does not add any new records. Some times
the whole migration goes through without any errors.
Kindly let me know in case some setting needs to be required on Solr
side, which ensures that until the Solr commits the index, the next set
of records should not be added.
Thanks,
Param
Re: Solr configuration to enable indexing/searching webapp log files
Posted by Lance Norskog <go...@gmail.com>.
It sounds like you want a data warehouse, not a text search engine.
Splunk and Pentaho are good things to try.
On Thu, Apr 29, 2010 at 12:03 PM, Jon Baer <jo...@gmail.com> wrote:
> To follow up it ... it seems dumping to Solr is common ...
>
> http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data
>
> - Jon
>
> On Apr 29, 2010, at 1:58 PM, Jon Baer wrote:
>
>> Good question, +1 on finding answer, my take ...
>>
>> Depending on how large of log files you are talking about it might be better off to do this w/ HDFS / Hadoop (and a script language like Pig) (or Amazon EMR)
>>
>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=873
>>
>> Theoretically you could split the logs to fields, use a dataimporter and search / sort w/ something like LineEntityProcessor.
>>
>> http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor
>>
>> I've tried to use Solr as a log analytics tool (before dataimporthandler) and it was not worth the disk space or practical but I'd love to hear otherwise. In general you could flush daily logs to an index but working w/ the data in another context if you had to seems better fit for HDFS use (I think).
>>
>> - Jon
>>
>> On Apr 29, 2010, at 1:46 PM, Stefan Maric wrote:
>>
>>>
>>> I thought i remembered seeing some information about this, but have been
>>> unable to find it
>>>
>>> Does anyone know if there is a configuration / module that would allow us to
>>> setup Solr to take in the (large) log files generated by our web/app
>>> servers, so that we can query for things like peak time requests or most
>>> frequently requested web page etc
>>>
>>> Thanks
>>> Stefan Maric
>>>
>>
>
>
--
Lance Norskog
goksron@gmail.com
Re: Solr configuration to enable indexing/searching webapp log files
Posted by Jon Baer <jo...@gmail.com>.
To follow up it ... it seems dumping to Solr is common ...
http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data
- Jon
On Apr 29, 2010, at 1:58 PM, Jon Baer wrote:
> Good question, +1 on finding answer, my take ...
>
> Depending on how large of log files you are talking about it might be better off to do this w/ HDFS / Hadoop (and a script language like Pig) (or Amazon EMR)
>
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=873
>
> Theoretically you could split the logs to fields, use a dataimporter and search / sort w/ something like LineEntityProcessor.
>
> http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor
>
> I've tried to use Solr as a log analytics tool (before dataimporthandler) and it was not worth the disk space or practical but I'd love to hear otherwise. In general you could flush daily logs to an index but working w/ the data in another context if you had to seems better fit for HDFS use (I think).
>
> - Jon
>
> On Apr 29, 2010, at 1:46 PM, Stefan Maric wrote:
>
>>
>> I thought i remembered seeing some information about this, but have been
>> unable to find it
>>
>> Does anyone know if there is a configuration / module that would allow us to
>> setup Solr to take in the (large) log files generated by our web/app
>> servers, so that we can query for things like peak time requests or most
>> frequently requested web page etc
>>
>> Thanks
>> Stefan Maric
>>
>
Re: Solr configuration to enable indexing/searching webapp log files
Posted by Jon Baer <jo...@gmail.com>.
Good question, +1 on finding answer, my take ...
Depending on how large of log files you are talking about it might be better off to do this w/ HDFS / Hadoop (and a script language like Pig) (or Amazon EMR)
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=873
Theoretically you could split the logs to fields, use a dataimporter and search / sort w/ something like LineEntityProcessor.
http://wiki.apache.org/solr/DataImportHandler#LineEntityProcessor
I've tried to use Solr as a log analytics tool (before dataimporthandler) and it was not worth the disk space or practical but I'd love to hear otherwise. In general you could flush daily logs to an index but working w/ the data in another context if you had to seems better fit for HDFS use (I think).
- Jon
On Apr 29, 2010, at 1:46 PM, Stefan Maric wrote:
>
> I thought i remembered seeing some information about this, but have been
> unable to find it
>
> Does anyone know if there is a configuration / module that would allow us to
> setup Solr to take in the (large) log files generated by our web/app
> servers, so that we can query for things like peak time requests or most
> frequently requested web page etc
>
> Thanks
> Stefan Maric
>
Solr configuration to enable indexing/searching webapp log files
Posted by Stefan Maric <sm...@ntlworld.com>.
I thought i remembered seeing some information about this, but have been
unable to find it
Does anyone know if there is a configuration / module that would allow us to
setup Solr to take in the (large) log files generated by our web/app
servers, so that we can query for things like peak time requests or most
frequently requested web page etc
Thanks
Stefan Maric
RE: XSD for Solrv1.4
Posted by Stefan Maric <sm...@ntlworld.com>.
Thanks I'm taking a look at SolrJ
Longer term i'd still like to have access to an XSD - then i can see us
integrating this better in the Oracle Service Bus and writing less Java code
in our webApp
-----Original Message-----
From: hkmortensen [mailto:kokdk@yahoo.com]
Sent: 15 April 2010 21:26
To: solr-user@lucene.apache.org
Subject: Re: XSD for Solrv1.4
Smaric-2 wrote:
>
> Are there any plans to release an xsd (& preferably a set of JAXB classes)
> so we can process the xml returned for a search request
>
>
>
I do not know. I would recommend to use solrj, the java client. I always do
that myself, do you have a reason not to do that?
--
View this message in context:
http://n3.nabble.com/weight-and-ranking-tp720944p722255.html
Sent from the Solr - User mailing list archive at Nabble.com.
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.437 / Virus Database: 271.1.1/2812 - Release Date: 04/15/10
06:31:00
RE: solr configuration for Subversion
Posted by Stefan Maric <sm...@ntlworld.com>.
Thanks. I'll take a look at this
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: 27 May 2010 01:54
To: solr-user@lucene.apache.org
Subject: Re: solr configuration for Subversion
: I've seen the info about SvnQuery & wondered if anyone has a Solr
: configuration / loader module
I've never heard of SvnQuery until your email, but it seems to be built
using Lucene.Net...
http://svnquery.tigris.org/
If you're looking for tools for indexing subversion repos with solr
specificly, you might want to check out ReposSearch...
http://repossearch.com/
-Hoss
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.437 / Virus Database: 271.1.1/2898 - Release Date: 05/26/10
18:26:00
Re: solr configuration for Subversion
Posted by Chris Hostetter <ho...@fucit.org>.
: I've seen the info about SvnQuery & wondered if anyone has a Solr
: configuration / loader module
I've never heard of SvnQuery until your email, but it seems to be built
using Lucene.Net...
http://svnquery.tigris.org/
If you're looking for tools for indexing subversion repos with solr
specificly, you might want to check out ReposSearch...
http://repossearch.com/
-Hoss
solr configuration for Subversion
Posted by Stefan Maric <sm...@ntlworld.com>.
I've seen the info about SvnQuery & wondered if anyone has a Solr
configuration / loader module
regards
Stefan Maric
Re: XSD for Solrv1.4
Posted by hkmortensen <ko...@yahoo.com>.
Smaric-2 wrote:
>
> Are there any plans to release an xsd (& preferably a set of JAXB classes)
> so we can process the xml returned for a search request
>
>
>
I do not know. I would recommend to use solrj, the java client. I always do
that myself, do you have a reason not to do that?
--
View this message in context: http://n3.nabble.com/weight-and-ranking-tp720944p722255.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: XSD for Solrv1.4
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Stefan,
I'm not sure about releasing one for 1.4, but I was making some progress on SOLR-17 [1] (and see the linked issues from there) on pushing this forward. It dropped off my radar for a while, but I'd be happy to pick it up back up.
Let me see what I can do.
Thanks!
Cheers,
Chris
[1] http://issues.apache.org/jira/browse/SOLR-17
On 4/15/10 1:19 PM, "Stefan Maric" <sm...@ntlworld.com> wrote:
Are there any plans to release an xsd (& preferably a set of JAXB classes)
so we can process the xml returned for a search request
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++