You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Vijay Veeraraghavan <vi...@gmail.com> on 2010/04/30 06:48:47 UTC
Using lucene in NFS
dear all,
I have a problem using lucene in NFS. A scheduler runs periodically
generating reports in pdf format and saves it to a file server. The
drive of the file server is mounted to the scheduler server (NFS).
After generating reports finally the scheduler indexes the names of
the report and its path. This index is also stored to the same context
folder the reports are created, ie., mounted NFS folder. When i create
index in the local system (JFS2) its as fast as possible. (the local
system is AIX uses jfs2.) But in the NFS folder it keep doing and
doing... i didnt exactly know what happened inside. little later the
application freezed.I saw some lock files created inside the folder
but nothing like index files that usually created by the lucene. from
the web page i search this index in nfs for the report and read it. My
question is how to use lucene in NFS. anything special we need to do?
any reference codes? docs? How do i lock the index when my scheduler
is updating the index?
P.S: Please add CC to vijayv@kggroup.com
Thanks
Vijay Veeraraghavan
--
Thanks & Regards
Vijay Veeraraghavan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Using lucene in NFS
Posted by Ian Lea <ia...@gmail.com>.
The suggestion was that your single indexing job should update a local
copy of the index and copy that to NFS for searching by other nodes.
That should work.
As for updating, you could index new reports into a new lucene index
and then merge that into the existing index
(IndexWriter.addIndexes()). Or you could simple add them directly to
the existing index, making sure that you open it with create = false.
--
Ian.
On Fri, Apr 30, 2010 at 1:01 PM, Vijay Veeraraghavan
<vi...@gmail.com> wrote:
> hi Ian,
> Thanks for your reply. I am using the Lucene core 3.0 version. The
> index created will be accessed by the web application. The web
> application contains 4 nodes, clustered. What if all the nodes access
> the index. I think no any prob may raise.
> If i have a local index then what about it in the clustered environment?
> The NFS given is working good for saving the report, but only while i
> start the indexing the app freezes. it creates a lock file, i cant see
> whats happening inside as i cant see my console outputs too.
> And i have once more question. How do u update the index. As my
> scheduler keeps producing more reports to the file server, the index
> need to be updated then and there. how do i update the new report
> files to the index than doing again all the files. Or can i create a
> local index of the newly created reports and join this with the old
> index? any other professional solution?
>
> Thanks
> Vijay
>
> On 4/30/10, Ian Lea <ia...@gmail.com> wrote:
>> You don't say what version of lucene you are using, but in recent
>> versions you may need to use SimpleFSLockFactory rather than the
>> default, NativeFSLockFactory. See the javadocs. Lucene in general
>> does work on NFS but there can be problems, particularly with
>> concurrent access from multiple servers.
>>
>> But your problem sounds different. Does the NFS mount work properly
>> for non-lucene operations? Or maybe you could maintain a copy of the
>> index on local disks and just copy it to NFS for access from other
>> servers. Lucene locking should take care of locking the index when
>> updates are running.
>>
>>
>> --
>> Ian.
>>
>>
>> On Fri, Apr 30, 2010 at 5:48 AM, Vijay Veeraraghavan
>> <vi...@gmail.com> wrote:
>>> dear all,
>>>
>>> I have a problem using lucene in NFS. A scheduler runs periodically
>>> generating reports in pdf format and saves it to a file server. The
>>> drive of the file server is mounted to the scheduler server (NFS).
>>> After generating reports finally the scheduler indexes the names of
>>> the report and its path. This index is also stored to the same context
>>> folder the reports are created, ie., mounted NFS folder. When i create
>>> index in the local system (JFS2) its as fast as possible. (the local
>>> system is AIX uses jfs2.) But in the NFS folder it keep doing and
>>> doing... i didnt exactly know what happened inside. little later the
>>> application freezed.I saw some lock files created inside the folder
>>> but nothing like index files that usually created by the lucene. from
>>> the web page i search this index in nfs for the report and read it. My
>>> question is how to use lucene in NFS. anything special we need to do?
>>> any reference codes? docs? How do i lock the index when my scheduler
>>> is updating the index?
>>>
>>> P.S: Please add CC to vijayv@kggroup.com
>>>
>>> Thanks
>>> Vijay Veeraraghavan
>>>
>>>
>>>
>>> --
>>> Thanks & Regards
>>> Vijay Veeraraghavan
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Thanks & Regards
> Vijay Veeraraghavan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Using lucene in NFS
Posted by Vijay Veeraraghavan <vi...@gmail.com>.
hi Ian,
Thanks for your reply. I am using the Lucene core 3.0 version. The
index created will be accessed by the web application. The web
application contains 4 nodes, clustered. What if all the nodes access
the index. I think no any prob may raise.
If i have a local index then what about it in the clustered environment?
The NFS given is working good for saving the report, but only while i
start the indexing the app freezes. it creates a lock file, i cant see
whats happening inside as i cant see my console outputs too.
And i have once more question. How do u update the index. As my
scheduler keeps producing more reports to the file server, the index
need to be updated then and there. how do i update the new report
files to the index than doing again all the files. Or can i create a
local index of the newly created reports and join this with the old
index? any other professional solution?
Thanks
Vijay
On 4/30/10, Ian Lea <ia...@gmail.com> wrote:
> You don't say what version of lucene you are using, but in recent
> versions you may need to use SimpleFSLockFactory rather than the
> default, NativeFSLockFactory. See the javadocs. Lucene in general
> does work on NFS but there can be problems, particularly with
> concurrent access from multiple servers.
>
> But your problem sounds different. Does the NFS mount work properly
> for non-lucene operations? Or maybe you could maintain a copy of the
> index on local disks and just copy it to NFS for access from other
> servers. Lucene locking should take care of locking the index when
> updates are running.
>
>
> --
> Ian.
>
>
> On Fri, Apr 30, 2010 at 5:48 AM, Vijay Veeraraghavan
> <vi...@gmail.com> wrote:
>> dear all,
>>
>> I have a problem using lucene in NFS. A scheduler runs periodically
>> generating reports in pdf format and saves it to a file server. The
>> drive of the file server is mounted to the scheduler server (NFS).
>> After generating reports finally the scheduler indexes the names of
>> the report and its path. This index is also stored to the same context
>> folder the reports are created, ie., mounted NFS folder. When i create
>> index in the local system (JFS2) its as fast as possible. (the local
>> system is AIX uses jfs2.) But in the NFS folder it keep doing and
>> doing... i didnt exactly know what happened inside. little later the
>> application freezed.I saw some lock files created inside the folder
>> but nothing like index files that usually created by the lucene. from
>> the web page i search this index in nfs for the report and read it. My
>> question is how to use lucene in NFS. anything special we need to do?
>> any reference codes? docs? How do i lock the index when my scheduler
>> is updating the index?
>>
>> P.S: Please add CC to vijayv@kggroup.com
>>
>> Thanks
>> Vijay Veeraraghavan
>>
>>
>>
>> --
>> Thanks & Regards
>> Vijay Veeraraghavan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Thanks & Regards
Vijay Veeraraghavan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Using lucene in NFS
Posted by Ian Lea <ia...@gmail.com>.
You don't say what version of lucene you are using, but in recent
versions you may need to use SimpleFSLockFactory rather than the
default, NativeFSLockFactory. See the javadocs. Lucene in general
does work on NFS but there can be problems, particularly with
concurrent access from multiple servers.
But your problem sounds different. Does the NFS mount work properly
for non-lucene operations? Or maybe you could maintain a copy of the
index on local disks and just copy it to NFS for access from other
servers. Lucene locking should take care of locking the index when
updates are running.
--
Ian.
On Fri, Apr 30, 2010 at 5:48 AM, Vijay Veeraraghavan
<vi...@gmail.com> wrote:
> dear all,
>
> I have a problem using lucene in NFS. A scheduler runs periodically
> generating reports in pdf format and saves it to a file server. The
> drive of the file server is mounted to the scheduler server (NFS).
> After generating reports finally the scheduler indexes the names of
> the report and its path. This index is also stored to the same context
> folder the reports are created, ie., mounted NFS folder. When i create
> index in the local system (JFS2) its as fast as possible. (the local
> system is AIX uses jfs2.) But in the NFS folder it keep doing and
> doing... i didnt exactly know what happened inside. little later the
> application freezed.I saw some lock files created inside the folder
> but nothing like index files that usually created by the lucene. from
> the web page i search this index in nfs for the report and read it. My
> question is how to use lucene in NFS. anything special we need to do?
> any reference codes? docs? How do i lock the index when my scheduler
> is updating the index?
>
> P.S: Please add CC to vijayv@kggroup.com
>
> Thanks
> Vijay Veeraraghavan
>
>
>
> --
> Thanks & Regards
> Vijay Veeraraghavan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org