You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by George Aroush <ge...@aroush.net> on 2007/05/01 03:02:05 UTC
RE: Submitting new filter
Hi Petr,
No, you don't have to port it to Java (unless if you see a value in doing so
and you want to.)
The process of submitting new code is documented here:
http://wiki.apache.org/lucene-java/HowToContribute Yes, this is for the
Java Lucene, but the process is very much the same for Lucene.Net.
Let me know if any of this is not clear and we can work on it.
Regards,
-- George
> -----Original Message-----
> From: Petr Kadlec [mailto:petr.kadlec@gmail.com]
> Sent: Monday, April 30, 2007 11:01 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: Submitting new filter
>
> Hi all!
>
> I have created a new filter for Lucene that strips diacritics
> (there is a similar one, but it is able to process Latin-1
> charset only), because that makes one of the basic
> capabilities for a search engine for the Czech language.
>
> And my question is: do I have to write a Java version of this
> filter and get it submitted into the Java Lucene, and only
> after that would the filter be admitted to Lucene.Net? (Not
> that it would be any problem, the filter is quite simple.)
>
> Regards,
> -- Petr Kadlec
>
Re: Submitting new filter
Posted by Petr Kadlec <pe...@gmail.com>.
Hi!
> What are the glitches that you found in the Demo project? Lets fix them.
Well, without any real testing, I have found a few tiny problems:
***
--- IndexFiles.cs (revision 534902)
+++ IndexFiles.cs (working copy)
@@ -78,7 +78,7 @@
writer.Close();
System.DateTime end = System.DateTime.Now;
-
System.Console.Out.WriteLine(end.Millisecond - start.Millisecond + "
total milliseconds");
+ System.Console.Out.WriteLine((end -
start).TotalMilliseconds + " total milliseconds");
}
catch (System.IO.IOException e)
{
--- FileDocument.cs (revision 536850)
+++ FileDocument.cs (working copy)
@@ -53,7 +53,7 @@
// Add the last modified date of the file a field named
"modified". Use
// a field that is indexed (i.e. searchable), but don't
tokenize the field
// into words.
- doc.Add(new Field("modified",
DateTools.TimeToString(f.LastWriteTime.Millisecond,
DateTools.Resolution.MINUTE), Field.Store.YES,
Field.Index.UN_TOKENIZED));
+ doc.Add(new Field("modified",
DateTools.TimeToString(f.LastWriteTime.Ticks,
DateTools.Resolution.MINUTE), Field.Store.YES,
Field.Index.UN_TOKENIZED));
// Add the contents of the file to a field named
"contents". Specify a Reader,
// so that the text of the file is tokenized and indexed,
but not stored.
***
But, in general, I was quite surprised by the behavior of the
searching application with regards to the charset. It forces a
hard-coded character encoding, and it does it in a strange way that
makes it impossible for me to search for accented characters in the
interactive mode. (The batch mode using -queries works fine if the
given file is in ANSI encoding.) So my question is whether the
behavior is taken from the Java version (which I did not test), or if
it is a bug made during conversion to .NET.
> Also, the demo IndexHtml isn't working. Do you want to take a look?
Well, I might try to test that.
-- Petr
Extending IndexModifier
Posted by Digy <di...@gmail.com>.
Hi George,
I want to propose some extensions for IndexModifier.
1- Adding method "AddIndexes" asked by Dave (Sun, 13 May 2007 23:29:37 GMT).
2- Adding method "UpdateDocuments".
I have a program that performs the below task every night
for each updated documents
delete documents(term)
add document(doc)
endfor
But as mentioned in the document
(http://lucene.apache.org/java/1_9_0/api/org/apache/lucene/index/IndexModifi
er.html
"While you can freely mix calls to add() and delete() using this class, you
should batch you calls for best performance. For example, if you want to
update 20 documents, you should first delete all those documents, then add
all the new documents. ")
it is not a good choise for coding.
Of course the programming logic can be changed, but i think it could be a
good solution for everyone to add a performance upgrade to the IndexModifier
where IndexModifiers defers the "AddDocument" method and continues to
perform "DeleteDocuments" until some threshold value is reached.
With this patch, my nightly job turns into
for each updated documents
UpdateDocuments(term,doc)
Endfor
So, if you don't have any objection about extending classes of
Java-compatible-Lucene.Net, i/you can open an issue on JIRA.
I attached the patch for my proposed solution to the end of this e-mail.
DIGY
PATCH:
+-+-+-+-+-+-+-+-+-+-+- IndexModifier +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Index: IndexModifier.cs
===================================================================
--- IndexModifier.cs (revision 536611)
+++ IndexModifier.cs (working copy)
@@ -108,6 +108,11 @@
protected internal int maxBufferedDocs;
protected internal int maxFieldLength;
protected internal int mergeFactor;
+
+ public const int DEFAULT_MAX_WAITING_DOCS_TO_BE_INDEXED = 20;
+ public int MaxWaitingDocsToBeIndexed =
DEFAULT_MAX_WAITING_DOCS_TO_BE_INDEXED;
+
+ internal System.Collections.Generic.List<SingleDoc> DocsToIndex =
new System.Collections.Generic.List<SingleDoc>();
/// <summary> Open an index with write access.
///
@@ -210,6 +215,7 @@
{
if (indexWriter != null)
{
+ UpdateIndex(true);
indexWriter.Close();
indexWriter = null;
}
@@ -224,6 +230,7 @@
lock (directory)
{
AssureOpen();
+ UpdateIndex(true);
if (indexWriter != null)
{
indexWriter.Close();
@@ -540,6 +547,9 @@
{
if (!open)
throw new
System.SystemException("Index is closed already");
+
+ UpdateIndex(true);
+
if (indexWriter != null)
{
indexWriter.Close();
@@ -575,5 +585,81 @@
System.out.println(indexModifier.docCount() + " docs in
index");
indexModifier.close();
}*/
+
+
+ /// <summary>
+ /// @link
Lucene.Net.Index.IndexWriter#AddIndexes(Lucene.Net.Index.IndexReader[])
+ /// </summary>
+ /// <param name="readers"></param>
+ public void AddIndexes(IndexReader[] readers)
+ {
+ AssureOpen();
+ CreateIndexWriter();
+ indexWriter.AddIndexes(readers);
+ }
+
+ /// <summary>
+ /// @link
Lucene.Net.Index.IndexWriter#AddIndexes(Lucene.Net.Store.Directory[])
+ /// </summary>
+ /// <param name="dirs"></param>
+ public void AddIndexes(Lucene.Net.Store.Directory[] dirs)
+ {
+ AssureOpen();
+ CreateIndexWriter();
+ indexWriter.AddIndexes(dirs);
+ }
+
+
+ private void UpdateIndex(bool IndexImmediately)
+ {
+ if (DocsToIndex.Count == 0) return;
+
+ if (IndexImmediately == true || DocsToIndex.Count >=
MaxWaitingDocsToBeIndexed)
+ {
+ foreach (SingleDoc sd in DocsToIndex)
+ {
+ this.AddDocument(sd.Doc, sd.Analyzer);
+ }
+ DocsToIndex.Clear();
+ }
+ }
+
+ /// <summary>
+ /// Updates the index. First deletes the document(s) matching
"TermToDelete" then adds the "NewDoc" to the index.
+ /// IndexImmediately is used to defer the indexing for speed
optimization.
+ /// </summary>
+ /// <param name="TermToDelete"></param>
+ /// <param name="NewDoc"></param>
+ /// <param name="Analyzer"></param>
+ /// <param name="IndexImmediately">when false(default) Adding To
index is deferred until Document count waiting to be indexed reaches
MaxWaitingDocsToBeIndexed(Default=20)</param>
+ public void UpdateDocuments(Term TermToDelete, Document NewDoc,
Analyzer Analyzer, bool IndexImmediately)
+ {
+ this.DeleteDocuments(TermToDelete);
+ DocsToIndex.Add(new SingleDoc(NewDoc, Analyzer));
+ UpdateIndex(IndexImmediately);
+ }
+
+ public void UpdateDocuments(Term TermToDelete, Document NewDoc,
Analyzer Analyzer)
+ {
+ UpdateDocuments(TermToDelete, NewDoc, Analyzer, false);
+ }
+
+ public void UpdateDocuments(Term TermToDelete, Document NewDoc)
+ {
+ UpdateDocuments(TermToDelete, NewDoc, null, false);
+ }
}
+
+ internal class SingleDoc
+ {
+ internal Document Doc = null;
+ internal Analyzer Analyzer = null;
+
+ internal SingleDoc(Document Doc, Analyzer Analyzer)
+ {
+ this.Doc = Doc;
+ this.Analyzer = Analyzer;
+ }
+ }
+
}
\ No newline at end of file
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Re: Problem in searching the indexed files.
Posted by "naveen.srikakolanu" <na...@exensys.com>.
Dear Ayende Rahien,
Thanks for your response. I am not able to index the files which are
ascii encrypted means i am encoding the file extension so that the files are
getting skipped to index. Please clarify me whether the indexing is done
based on IFilters for each provided file type. If i want to index any new
file type then please let me know the IFilters available for the existing
Lucene Indexing.
With Warm Regards,
S. Naveen Kumar.
Senior Software Engineer
Exensys Software Solutions Ltd.
Phone: +91-40-23000891/2/3
Extn:1050
Fax: +91-40-23391105
Mobile: 9949519255
Website: www.exensys.com
---------------------Legal Disclaimer---------------------------------
Confidential Information: The information contained in this mail is
confidential and protected from general disclosure. If the recipient or the
reader of this e-mail is not the intended recipient, or person responsible
to receive this e-mail, you are requested to delete this mail immediately
and do not disseminate or distribute or copy. If you have received this
e-mail by mistake, please notify us immediately by replying to the message
so that we can take appropriate action immediately and see to it that this
mistake
----- Original Message -----
From: "Ayende Rahien" <ay...@ayende.com>
To: <lu...@incubator.apache.org>
Sent: Friday, May 18, 2007 1:27 PM
Subject: Re: Problem in searching the indexed files.
> It would help to know what the problem is? Full exception stack + message
> would be a good start.
>
> On 5/18/07, naveen.srikakolanu <na...@exensys.com> wrote:
> >
> > Dear George,
> >
> > I have implemented Lucene Indexing well and it was working fine
in
> > windows application but i am not able to search the indexed files from
web
> > application can you please tell me how to deal this issue.
> >
> > I have one more problem i am not able to index the hashed files.
> > please help me in these issues as soon as i am in urgent.
> >
> >
> > Thanks,
> > With Warm Regards,
> >
> > S. Naveen Kumar.
> > Senior Software Engineer
> > Exensys Software Solutions Ltd.
> > Phone: +91-40-23000891/2/3
> > Extn:1050
> > Fax: +91-40-23391105
> > Mobile: 9949519255
> > Website: www.exensys.com
> >
> > ---------------------Legal Disclaimer---------------------------------
> > Confidential Information: The information contained in this mail is
> > confidential and protected from general disclosure. If the recipient or
> > the
> > reader of this e-mail is not the intended recipient, or person
responsible
> > to receive this e-mail, you are requested to delete this mail
immediately
> > and do not disseminate or distribute or copy. If you have received this
> > e-mail by mistake, please notify us immediately by replying to the
message
> > so that we can take appropriate action immediately and see to it that
this
> > mistake
> >
> >
> >
> >
> >
> >
>
Re: Problem in searching the indexed files.
Posted by Ayende Rahien <ay...@ayende.com>.
It would help to know what the problem is? Full exception stack + message
would be a good start.
On 5/18/07, naveen.srikakolanu <na...@exensys.com> wrote:
>
> Dear George,
>
> I have implemented Lucene Indexing well and it was working fine in
> windows application but i am not able to search the indexed files from web
> application can you please tell me how to deal this issue.
>
> I have one more problem i am not able to index the hashed files.
> please help me in these issues as soon as i am in urgent.
>
>
> Thanks,
> With Warm Regards,
>
> S. Naveen Kumar.
> Senior Software Engineer
> Exensys Software Solutions Ltd.
> Phone: +91-40-23000891/2/3
> Extn:1050
> Fax: +91-40-23391105
> Mobile: 9949519255
> Website: www.exensys.com
>
> ---------------------Legal Disclaimer---------------------------------
> Confidential Information: The information contained in this mail is
> confidential and protected from general disclosure. If the recipient or
> the
> reader of this e-mail is not the intended recipient, or person responsible
> to receive this e-mail, you are requested to delete this mail immediately
> and do not disseminate or distribute or copy. If you have received this
> e-mail by mistake, please notify us immediately by replying to the message
> so that we can take appropriate action immediately and see to it that this
> mistake
>
>
>
>
>
>
Problem in searching the indexed files.
Posted by "naveen.srikakolanu" <na...@exensys.com>.
Dear George,
I have implemented Lucene Indexing well and it was working fine in
windows application but i am not able to search the indexed files from web
application can you please tell me how to deal this issue.
I have one more problem i am not able to index the hashed files.
please help me in these issues as soon as i am in urgent.
Thanks,
With Warm Regards,
S. Naveen Kumar.
Senior Software Engineer
Exensys Software Solutions Ltd.
Phone: +91-40-23000891/2/3
Extn:1050
Fax: +91-40-23391105
Mobile: 9949519255
Website: www.exensys.com
---------------------Legal Disclaimer---------------------------------
Confidential Information: The information contained in this mail is
confidential and protected from general disclosure. If the recipient or the
reader of this e-mail is not the intended recipient, or person responsible
to receive this e-mail, you are requested to delete this mail immediately
and do not disseminate or distribute or copy. If you have received this
e-mail by mistake, please notify us immediately by replying to the message
so that we can take appropriate action immediately and see to it that this
mistake
RE: Submitting new filter
Posted by George Aroush <ge...@aroush.net>.
Thanks Petr. I will add this to SVN over the weekend.
What are the glitches that you found in the Demo project? Lets fix them.
Also, the demo IndexHtml isn't working. Do you want to take a look?
-- George
> -----Original Message-----
> From: Petr Kadlec [mailto:petr.kadlec@gmail.com]
> Sent: Thursday, May 03, 2007 11:32 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: Re: Submitting new filter
>
> Hi!
>
> Thanks for the info, I have created an issue in Jira and
> attached the code to it, would you take a look at it, please?
>
> I have also prepared a simple Analyzer class for the Czech
> language that uses this filter (and supplies a stopwords
> list), but my question
> is: where should that go? In Java Lucene, those
> language-specific analyzers are in contrib, although there is
> an empty directory "de" in main code's Analysis. But I guess
> that should be the proper place:
> there is the directory Standard, and "non-standard" analyzers
> should be there, too (IMHO). Anyway: when you have the
> language-specific analyzers split to another
> project/solution, it gets quite inconvenient.
>
> P.S. I have also found some glitches in the Demo project, but
> it seems to me to be quite problematic generally, so posting
> tiny bugfixes is maybe not too useful.
>
> Regards,
> -- Petr Kadlec
>
Re: Submitting new filter
Posted by Petr Kadlec <pe...@gmail.com>.
Hi!
Thanks for the info, I have created an issue in Jira and attached the
code to it, would you take a look at it, please?
I have also prepared a simple Analyzer class for the Czech language
that uses this filter (and supplies a stopwords list), but my question
is: where should that go? In Java Lucene, those language-specific
analyzers are in contrib, although there is an empty directory "de" in
main code's Analysis. But I guess that should be the proper place:
there is the directory Standard, and "non-standard" analyzers should
be there, too (IMHO). Anyway: when you have the language-specific
analyzers split to another project/solution, it gets quite
inconvenient.
P.S. I have also found some glitches in the Demo project, but it seems
to me to be quite problematic generally, so posting tiny bugfixes is
maybe not too useful.
Regards,
-- Petr Kadlec