You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by David Worms <da...@simpledesign.com> on 2003/02/17 23:32:44 UTC

[LARM] using SEDA, pros and cons

In order to move our project to the next level, it has been discussed 
the use of an SEDA architecture.

With the help of the Excalibur / Event library, I created a simplified 
version of Larm which use SEDA as its backbone architecture. The goal 
was not to make it pretty or even working, but to see how each 
application components could feet in an event stage architecture.

The great thing I notice is how flexible the application become. it is 
extremely easy to map each stage with the others. One of the features 
of LARM is the ability to have different sources( db, web, filesystem, 
... ), process them, and store them (lucene index, log, ...). This 
seems easily achieveable with SEDA. Also clustering the LARM could be 
done through a specific stage implementation.

This was for the pros. However, it will be great to get some feedback 
because I am really not sure on how to deal with SEDA. here is some 
problems I am facing.

- In order for the crawler to be efficient, I had to raise the number 
of threads, but from what I read in the past, only one or two threads 
should be used in a SEDA environment.

- Also, it looks to consume a lot of memory, which could be due to the 
number of messages put into the queue.

Clemens, and others, please have a look of it, and give me some 
feedback.

http://67.116.155.180/~wdavidw/stage.zip

David.


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: PLAN: WebLucene -- Lucene Web interface, use XML as a lightweight protocol.

Posted by Che Dong <ch...@hotmail.com>.
Yes, I think compare to JavaBean and SOAP based API,   HTTP/URI/XML-based API is much simpler.

I'd like to here more opintions at this project starting.

Regards

Che, Dong
----- Original Message ----- 
From: "Michael Wechner" <mi...@wyona.org>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Cc: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Thursday, February 20, 2003 6:13 PM
Subject: Re: PLAN: WebLucene -- Lucene Web interface, use XML as a lightweight protocol.


> That's very interesting.
> 
> I have tried something similar by integrating
> Lucene into Wyona, which is a CMS based on Cocoon,
> and I also separated Structure from Layout. You can try it out at
> 
> HTML:
> 
> http://195.226.6.70:8080/wyona-cms/oscom/search-oscom/lucene?publication-id=all&queryString=Cocoon+Wyona&fields=all&find=Search
> 
> XML:
> 
> http://195.226.6.70:8080/wyona-cms/oscom/search-oscom/lucene.xml?publication-id=all&queryString=Cocoon+Wyona&fields=all&find=Search
> 
> I think XooMLe also did a pretty good job:
> 
> http://www.dentedreality.com.au/xoomle/search/
> 
> Maybe we find a way how to join efforts
> 
> Thanks
> 
> Michael
> 
> 
> Che Dong wrote:
> > http://sourceforge.net/projects/weblucene/
> > 
> > WebLucene: Lucene Web interface, use XML as a lightweight protocol. 
> > 
> > Developer convert data source (text, DB, MS Word, PDF... etc) into standard xml format indexing with lucene engine, and get full text search result via HTTP, with XML format output, user can easily intergrated with JSP ASP PHP front end or use XSLT at server side transform output.
> > 
> > Developer can intergrate lucene full text search engine with old MSSQL + ASP MySQL + PHP Oracle + JSP based web applications.
> > 
> >     MySQL  \                                                  / JSP    
> >     Oracle - DB      -  ==>   XML ==> (Lucene Index) ==> XML  -  ASP    
> >     MSSQL  /                                                  -  PHP            
> >              MS Word /                                        \         / XHTML
> >                  PDF /                                         =XSLT=> -  text
> >                                                                         \ XML
> >                                                                      
> >                          \_________Web Lucene____________/ 
> >                            
> > i18n issue: for Java is Unicode based, user can indexing data source(XML) in different charset into one lucene index(in unicode) and output result according to client browser support languages.
> >       GBK          \                                       / BIG5
> >       BIG5          -  UNICODE        ====>       Unicode -  GB2312
> >       SJIS          -   (XML)                     (XML)   -  SJIS
> >       ISO-8859-1   /                                       \ ISO-8859-1
> > 
> > 
> > Che, Dong
> > http://www.chedong.com/tech/
> > 
> > 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 

Re: PLAN: WebLucene -- Lucene Web interface, use XML as a lightweight protocol.

Posted by Michael Wechner <mi...@wyona.org>.
That's very interesting.

I have tried something similar by integrating
Lucene into Wyona, which is a CMS based on Cocoon,
and I also separated Structure from Layout. You can try it out at

HTML:

http://195.226.6.70:8080/wyona-cms/oscom/search-oscom/lucene?publication-id=all&queryString=Cocoon+Wyona&fields=all&find=Search

XML:

http://195.226.6.70:8080/wyona-cms/oscom/search-oscom/lucene.xml?publication-id=all&queryString=Cocoon+Wyona&fields=all&find=Search

I think XooMLe also did a pretty good job:

http://www.dentedreality.com.au/xoomle/search/

Maybe we find a way how to join efforts

Thanks

Michael


Che Dong wrote:
> http://sourceforge.net/projects/weblucene/
> 
> WebLucene: Lucene Web interface, use XML as a lightweight protocol. 
> 
> Developer convert data source (text, DB, MS Word, PDF... etc) into standard xml format indexing with lucene engine, and get full text search result via HTTP, with XML format output, user can easily intergrated with JSP ASP PHP front end or use XSLT at server side transform output.
> 
> Developer can intergrate lucene full text search engine with old MSSQL + ASP MySQL + PHP Oracle + JSP based web applications.
> 
>     MySQL  \                                                  / JSP    
>     Oracle - DB      -  ==>   XML ==> (Lucene Index) ==> XML  -  ASP    
>     MSSQL  /                                                  -  PHP            
>              MS Word /                                        \         / XHTML
>                  PDF /                                         =XSLT=> -  text
>                                                                         \ XML
>                                                                      
>                          \_________Web Lucene____________/ 
>                            
> i18n issue: for Java is Unicode based, user can indexing data source(XML) in different charset into one lucene index(in unicode) and output result according to client browser support languages.
>       GBK          \                                       / BIG5
>       BIG5          -  UNICODE        ====>       Unicode -  GB2312
>       SJIS          -   (XML)                     (XML)   -  SJIS
>       ISO-8859-1   /                                       \ ISO-8859-1
> 
> 
> Che, Dong
> http://www.chedong.com/tech/
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Re: PLAN: WebLucene -- Lucene Web interface, use XML as a lightweight protocol.

Posted by Michael Wechner <mi...@wyona.org>.
That's very interesting.

I have tried something similar by integrating
Lucene into Wyona, which is a CMS based on Cocoon,
and I also separated Structure from Layout. You can try it out at

HTML:

http://195.226.6.70:8080/wyona-cms/oscom/search-oscom/lucene?publication-id=all&queryString=Cocoon+Wyona&fields=all&find=Search

XML:

http://195.226.6.70:8080/wyona-cms/oscom/search-oscom/lucene.xml?publication-id=all&queryString=Cocoon+Wyona&fields=all&find=Search

I think XooMLe also did a pretty good job:

http://www.dentedreality.com.au/xoomle/search/

Maybe we find a way how to join efforts

Thanks

Michael


Che Dong wrote:
> http://sourceforge.net/projects/weblucene/
> 
> WebLucene: Lucene Web interface, use XML as a lightweight protocol. 
> 
> Developer convert data source (text, DB, MS Word, PDF... etc) into standard xml format indexing with lucene engine, and get full text search result via HTTP, with XML format output, user can easily intergrated with JSP ASP PHP front end or use XSLT at server side transform output.
> 
> Developer can intergrate lucene full text search engine with old MSSQL + ASP MySQL + PHP Oracle + JSP based web applications.
> 
>     MySQL  \                                                  / JSP    
>     Oracle - DB      -  ==>   XML ==> (Lucene Index) ==> XML  -  ASP    
>     MSSQL  /                                                  -  PHP            
>              MS Word /                                        \         / XHTML
>                  PDF /                                         =XSLT=> -  text
>                                                                         \ XML
>                                                                      
>                          \_________Web Lucene____________/ 
>                            
> i18n issue: for Java is Unicode based, user can indexing data source(XML) in different charset into one lucene index(in unicode) and output result according to client browser support languages.
>       GBK          \                                       / BIG5
>       BIG5          -  UNICODE        ====>       Unicode -  GB2312
>       SJIS          -   (XML)                     (XML)   -  SJIS
>       ISO-8859-1   /                                       \ ISO-8859-1
> 
> 
> Che, Dong
> http://www.chedong.com/tech/
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


PLAN: WebLucene -- Lucene Web interface, use XML as a lightweight protocol.

Posted by Che Dong <ch...@hotmail.com>.
http://sourceforge.net/projects/weblucene/

WebLucene: Lucene Web interface, use XML as a lightweight protocol. 

Developer convert data source (text, DB, MS Word, PDF... etc) into standard xml format indexing with lucene engine, and get full text search result via HTTP, with XML format output, user can easily intergrated with JSP ASP PHP front end or use XSLT at server side transform output.

Developer can intergrate lucene full text search engine with old MSSQL + ASP MySQL + PHP Oracle + JSP based web applications.

    MySQL  \                                                  / JSP    
    Oracle - DB      -  ==>   XML ==> (Lucene Index) ==> XML  -  ASP    
    MSSQL  /                                                  -  PHP            
             MS Word /                                        \         / XHTML
                 PDF /                                         =XSLT=> -  text
                                                                        \ XML
                                                                     
                         \_________Web Lucene____________/ 
                           
i18n issue: for Java is Unicode based, user can indexing data source(XML) in different charset into one lucene index(in unicode) and output result according to client browser support languages.
      GBK          \                                       / BIG5
      BIG5          -  UNICODE        ====>       Unicode -  GB2312
      SJIS          -   (XML)                     (XML)   -  SJIS
      ISO-8859-1   /                                       \ ISO-8859-1


Che, Dong
http://www.chedong.com/tech/



Re: [LARM] using SEDA, pros and cons

Posted by Clemens Marschner <cm...@lanlab.de>.
Couldn't look at your code yet, but:

AFAIK Seda was developed for asynchronous I/O, which would mean a redesign
of the central FetcherTask class. If every thread downloads 50 files at
once, you only need a couple of them in parallel to saturate the network
interfaces.

Clemens


----- Original Message -----
From: "David Worms" <da...@simpledesign.com>
To: <lu...@jakarta.apache.org>
Cc: "Avalon users" <av...@jakarta.apache.org>
Sent: Monday, February 17, 2003 11:32 PM
Subject: [LARM] using SEDA, pros and cons


>
> In order to move our project to the next level, it has been discussed
> the use of an SEDA architecture.
>
> With the help of the Excalibur / Event library, I created a simplified
> version of Larm which use SEDA as its backbone architecture. The goal
> was not to make it pretty or even working, but to see how each
> application components could feet in an event stage architecture.
>
> The great thing I notice is how flexible the application become. it is
> extremely easy to map each stage with the others. One of the features
> of LARM is the ability to have different sources( db, web, filesystem,
> ... ), process them, and store them (lucene index, log, ...). This
> seems easily achieveable with SEDA. Also clustering the LARM could be
> done through a specific stage implementation.
>
> This was for the pros. However, it will be great to get some feedback
> because I am really not sure on how to deal with SEDA. here is some
> problems I am facing.
>
> - In order for the crawler to be efficient, I had to raise the number
> of threads, but from what I read in the past, only one or two threads
> should be used in a SEDA environment.
>
> - Also, it looks to consume a lot of memory, which could be due to the
> number of messages put into the queue.
>
> Clemens, and others, please have a look of it, and give me some
> feedback.
>
> http://67.116.155.180/~wdavidw/stage.zip
>
> David.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


PLAN: WebLucene -- Lucene Web interface, use XML as a lightweight protocol.

Posted by Che Dong <ch...@hotmail.com>.
http://sourceforge.net/projects/weblucene/

WebLucene: Lucene Web interface, use XML as a lightweight protocol. 

Developer convert data source (text, DB, MS Word, PDF... etc) into standard xml format indexing with lucene engine, and get full text search result via HTTP, with XML format output, user can easily intergrated with JSP ASP PHP front end or use XSLT at server side transform output.

Developer can intergrate lucene full text search engine with old MSSQL + ASP MySQL + PHP Oracle + JSP based web applications.

    MySQL  \                                                  / JSP    
    Oracle - DB      -  ==>   XML ==> (Lucene Index) ==> XML  -  ASP    
    MSSQL  /                                                  -  PHP            
             MS Word /                                        \         / XHTML
                 PDF /                                         =XSLT=> -  text
                                                                        \ XML
                                                                     
                         \_________Web Lucene____________/ 
                           
i18n issue: for Java is Unicode based, user can indexing data source(XML) in different charset into one lucene index(in unicode) and output result according to client browser support languages.
      GBK          \                                       / BIG5
      BIG5          -  UNICODE        ====>       Unicode -  GB2312
      SJIS          -   (XML)                     (XML)   -  SJIS
      ISO-8859-1   /                                       \ ISO-8859-1


Che, Dong
http://www.chedong.com/tech/