You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lenya.apache.org by Andreas Hartmann <an...@apache.org> on 2009/06/30 17:36:29 UTC

Tag cloud

Hi Lenya devs,

I volunteered to take the lead in the implementation of the "tag cloud" 
feature (see [1]).

Some initial ideas:

IMO it makes sense to use the Dublin Core element "subject" to assign 
tags to a document [2].

Definition: "The topic of the resource."
Comment: "Typically, the subject will be represented using keywords, key 
phrases, or classification codes. Recommended best practice is to use a 
controlled vocabulary. To describe the spatial or temporal topic of the 
resource, use the Coverage element."

I guess this can be made configurable, we could just use the DC subject 
as the default. Since tags can contain spaces, we should use multiple 
meta data values to store multiple tags. A nice GUI for this has to be 
implemented. Would it be sufficient to extend the standard meta data GUI 
to allow entering multiple values, or do we need a dedicated tag 
management GUI? I'd suggest to start with the existing meta data GUI.


Finding all documents with a certain tag is rather simple since all meta 
data are indexed. The real challenge is to generate a list of all 
existing tags.

Maybe there is a performant way to generate the cloud using the index, 
e.g. via a wildcard query. But this still needs some postprocessing, so 
we'll probably have to cache the tag cloud.

If Lucene doesn't help, we have another nifty feature for this purpose: 
the RepositoryListener interface. By registering a listener with the 
repository, we can extract the tags of a document when it is saved, and 
update the tag cloud accordingly. The cloud also has to be updated when 
a document is removed. The details are a bit tricky (concurrency, 
queuing), but I think there's nothing that can't be solved. In this case 
we have to store the tag cloud. My first idea would be to use a 
dedicated document for this purpose.

I'd prefer the dynamic generation using Lucene, though, because 
otherwise we store redundant information in the repository which always 
carries a certain risk.


Another issue is supporting the user when she enters the tags. The 
system should present a list of existing tags, possibly with some kind 
of autocomplete functionality. But I guess when we manage to generate 
the cloud, this feature can easily be added.


Any comments and ideas are very welcome!


[1] http://wiki.apache.org/lenya/ModulesIdeas
[2] http://dublincore.org/documents/dces/

-- Andreas


-- 
Andreas Hartmann, CTO
BeCompany GmbH
http://www.becompany.ch
Tel.: +41 (0) 43 818 57 01


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lenya.apache.org
For additional commands, e-mail: dev-help@lenya.apache.org


Re: Tag cloud

Posted by Andreas Hartmann <an...@apache.org>.
Andreas Hartmann schrieb:
> Andreas Hartmann schrieb:
>> I volunteered to take the lead in the implementation of the "tag 
>> cloud" feature (see [1]).
> 
> I added a first version to the contributions area. The tag cloud is 
> visible in the defaultfiredocs publication.
> 
> 
>> Some initial ideas:
>>
>> IMO it makes sense to use the Dublin Core element "subject" to assign 
>> tags to a document [2].

[…]

> I didn't take care of the multi-value handling yet. The tags are just 
> the terms which are indexed by Lucene. TODO: Define the meta element 
> values as keyword index fields instead of text fields to support phrases 
> (multi-word terms).

I have added the option to use keyword-type indexing for meta data, it 
seems to work well. I'm not sure though if we should make the dc:subject 
keyword-style indexed by default, maybe the more common usecase is 
text-style indexing.

BTW, I added the index type configuration to the docs:
https://lenya.zones.apache.org/cms/docu/authoring/docu20/reference/metadata.html

-- Andreas

-- 
Andreas Hartmann, CTO
BeCompany GmbH
http://www.becompany.ch
Tel.: +41 (0) 43 818 57 01


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lenya.apache.org
For additional commands, e-mail: dev-help@lenya.apache.org


Re: Tag cloud

Posted by Andreas Hartmann <an...@apache.org>.
Andreas Hartmann schrieb:
> I volunteered to take the lead in the implementation of the "tag cloud" 
> feature (see [1]).

I added a first version to the contributions area. The tag cloud is 
visible in the defaultfiredocs publication.


> Some initial ideas:
> 
> IMO it makes sense to use the Dublin Core element "subject" to assign 
> tags to a document [2].

I hard-coded this element for the moment.

> Definition: "The topic of the resource."
> Comment: "Typically, the subject will be represented using keywords, key 
> phrases, or classification codes. Recommended best practice is to use a 
> controlled vocabulary. To describe the spatial or temporal topic of the 
> resource, use the Coverage element."
> 
> I guess this can be made configurable, we could just use the DC subject 
> as the default. Since tags can contain spaces, we should use multiple 
> meta data values to store multiple tags. A nice GUI for this has to be 
> implemented. Would it be sufficient to extend the standard meta data GUI 
> to allow entering multiple values, or do we need a dedicated tag 
> management GUI? I'd suggest to start with the existing meta data GUI.

I didn't take care of the multi-value handling yet. The tags are just 
the terms which are indexed by Lucene. TODO: Define the meta element 
values as keyword index fields instead of text fields to support phrases 
(multi-word terms).

> Finding all documents with a certain tag is rather simple since all meta 
> data are indexed.

I used the standard search for this purpose. I had to extend the lucene 
module sitemap with a "raw" query type. The query looks like this:

   \{http\://purl.org/dc/elements/1.1/\}subject:foobar

It's rather ugly that this appears in the search box, maybe we have to 
add a dedicated meta data search or use another concept for special 
search terms.


> The real challenge is to generate a list of all 
> existing tags.
> 
> Maybe there is a performant way to generate the cloud using the index, 
> e.g. via a wildcard query. But this still needs some postprocessing, so 
> we'll probably have to cache the tag cloud.

Lucene allows to enumerate all terms for a particular field. To filter 
the language, I had to add a loop which searches the index for each 
term. I guess this takes quite a lot of time. Maybe someone knows a 
better solution? Or maybe a new version of Lucene has a more flexible 
API for term enumeration?

If you omit the language parameter of the IndexTermsGenerator, the 
language filtering is skipped and the listing of the terms is probably 
pretty fast.


> If Lucene doesn't help, we have another nifty feature for this purpose: 
> the RepositoryListener interface. By registering a listener with the 
> repository, we can extract the tags of a document when it is saved, and 
> update the tag cloud accordingly. The cloud also has to be updated when 
> a document is removed. The details are a bit tricky (concurrency, 
> queuing), but I think there's nothing that can't be solved. In this case 
> we have to store the tag cloud. My first idea would be to use a 
> dedicated document for this purpose.
> 
> I'd prefer the dynamic generation using Lucene, though, because 
> otherwise we store redundant information in the repository which always 
> carries a certain risk.

I think we can use Lucene. No need for the repository listening.

> Another issue is supporting the user when she enters the tags. The 
> system should present a list of existing tags, possibly with some kind 
> of autocomplete functionality. But I guess when we manage to generate 
> the cloud, this feature can easily be added.

I didn't tackle this issue yet.

Any comments and improvements are greatly appreciated!

-- Andreas


-- 
Andreas Hartmann, CTO
BeCompany GmbH
http://www.becompany.ch
Tel.: +41 (0) 43 818 57 01


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lenya.apache.org
For additional commands, e-mail: dev-help@lenya.apache.org