You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by eks dev <ek...@yahoo.co.uk> on 2006/05/31 16:51:22 UTC
trivial util to Visualize BitSets (Query results actually)
Maybe there are some more people that like to see bits. Feel free to do whatever you like with it.
Idea is simple, map 8 bits from HitCollector to one pixel by changing gray levels. Implementation is Quick 'n Dirty, but does the job.
/**
* Copyright 2004 The Apache Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import javax.imageio.ImageIO;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.queryParser.ParseException;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.search.HitCollector;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.Searcher;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import java.awt.image.*;
import java.io.File;
import java.io.IOException;
import java.util.Arrays;
import java.util.BitSet;
/*
* Generates .png image of the search result. Maps bits from result
* sequentially from left to right untill specified image lenth is achieved
* and then continues from the begining in the next line.
*
* Mapps 8 bits to one pixel, by changing leves of gray
* e.g
* WHITE PIXEL: 0 of eight bits set
* GRAY PIXEL: some bits set (more set => darker)
* BLACK PIXEL: all 8 bits set
*
* main has 3 params: index path, Query, and optional imageWidth
*/
public class VisualizeBitVector {
// Invoke the init method in BufImagePanel to build the image.
public static void main(String[] args) throws IOException, ParseException {
Img fImg;
Directory dir;
IndexReader reader;
fImg = new Img();
if (args.length < 2) {
System.out.println("VisualizeBitVector index_path query [imageWidth]");
System.exit(-1);
}
String path = args[0];
String query = args[1];
int imageWidth = 800;
if(args.length > 2){
imageWidth = Integer.parseInt(args[2]);
}
System.getProperties().setProperty(
"org.apache.lucene.FSDirectory.class",
"org.apache.lucene.store.MMapDirectory");
dir = FSDirectory.getDirectory(path, false);
reader = IndexReader.open(dir);
BufferedImage img;
img = fImg.makeImage(reader, query, imageWidth);
File file = new File("queryBits.png");
ImageIO.write(img, "png", file);
}
}
class Img {
BitSet bs;
BufferedImage fBufferedImage = null;
int fWidth = 0, fHeight = 0;
byte[] fPixels = null;
/** Build a BufferedImage from a pixel array. *
* @throws ParseException */
public BufferedImage makeImage(IndexReader reader, String query, int imageWidth)
throws IOException, ParseException {
int maxDocs = reader.maxDoc();
fWidth = imageWidth;
fHeight = (maxDocs / 8 / fWidth) + 1;
fPixels = new byte[fWidth * fHeight];
//reset
Arrays.fill(fPixels, (byte) (0 | 0xFF));
Searcher s = new IndexSearcher(reader);
bs = new BitSet(maxDocs);
HitCollector hc = new HitCollector() {
public void collect(int doc, float score) {
bs.set(doc);
}
};
Analyzer analyzer = new StandardAnalyzer();
QueryParser parser = new QueryParser("", analyzer);
Query q = parser.parse(query);
s.search(q, null, hc);
//This is the only interesting thing here
for (int i = bs.nextSetBit(0); i >= 0; i = bs.nextSetBit(i + 1)) {
int pos = i >> 3;
fPixels[pos] = (byte) ((fPixels[pos] & 0xFF) >>> 1);
}
s.close();
reader.close();
// Create a BufferedIamge of the gray values in bytes.
fBufferedImage = new BufferedImage(fWidth, fHeight,
BufferedImage.TYPE_BYTE_GRAY);
// Get the writable raster so that data can be changed.
WritableRaster wr = fBufferedImage.getRaster();
// Now write the byte data to the raster
wr.setDataElements(0, 0, fWidth, fHeight, fPixels);
return fBufferedImage;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Lexicon access questions
Posted by Chuck Williams <ch...@manawiz.com>.
This approach comes to mind. You could model your semantic tags as
tokens and index them at the same positions as the words or phrases to
which they apply. This is particularly easy if you can integrate your
taggers with your Analyzer. You would probably want to create one or
more new Query subclasses to facilitate certain types of matching,
making it easy to associate terms/phrases with different tags (e.g.,
OverlappingQuery). This approach would support generation of queries
that are tag-dependent, but would not directly help using tags in a
ranking algorithm for tag-independent queries. As an off-hand thought,
you might be able to extend the idea to support this by naming your tags
something like TERM_TAG where TERM is the term they apply to (best if
the character used for '_' cannot occur in any term). Then something
like a TaggedTermQuery could easily find the tags relevant to a term in
the query and iterate their docs/positions in parallel with those of the
term (rougly equilvaent to OverlappingQuery(term, PrefixQuery(term_*))).
Top-of-mind thoughts,
Chuck
eks dev wrote on 06/01/2006 12:10 AM:
> We have faced the following use case:
>
> In order to optimize performance and more importantly quality of search results we are forced to attach more attributes to particular words (Terms). Generic attributes like TF, IDF are usefull to model our "similarity" only up to some level.
>
> Examples:
> 1. Is one Term first or last name, (e.g. we have comprehensive list of such words). This enables us to make smarter (faster and better queries) in case someone has multiple first names, it influences ranking...
> 2. Agreement weight and Disagreement weigt of some words is modelled diferently.
> 3. Semantic classes of words influence ranking (if something verb or noun changes search strategy and ranking radically)
>
> On top of that, we can afford to load all terms in memory, in order to alow fast string distance callculations and some limited pattern matching using some strange Trie-s.
>
> Today, we solve these things by implementing totally redundant data structures that keep some kind of map Term->ValuesObject, which is redundant to Lucene Lexicon storage. Instead of "one access gets all" we have two access terms using two diferent access paths, once using our dictionary and second time implicitly via Query or so... So we introduce performance/memory penalties. (Pls. do not forget, we need to access copy of analyzed document in order to attach "additional info" to Terms)
>
> I guess we are not the only ones to face such a case, as increase in precision above TF/IDF can be only achieved by introducing some "domain semantics" where available. For this, "attaching" domain specific info to Term would be perfect solution. Also, enabling flexible implementations for Lexicon access could give us some flexibility (e.g. implementation in mg4j goes in that direction)
>
> Could somebody imagine 2.x version of Lucene to have some Interface that needs to be implemented with clear contract, that would enable us to attach our implementation for accessing lexicon?
>
> Or even better, some hints how I can do it today :)
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Lexicon access questions
Posted by eks dev <ek...@yahoo.co.uk>.
We have faced the following use case:
In order to optimize performance and more importantly quality of search results we are forced to attach more attributes to particular words (Terms). Generic attributes like TF, IDF are usefull to model our "similarity" only up to some level.
Examples:
1. Is one Term first or last name, (e.g. we have comprehensive list of such words). This enables us to make smarter (faster and better queries) in case someone has multiple first names, it influences ranking...
2. Agreement weight and Disagreement weigt of some words is modelled diferently.
3. Semantic classes of words influence ranking (if something verb or noun changes search strategy and ranking radically)
On top of that, we can afford to load all terms in memory, in order to alow fast string distance callculations and some limited pattern matching using some strange Trie-s.
Today, we solve these things by implementing totally redundant data structures that keep some kind of map Term->ValuesObject, which is redundant to Lucene Lexicon storage. Instead of "one access gets all" we have two access terms using two diferent access paths, once using our dictionary and second time implicitly via Query or so... So we introduce performance/memory penalties. (Pls. do not forget, we need to access copy of analyzed document in order to attach "additional info" to Terms)
I guess we are not the only ones to face such a case, as increase in precision above TF/IDF can be only achieved by introducing some "domain semantics" where available. For this, "attaching" domain specific info to Term would be perfect solution. Also, enabling flexible implementations for Lexicon access could give us some flexibility (e.g. implementation in mg4j goes in that direction)
Could somebody imagine 2.x version of Lucene to have some Interface that needs to be implemented with clear contract, that would enable us to attach our implementation for accessing lexicon?
Or even better, some hints how I can do it today :)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: trivial util to Visualize BitSets (Query results actually)
Posted by eks dev <ek...@yahoo.co.uk>.
right, luke rocks. Such things are best placed in luke indeed...
Image gets just too big without intensity trick, for really large bit vectors, I had to use 32 bits per pixel.
how I can see these plugins, is it somewhere on http://www.getopt.org/luke/?
----- Original Message ----
From: mark harwood <ma...@yahoo.co.uk>
To: java-dev@lucene.apache.org; eks dev <ek...@yahoo.co.uk>
Sent: Wednesday, 31 May, 2006 5:12:06 PM
Subject: Re: trivial util to Visualize BitSets (Query results actually)
I added something similar to Luke but without the
colour intensity - I may add your code in to do this.
Another Luke plugin I have visualizes "vocabulary
growth" for a field as a chart over time. This is
useful to see if a field is "matured" or is still
accumulating new terms.
A Zipf term distribution would be another good
visualization.
--- eks dev <ek...@yahoo.co.uk> wrote:
> Maybe there are some more people that like to see
> bits. Feel free to do whatever you like with it.
>
> Idea is simple, map 8 bits from HitCollector to one
> pixel by changing gray levels. Implementation is
> Quick 'n Dirty, but does the job.
>
> /**
> * Copyright 2004 The Apache Software Foundation
> *
> * Licensed under the Apache License, Version 2.0
> (the "License");
> * you may not use this file except in compliance
> with the License.
> * You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in
> writing, software
> * distributed under the License is distributed on
> an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
> either express or implied.
> * See the License for the specific language
> governing permissions and
> * limitations under the License.
> */
>
> import javax.imageio.ImageIO;
>
> import org.apache.lucene.analysis.Analyzer;
> import
>
org.apache.lucene.analysis.standard.StandardAnalyzer;
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.queryParser.ParseException;
> import org.apache.lucene.queryParser.QueryParser;
> import org.apache.lucene.search.HitCollector;
> import org.apache.lucene.search.IndexSearcher;
> import org.apache.lucene.search.Query;
> import org.apache.lucene.search.Searcher;
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.FSDirectory;
>
> import java.awt.image.*;
> import java.io.File;
> import java.io.IOException;
> import java.util.Arrays;
> import java.util.BitSet;
>
> /*
> * Generates .png image of the search result. Maps
> bits from result
> * sequentially from left to right untill specified
> image lenth is achieved
> * and then continues from the begining in the next
> line.
> *
> * Mapps 8 bits to one pixel, by changing leves of
> gray
> * e.g
> * WHITE PIXEL: 0 of eight bits set
> * GRAY PIXEL: some bits set (more set => darker)
> * BLACK PIXEL: all 8 bits set
> *
> * main has 3 params: index path, Query, and
> optional imageWidth
> */
> public class VisualizeBitVector {
>
> // Invoke the init method in BufImagePanel to
> build the image.
> public static void main(String[] args) throws
> IOException, ParseException {
> Img fImg;
> Directory dir;
> IndexReader reader;
>
> fImg = new Img();
> if (args.length < 2) {
> System.out.println("VisualizeBitVector
> index_path query [imageWidth]");
> System.exit(-1);
> }
>
> String path = args[0];
> String query = args[1];
>
> int imageWidth = 800;
> if(args.length > 2){
> imageWidth = Integer.parseInt(args[2]);
> }
>
> System.getProperties().setProperty(
>
> "org.apache.lucene.FSDirectory.class",
>
> "org.apache.lucene.store.MMapDirectory");
> dir = FSDirectory.getDirectory(path, false);
> reader = IndexReader.open(dir);
>
> BufferedImage img;
>
> img = fImg.makeImage(reader, query,
> imageWidth);
> File file = new File("queryBits.png");
> ImageIO.write(img, "png", file);
>
> }
>
> }
>
>
> class Img {
> BitSet bs;
>
> BufferedImage fBufferedImage = null;
>
> int fWidth = 0, fHeight = 0;
>
> byte[] fPixels = null;
>
> /** Build a BufferedImage from a pixel array. *
> * @throws ParseException */
>
> public BufferedImage makeImage(IndexReader
> reader, String query, int imageWidth)
> throws IOException, ParseException {
>
> int maxDocs = reader.maxDoc();
>
> fWidth = imageWidth;
> fHeight = (maxDocs / 8 / fWidth) + 1;
> fPixels = new byte[fWidth * fHeight];
>
> //reset
> Arrays.fill(fPixels, (byte) (0 | 0xFF));
>
> Searcher s = new IndexSearcher(reader);
> bs = new BitSet(maxDocs);
>
> HitCollector hc = new HitCollector() {
> public void collect(int doc, float
> score) {
> bs.set(doc);
> }
> };
>
> Analyzer analyzer = new StandardAnalyzer();
> QueryParser parser = new QueryParser("",
> analyzer);
> Query q = parser.parse(query);
>
> s.search(q, null, hc);
>
> //This is the only interesting thing here
> for (int i = bs.nextSetBit(0); i >= 0; i =
> bs.nextSetBit(i + 1)) {
> int pos = i >> 3;
> fPixels[pos] = (byte) ((fPixels[pos] &
> 0xFF) >>> 1);
> }
>
>
> s.close();
> reader.close();
>
> // Create a BufferedIamge of the gray values
> in bytes.
> fBufferedImage = new BufferedImage(fWidth,
> fHeight,
> BufferedImage.TYPE_BYTE_GRAY);
>
> // Get the writable raster so that data can
> be changed.
> WritableRaster wr =
> fBufferedImage.getRaster();
>
> // Now write the byte data to the raster
> wr.setDataElements(0, 0, fWidth, fHeight,
> fPixels);
> return fBufferedImage;
> }
>
> }
>
>
>
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail:
> java-dev-help@lucene.apache.org
>
>
___________________________________________________________
Next-generation email? Now you can have it with All New Yahoo! Mail. http://uk.docs.yahoo.com/nowyoucan.html
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: trivial util to Visualize BitSets (Query results actually)
Posted by mark harwood <ma...@yahoo.co.uk>.
I added something similar to Luke but without the
colour intensity - I may add your code in to do this.
Another Luke plugin I have visualizes "vocabulary
growth" for a field as a chart over time. This is
useful to see if a field is "matured" or is still
accumulating new terms.
A Zipf term distribution would be another good
visualization.
--- eks dev <ek...@yahoo.co.uk> wrote:
> Maybe there are some more people that like to see
> bits. Feel free to do whatever you like with it.
>
> Idea is simple, map 8 bits from HitCollector to one
> pixel by changing gray levels. Implementation is
> Quick 'n Dirty, but does the job.
>
> /**
> * Copyright 2004 The Apache Software Foundation
> *
> * Licensed under the Apache License, Version 2.0
> (the "License");
> * you may not use this file except in compliance
> with the License.
> * You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in
> writing, software
> * distributed under the License is distributed on
> an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
> either express or implied.
> * See the License for the specific language
> governing permissions and
> * limitations under the License.
> */
>
> import javax.imageio.ImageIO;
>
> import org.apache.lucene.analysis.Analyzer;
> import
>
org.apache.lucene.analysis.standard.StandardAnalyzer;
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.queryParser.ParseException;
> import org.apache.lucene.queryParser.QueryParser;
> import org.apache.lucene.search.HitCollector;
> import org.apache.lucene.search.IndexSearcher;
> import org.apache.lucene.search.Query;
> import org.apache.lucene.search.Searcher;
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.FSDirectory;
>
> import java.awt.image.*;
> import java.io.File;
> import java.io.IOException;
> import java.util.Arrays;
> import java.util.BitSet;
>
> /*
> * Generates .png image of the search result. Maps
> bits from result
> * sequentially from left to right untill specified
> image lenth is achieved
> * and then continues from the begining in the next
> line.
> *
> * Mapps 8 bits to one pixel, by changing leves of
> gray
> * e.g
> * WHITE PIXEL: 0 of eight bits set
> * GRAY PIXEL: some bits set (more set => darker)
> * BLACK PIXEL: all 8 bits set
> *
> * main has 3 params: index path, Query, and
> optional imageWidth
> */
> public class VisualizeBitVector {
>
> // Invoke the init method in BufImagePanel to
> build the image.
> public static void main(String[] args) throws
> IOException, ParseException {
> Img fImg;
> Directory dir;
> IndexReader reader;
>
> fImg = new Img();
> if (args.length < 2) {
> System.out.println("VisualizeBitVector
> index_path query [imageWidth]");
> System.exit(-1);
> }
>
> String path = args[0];
> String query = args[1];
>
> int imageWidth = 800;
> if(args.length > 2){
> imageWidth = Integer.parseInt(args[2]);
> }
>
> System.getProperties().setProperty(
>
> "org.apache.lucene.FSDirectory.class",
>
> "org.apache.lucene.store.MMapDirectory");
> dir = FSDirectory.getDirectory(path, false);
> reader = IndexReader.open(dir);
>
> BufferedImage img;
>
> img = fImg.makeImage(reader, query,
> imageWidth);
> File file = new File("queryBits.png");
> ImageIO.write(img, "png", file);
>
> }
>
> }
>
>
> class Img {
> BitSet bs;
>
> BufferedImage fBufferedImage = null;
>
> int fWidth = 0, fHeight = 0;
>
> byte[] fPixels = null;
>
> /** Build a BufferedImage from a pixel array. *
> * @throws ParseException */
>
> public BufferedImage makeImage(IndexReader
> reader, String query, int imageWidth)
> throws IOException, ParseException {
>
> int maxDocs = reader.maxDoc();
>
> fWidth = imageWidth;
> fHeight = (maxDocs / 8 / fWidth) + 1;
> fPixels = new byte[fWidth * fHeight];
>
> //reset
> Arrays.fill(fPixels, (byte) (0 | 0xFF));
>
> Searcher s = new IndexSearcher(reader);
> bs = new BitSet(maxDocs);
>
> HitCollector hc = new HitCollector() {
> public void collect(int doc, float
> score) {
> bs.set(doc);
> }
> };
>
> Analyzer analyzer = new StandardAnalyzer();
> QueryParser parser = new QueryParser("",
> analyzer);
> Query q = parser.parse(query);
>
> s.search(q, null, hc);
>
> //This is the only interesting thing here
> for (int i = bs.nextSetBit(0); i >= 0; i =
> bs.nextSetBit(i + 1)) {
> int pos = i >> 3;
> fPixels[pos] = (byte) ((fPixels[pos] &
> 0xFF) >>> 1);
> }
>
>
> s.close();
> reader.close();
>
> // Create a BufferedIamge of the gray values
> in bytes.
> fBufferedImage = new BufferedImage(fWidth,
> fHeight,
> BufferedImage.TYPE_BYTE_GRAY);
>
> // Get the writable raster so that data can
> be changed.
> WritableRaster wr =
> fBufferedImage.getRaster();
>
> // Now write the byte data to the raster
> wr.setDataElements(0, 0, fWidth, fHeight,
> fPixels);
> return fBufferedImage;
> }
>
> }
>
>
>
>
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail:
> java-dev-help@lucene.apache.org
>
>
___________________________________________________________
Next-generation email? Now you can have it with All New Yahoo! Mail. http://uk.docs.yahoo.com/nowyoucan.html
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org