You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by blazingwolf7 <bl...@gmail.com> on 2008/07/10 03:01:10 UTC
.fdt file
Hi,
I had recently found out that Lucene will retrieve the content of a document
from a file ".fdt". I am trying to retrieve the entire file in one go
instead of retrieving it based on document number. can it be done?
--
View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18373913.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by blazingwolf7 <bl...@gmail.com>.
Thanks. I think I will follow the advice. But just for the sack of curiosity,
can what I suggest be done ?
Yonik Seeley wrote:
>
> On Thu, Jul 10, 2008 at 1:42 AM, blazingwolf7 <bl...@gmail.com>
> wrote:
>> Well, I am trying to extract the URL and contentLength from the ".fdt"
>> file.
>> I am planning to use both of these values in a filter to remove certain
>> links to be display in the search result. The problem is, I am told not
>> to
>> use the IndexReader to retrieve these values for each document found
>> matching with the query.
>>
>> So now, instead, I will have to retrieve the entire .fdt file, extract
>> both
>> the values and store it into an arraylist which will be use later. I am
>> having problem extracting the entire file without using all the seek()
>> method to determine the position of the document.
>>
>> Any suggestion?
>
> You're trying to do things at too low of a level (bypassing Lucene's
> public APIs)
> I suggested earlier that you index the URL untokenized, and then use
> the FieldCache. That will allow you to easily retrieve a String[] of
> all the URLs.
>
> -Yonik
>
>
>> Yonik Seeley wrote:
>>>
>>> On Wed, Jul 9, 2008 at 11:13 PM, blazingwolf7 <bl...@gmail.com>
>>> wrote:
>>>> Sorry,but I am still quite new to Lucene. What exactly is "cp"?
>>>
>>> The unix command for copy (hence the smiley).
>>>
>>> Some of your recent questions seem to be suffering from an XY problem:
>>> http://www.perlmonks.org/index.pl?node_id=542341
>>> You may get more help by explaining what you are trying to do.
>>>
>>> -Yonik
>>>
>>>> Yonik Seeley wrote:
>>>>>
>>>>> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <bl...@gmail.com>
>>>>> wrote:
>>>>>> I had recently found out that Lucene will retrieve the content of a
>>>>>> document
>>>>>> from a file ".fdt". I am trying to retrieve the entire file in one go
>>>>>> instead of retrieving it based on document number. can it be done?
>>>>>
>>>>> "cp" can retrieve the file on one go ;-)
>>>>>
>>>>> Other than that, the format is documented here:
>>>>> http://lucene.apache.org/java/docs/fileformats.html
>>>>> But I'm not sure why retrieving by document number won't work for you.
>>>>>
>>>>> -Yonik
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/.fdt-file-tp18373913p18376301.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18394786.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Jul 10, 2008 at 1:42 AM, blazingwolf7 <bl...@gmail.com> wrote:
> Well, I am trying to extract the URL and contentLength from the ".fdt" file.
> I am planning to use both of these values in a filter to remove certain
> links to be display in the search result. The problem is, I am told not to
> use the IndexReader to retrieve these values for each document found
> matching with the query.
>
> So now, instead, I will have to retrieve the entire .fdt file, extract both
> the values and store it into an arraylist which will be use later. I am
> having problem extracting the entire file without using all the seek()
> method to determine the position of the document.
>
> Any suggestion?
You're trying to do things at too low of a level (bypassing Lucene's
public APIs)
I suggested earlier that you index the URL untokenized, and then use
the FieldCache. That will allow you to easily retrieve a String[] of
all the URLs.
-Yonik
> Yonik Seeley wrote:
>>
>> On Wed, Jul 9, 2008 at 11:13 PM, blazingwolf7 <bl...@gmail.com>
>> wrote:
>>> Sorry,but I am still quite new to Lucene. What exactly is "cp"?
>>
>> The unix command for copy (hence the smiley).
>>
>> Some of your recent questions seem to be suffering from an XY problem:
>> http://www.perlmonks.org/index.pl?node_id=542341
>> You may get more help by explaining what you are trying to do.
>>
>> -Yonik
>>
>>> Yonik Seeley wrote:
>>>>
>>>> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <bl...@gmail.com>
>>>> wrote:
>>>>> I had recently found out that Lucene will retrieve the content of a
>>>>> document
>>>>> from a file ".fdt". I am trying to retrieve the entire file in one go
>>>>> instead of retrieving it based on document number. can it be done?
>>>>
>>>> "cp" can retrieve the file on one go ;-)
>>>>
>>>> Other than that, the format is documented here:
>>>> http://lucene.apache.org/java/docs/fileformats.html
>>>> But I'm not sure why retrieving by document number won't work for you.
>>>>
>>>> -Yonik
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18376301.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by Michael McCandless <lu...@mikemccandless.com>.
I think the FieldCache API fits for this...
Its purpose is to do a one-time caching of a particular field's
values, rendered as a straight array keyed by document ID. So
building that cache initially takes time and memory, but then getting
the value for a particular document is extremely fast assuming your OS
is not swapping...
Mike
blazingwolf7 wrote:
>
> Well, according to him, using the reader to access the index
> everytime a
> document is found to retrieve certain values is inefficient. Meaning
> if
> there is 500k document, the index will be access 500k times. It
> might affect
> the performance of the search.
>
> So I am instructed to retrieve all the necessary values at the
> beginning of
> the search and store it. Later the values will be retrieve from
> there. I am
> cracking my head trying to do that%-|
>
>
> Grant Ingersoll-6 wrote:
>>
>>
>> On Jul 10, 2008, at 1:42 AM, blazingwolf7 wrote:
>>
>>>
>>> Well, I am trying to extract the URL and contentLength from the
>>> ".fdt" file.
>>> I am planning to use both of these values in a filter to remove
>>> certain
>>> links to be display in the search result. The problem is, I am told
>>> not to
>>> use the IndexReader to retrieve these values for each document found
>>> matching with the query.
>>
>> Are you implying that using the IR would solve your problem, but for
>> some reason you're architect (or whatever you call the person making
>> the decisions) told you not to? If so, can you explain more the
>> reasoning?
>>
>>>
>>>
>>> So now, instead, I will have to retrieve the entire .fdt file,
>>> extract both
>>> the values and store it into an arraylist which will be use later.
>>> I am
>>> having problem extracting the entire file without using all the
>>> seek()
>>> method to determine the position of the document.
>>>
>>> Any suggestion?
>>>
>>>
>>> Yonik Seeley wrote:
>>>>
>>>> On Wed, Jul 9, 2008 at 11:13 PM, blazingwolf7
>>>> <bl...@gmail.com>
>>>> wrote:
>>>>> Sorry,but I am still quite new to Lucene. What exactly is "cp"?
>>>>
>>>> The unix command for copy (hence the smiley).
>>>>
>>>> Some of your recent questions seem to be suffering from an XY
>>>> problem:
>>>> http://www.perlmonks.org/index.pl?node_id=542341
>>>> You may get more help by explaining what you are trying to do.
>>>>
>>>> -Yonik
>>>>
>>>>> Yonik Seeley wrote:
>>>>>>
>>>>>> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <blazingwolf7@gmail.com
>>>>>>>
>>>>>> wrote:
>>>>>>> I had recently found out that Lucene will retrieve the content
>>>>>>> of a
>>>>>>> document
>>>>>>> from a file ".fdt". I am trying to retrieve the entire file in
>>>>>>> one go
>>>>>>> instead of retrieving it based on document number. can it be
>>>>>>> done?
>>>>>>
>>>>>> "cp" can retrieve the file on one go ;-)
>>>>>>
>>>>>> Other than that, the format is documented here:
>>>>>> http://lucene.apache.org/java/docs/fileformats.html
>>>>>> But I'm not sure why retrieving by document number won't work for
>>>>>> you.
>>>>>>
>>>>>> -Yonik
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/.fdt-file-tp18373913p18376301.html
>>> Sent from the Lucene - Java Users mailing list archive at
>>> Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>> Lucene Helpful Hints:
>> http://wiki.apache.org/lucene-java/BasicsOfPerformance
>> http://wiki.apache.org/lucene-java/LuceneFAQ
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18395519.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by blazingwolf7 <bl...@gmail.com>.
Well, according to him, using the reader to access the index everytime a
document is found to retrieve certain values is inefficient. Meaning if
there is 500k document, the index will be access 500k times. It might affect
the performance of the search.
So I am instructed to retrieve all the necessary values at the beginning of
the search and store it. Later the values will be retrieve from there. I am
cracking my head trying to do that%-|
Grant Ingersoll-6 wrote:
>
>
> On Jul 10, 2008, at 1:42 AM, blazingwolf7 wrote:
>
>>
>> Well, I am trying to extract the URL and contentLength from the
>> ".fdt" file.
>> I am planning to use both of these values in a filter to remove
>> certain
>> links to be display in the search result. The problem is, I am told
>> not to
>> use the IndexReader to retrieve these values for each document found
>> matching with the query.
>
> Are you implying that using the IR would solve your problem, but for
> some reason you're architect (or whatever you call the person making
> the decisions) told you not to? If so, can you explain more the
> reasoning?
>
>>
>>
>> So now, instead, I will have to retrieve the entire .fdt file,
>> extract both
>> the values and store it into an arraylist which will be use later.
>> I am
>> having problem extracting the entire file without using all the seek()
>> method to determine the position of the document.
>>
>> Any suggestion?
>>
>>
>> Yonik Seeley wrote:
>>>
>>> On Wed, Jul 9, 2008 at 11:13 PM, blazingwolf7
>>> <bl...@gmail.com>
>>> wrote:
>>>> Sorry,but I am still quite new to Lucene. What exactly is "cp"?
>>>
>>> The unix command for copy (hence the smiley).
>>>
>>> Some of your recent questions seem to be suffering from an XY
>>> problem:
>>> http://www.perlmonks.org/index.pl?node_id=542341
>>> You may get more help by explaining what you are trying to do.
>>>
>>> -Yonik
>>>
>>>> Yonik Seeley wrote:
>>>>>
>>>>> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <blazingwolf7@gmail.com
>>>>> >
>>>>> wrote:
>>>>>> I had recently found out that Lucene will retrieve the content
>>>>>> of a
>>>>>> document
>>>>>> from a file ".fdt". I am trying to retrieve the entire file in
>>>>>> one go
>>>>>> instead of retrieving it based on document number. can it be done?
>>>>>
>>>>> "cp" can retrieve the file on one go ;-)
>>>>>
>>>>> Other than that, the format is documented here:
>>>>> http://lucene.apache.org/java/docs/fileformats.html
>>>>> But I'm not sure why retrieving by document number won't work for
>>>>> you.
>>>>>
>>>>> -Yonik
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/.fdt-file-tp18373913p18376301.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18395519.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by Grant Ingersoll <gs...@apache.org>.
On Jul 10, 2008, at 1:42 AM, blazingwolf7 wrote:
>
> Well, I am trying to extract the URL and contentLength from the
> ".fdt" file.
> I am planning to use both of these values in a filter to remove
> certain
> links to be display in the search result. The problem is, I am told
> not to
> use the IndexReader to retrieve these values for each document found
> matching with the query.
Are you implying that using the IR would solve your problem, but for
some reason you're architect (or whatever you call the person making
the decisions) told you not to? If so, can you explain more the
reasoning?
>
>
> So now, instead, I will have to retrieve the entire .fdt file,
> extract both
> the values and store it into an arraylist which will be use later.
> I am
> having problem extracting the entire file without using all the seek()
> method to determine the position of the document.
>
> Any suggestion?
>
>
> Yonik Seeley wrote:
>>
>> On Wed, Jul 9, 2008 at 11:13 PM, blazingwolf7
>> <bl...@gmail.com>
>> wrote:
>>> Sorry,but I am still quite new to Lucene. What exactly is "cp"?
>>
>> The unix command for copy (hence the smiley).
>>
>> Some of your recent questions seem to be suffering from an XY
>> problem:
>> http://www.perlmonks.org/index.pl?node_id=542341
>> You may get more help by explaining what you are trying to do.
>>
>> -Yonik
>>
>>> Yonik Seeley wrote:
>>>>
>>>> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <blazingwolf7@gmail.com
>>>> >
>>>> wrote:
>>>>> I had recently found out that Lucene will retrieve the content
>>>>> of a
>>>>> document
>>>>> from a file ".fdt". I am trying to retrieve the entire file in
>>>>> one go
>>>>> instead of retrieving it based on document number. can it be done?
>>>>
>>>> "cp" can retrieve the file on one go ;-)
>>>>
>>>> Other than that, the format is documented here:
>>>> http://lucene.apache.org/java/docs/fileformats.html
>>>> But I'm not sure why retrieving by document number won't work for
>>>> you.
>>>>
>>>> -Yonik
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18376301.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by blazingwolf7 <bl...@gmail.com>.
Well, I am trying to extract the URL and contentLength from the ".fdt" file.
I am planning to use both of these values in a filter to remove certain
links to be display in the search result. The problem is, I am told not to
use the IndexReader to retrieve these values for each document found
matching with the query.
So now, instead, I will have to retrieve the entire .fdt file, extract both
the values and store it into an arraylist which will be use later. I am
having problem extracting the entire file without using all the seek()
method to determine the position of the document.
Any suggestion?
Yonik Seeley wrote:
>
> On Wed, Jul 9, 2008 at 11:13 PM, blazingwolf7 <bl...@gmail.com>
> wrote:
>> Sorry,but I am still quite new to Lucene. What exactly is "cp"?
>
> The unix command for copy (hence the smiley).
>
> Some of your recent questions seem to be suffering from an XY problem:
> http://www.perlmonks.org/index.pl?node_id=542341
> You may get more help by explaining what you are trying to do.
>
> -Yonik
>
>> Yonik Seeley wrote:
>>>
>>> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <bl...@gmail.com>
>>> wrote:
>>>> I had recently found out that Lucene will retrieve the content of a
>>>> document
>>>> from a file ".fdt". I am trying to retrieve the entire file in one go
>>>> instead of retrieving it based on document number. can it be done?
>>>
>>> "cp" can retrieve the file on one go ;-)
>>>
>>> Other than that, the format is documented here:
>>> http://lucene.apache.org/java/docs/fileformats.html
>>> But I'm not sure why retrieving by document number won't work for you.
>>>
>>> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18376301.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Jul 9, 2008 at 11:13 PM, blazingwolf7 <bl...@gmail.com> wrote:
> Sorry,but I am still quite new to Lucene. What exactly is "cp"?
The unix command for copy (hence the smiley).
Some of your recent questions seem to be suffering from an XY problem:
http://www.perlmonks.org/index.pl?node_id=542341
You may get more help by explaining what you are trying to do.
-Yonik
> Yonik Seeley wrote:
>>
>> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <bl...@gmail.com>
>> wrote:
>>> I had recently found out that Lucene will retrieve the content of a
>>> document
>>> from a file ".fdt". I am trying to retrieve the entire file in one go
>>> instead of retrieving it based on document number. can it be done?
>>
>> "cp" can retrieve the file on one go ;-)
>>
>> Other than that, the format is documented here:
>> http://lucene.apache.org/java/docs/fileformats.html
>> But I'm not sure why retrieving by document number won't work for you.
>>
>> -Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by blazingwolf7 <bl...@gmail.com>.
Sorry,but I am still quite new to Lucene. What exactly is "cp"?
Yonik Seeley wrote:
>
> On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <bl...@gmail.com>
> wrote:
>> I had recently found out that Lucene will retrieve the content of a
>> document
>> from a file ".fdt". I am trying to retrieve the entire file in one go
>> instead of retrieving it based on document number. can it be done?
>
> "cp" can retrieve the file on one go ;-)
>
> Other than that, the format is documented here:
> http://lucene.apache.org/java/docs/fileformats.html
> But I'm not sure why retrieving by document number won't work for you.
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/.fdt-file-tp18373913p18375077.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: .fdt file
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Jul 9, 2008 at 9:01 PM, blazingwolf7 <bl...@gmail.com> wrote:
> I had recently found out that Lucene will retrieve the content of a document
> from a file ".fdt". I am trying to retrieve the entire file in one go
> instead of retrieving it based on document number. can it be done?
"cp" can retrieve the file on one go ;-)
Other than that, the format is documented here:
http://lucene.apache.org/java/docs/fileformats.html
But I'm not sure why retrieving by document number won't work for you.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org