You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Dawid Weiss <da...@cs.put.poznan.pl> on 2011/04/04 11:29:44 UTC
Automaton serialization (IndexOutput/IndexInput).
Hi guys. FST currently uses IndexInput/IndexOutput to perform
serialization and the suggester framework expects a single file. Is
there any nice way to do this? I ask because I considered these, but
they all seem clumsy:
a) serialize suggester data to a subfolder (using regular
FSDirectory); this would be different than other suggesters,
b) use FSDirectory, serialize using this and then move the file from a
subfolder, then delete it.
c) use custom subclasses of IndexInput IndexOutput that accept a
simple File and serialize directy.
At the moment I use option (c), actually, but better hints would be welcome.
Dawid
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Automaton serialization (IndexOutput/IndexInput).
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
No, it's a single file, but the only way to serialize it is via
IndexOutput/IndexInput and these require a Directory. I'll switch to
using a subfolder, but this use case makes me think a simple
serializer/deserializer from InputStream/OutputStream would be
probably a good addition to the automaton class...
Dawid
On Mon, Apr 4, 2011 at 1:39 PM, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 4/4/11 1:26 PM, Dawid Weiss wrote:
>>
>> Yeah, I know, but you create a subfile in that folder and in case of
>> FSDirectory a subfolder would be created for the automaton... so for
>> 'parent' you'd have something like:
>>
>> parent/fstlookup/fst.file
>>
>> Can it look this way or is:
>>
>> parent/fst.file
>>
>> more suitable? This is most likely a matter of taste, but I wanted to
>> confirm it first.
>
> I think that no-one should be looking in that folder unless they want to
> implement their own lookup (and then he just needs to avoid a name clash) or
> need to debug things. I think that if you need to create a bunch of files
> for your Lookup then it's better if they are in a subdir, so as not to mix
> them with data files from other Lookup-s.
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Automaton serialization (IndexOutput/IndexInput).
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 4/4/11 1:26 PM, Dawid Weiss wrote:
> Yeah, I know, but you create a subfile in that folder and in case of
> FSDirectory a subfolder would be created for the automaton... so for
> 'parent' you'd have something like:
>
> parent/fstlookup/fst.file
>
> Can it look this way or is:
>
> parent/fst.file
>
> more suitable? This is most likely a matter of taste, but I wanted to
> confirm it first.
I think that no-one should be looking in that folder unless they want to
implement their own lookup (and then he just needs to avoid a name
clash) or need to debug things. I think that if you need to create a
bunch of files for your Lookup then it's better if they are in a subdir,
so as not to mix them with data files from other Lookup-s.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Automaton serialization (IndexOutput/IndexInput).
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
Yeah, I know, but you create a subfile in that folder and in case of
FSDirectory a subfolder would be created for the automaton... so for
'parent' you'd have something like:
parent/fstlookup/fst.file
Can it look this way or is:
parent/fst.file
more suitable? This is most likely a matter of taste, but I wanted to
confirm it first.
Dawid
On Mon, Apr 4, 2011 at 11:40 AM, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 4/4/11 11:29 AM, Dawid Weiss wrote:
>>
>> Hi guys. FST currently uses IndexInput/IndexOutput to perform
>> serialization and the suggester framework expects a single file. Is
>> there any nice way to do this? I ask because I considered these, but
>> they all seem clumsy:
>>
>> a) serialize suggester data to a subfolder (using regular
>> FSDirectory); this would be different than other suggesters,
>> b) use FSDirectory, serialize using this and then move the file from a
>> subfolder, then delete it.
>> c) use custom subclasses of IndexInput IndexOutput that accept a
>> simple File and serialize directy.
>>
>> At the moment I use option (c), actually, but better hints would be
>> welcome.
>
> Suggester expects a File, but both Jaspell and TST actually use this File as
> a parent directory name where they store their data.
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Automaton serialization (IndexOutput/IndexInput).
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 4/4/11 11:29 AM, Dawid Weiss wrote:
> Hi guys. FST currently uses IndexInput/IndexOutput to perform
> serialization and the suggester framework expects a single file. Is
> there any nice way to do this? I ask because I considered these, but
> they all seem clumsy:
>
> a) serialize suggester data to a subfolder (using regular
> FSDirectory); this would be different than other suggesters,
> b) use FSDirectory, serialize using this and then move the file from a
> subfolder, then delete it.
> c) use custom subclasses of IndexInput IndexOutput that accept a
> simple File and serialize directy.
>
> At the moment I use option (c), actually, but better hints would be welcome.
Suggester expects a File, but both Jaspell and TST actually use this
File as a parent directory name where they store their data.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org