Indexing Content of a Direcory

[Robert Naczinski <ro...@googlemail.com>]

Hi all,

i want PERIODIC ( 1x per 30s ) index the content of a directory to
solr. In doing that i will to index only the new files and add it to
the index.

Can I use for that the standard solr, or must I write own indexer? Can
I implement custom DIH.

The files are readable via HTTP.

Can someone give me a tip?

Robert

Re: Indexing Content of a Direcory

[Gora Mohanty <go...@mimirtech.com>]

On Wed, May 11, 2011 at 2:24 PM, Robert Naczinski
<ro...@googlemail.com> wrote:
> Hi,
>
> we try to index logfiles. That are xmlfiles ( this can be not valid )
> and text files. Unfortunately have all files not the same content

Sorry: While Ahmet has done his best to help you out, you are
not providing enough details for people to do that. It is almost
certainly possible that your content can be indexed into Solr.
You need to provide much more detail about the format of the
files that you wish to index, how you have configured Solr,
what are the error messages from Solr logs, etc.

Please take a look at http://wiki.apache.org/solr/UsingMailingLists

Regards,
Gora

Re: Indexing Content of a Direcory

[Robert Naczinski <ro...@googlemail.com>]

Hi,

we try to index logfiles. That are xmlfiles ( this can be not valid )
and text files. Unfortunately have all files not the same content

Regards,

Robert

2011/5/11 Ahmet Arslan <io...@yahoo.com>:
>> Can I configure with this components native solr or must I
>> write own
>> Indexer that use
>> org.apache.solr.handler.dataimport.FileListEntityProcessor?
>
> You can use native solr. All things are configured in xml files, data-config.xml, schema.xml etc.
>
> There are examples in the directory example-DIH/solr{db,mail,rss,tika,ms}
> They all use dataimport.
>
>> Someware must I transform the inputdocuments to the schema.
>> My
>> iputdocuments are in different format.
>
> What are your input documents? pdf word excel?
>

Re: Indexing Content of a Direcory

[Ahmet Arslan <io...@yahoo.com>]

> Can I configure with this components native solr or must I
> write own
> Indexer that use
> org.apache.solr.handler.dataimport.FileListEntityProcessor?

You can use native solr. All things are configured in xml files, data-config.xml, schema.xml etc.

There are examples in the directory example-DIH/solr{db,mail,rss,tika,ms}
They all use dataimport. 

> Someware must I transform the inputdocuments to the schema.
> My
> iputdocuments are in different format.

What are your input documents? pdf word excel?

Re: Indexing Content of a Direcory

[Robert Naczinski <ro...@googlemail.com>]

Thanx for your help.

Can I configure with this components native solr or must I write own
Indexer that use
org.apache.solr.handler.dataimport.FileListEntityProcessor?

Someware must I transform the inputdocuments to the schema. My
iputdocuments are in different format.

Robert

2011/5/11 Ahmet Arslan <io...@yahoo.com>:
>
>> i want PERIODIC ( 1x per 30s ) index the content of a
>> directory to
>> solr. In doing that i will to index only the new files and
>> add it to
>> the index.
>
> Probably you can use DIH, there are different DataSource implementations.
> http://wiki.apache.org/solr/DataImportHandler#DataSource
>
> For example you can index files in a directory recursively using FileListEntityProcessor.
>

Re: Indexing Content of a Direcory

[Ahmet Arslan <io...@yahoo.com>]

> i want PERIODIC ( 1x per 30s ) index the content of a
> directory to
> solr. In doing that i will to index only the new files and
> add it to
> the index.

Probably you can use DIH, there are different DataSource implementations.
http://wiki.apache.org/solr/DataImportHandler#DataSource

For example you can index files in a directory recursively using FileListEntityProcessor.