You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by Benjamin Liles <bl...@library.tamu.edu> on 2009/02/25 15:32:51 UTC
DataImportHandler XPathEntityProcessor DeltaImport Support
I was looking at adding support to the XPathEntityProcessor for a
DeltaImport by adding a new parameter, deltaUrl, in order to support
harvesting from an OAI-PMH server into SOLR. I wasn't planning on
bothering with getting commit access to the repository so I was
looking to extend the XPathEntityProcessor in my own package. I ran
into a problem because most of the methods on the class are private
instead of protected. I only want to override a methods and not all of
them.
Should I just modify the XPathEntityProcessor to add the functionality
and submit a patch instead of trying to write my own class that
extends the XPathEntityProcessor?
Was there a reason for making the methods be private?
Benjamin Liles
Lead Software App Developer
Texas A&M University General Libraries
PH 979.862.4948 ex 122
bliles@library.tamu.edu
Re: DataImportHandler XPathEntityProcessor DeltaImport Support
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
On Wed, Feb 25, 2009 at 9:01 PM, Benjamin Liles <bl...@library.tamu.edu> wrote:
>>
>> Cool, does OAI-PMH expose a method identify deltas and deletions?
>>
>
> With OAI-PMH you can request items that have been modified in a date range.
> The deletes in OAI-PMH will require some kind of a transformer anyway
> because they appear as normal records with most of the metadata missing.
I guess it should be simple to do delta using full-import as I have mentioned
>
>> Just a thought, would adding a deltaUrl confuse users because most APIs do
>> not support delta? Another option (for your use-case) could be to add
>> another root entity specific for delta import only. DIH supports indexing
>> one or more root entities independently of others.
>>
>
> Maybe there should be a different name for it, but I think it should be
> fairly easy to explain that not all API's would support it.
>
>>
>> Public methods come with a baggage because changes have to be
>> back-compatible across releases. But we can make methods public which can
>> be
>> useful to a large number of use-cases.
>>
>
> I wasn't thinking of public methods, but protected. It would be the
> responsibility of the developer extending the class to keep their code up to
> date with the latest API.
even protected methods are public API's . I am not suggesting we will
not do it. Just let us know which all methods you wish to make
protected . Let us so what ever is best
>
> As mentioned in another e-mail. I may be trying to hard. I'm not sure, but I
> just had a realization that I may be able to include the parameters for the
> 'delta' in the baseUrl of an HttpDataSource and that would solve all of my
> problems.
>
>
> Benjamin Liles
> Lead Software App Developer
> Texas A&M University General Libraries
> PH 979.862.4948 ex 122
> bliles@library.tamu.edu
>
>
--
--Noble Paul
Re: DataImportHandler XPathEntityProcessor DeltaImport Support
Posted by Benjamin Liles <bl...@library.tamu.edu>.
>
> Cool, does OAI-PMH expose a method identify deltas and deletions?
>
With OAI-PMH you can request items that have been modified in a date
range. The deletes in OAI-PMH will require some kind of a transformer
anyway because they appear as normal records with most of the metadata
missing.
> Just a thought, would adding a deltaUrl confuse users because most
> APIs do
> not support delta? Another option (for your use-case) could be to add
> another root entity specific for delta import only. DIH supports
> indexing
> one or more root entities independently of others.
>
Maybe there should be a different name for it, but I think it should
be fairly easy to explain that not all API's would support it.
>
> Public methods come with a baggage because changes have to be
> back-compatible across releases. But we can make methods public
> which can be
> useful to a large number of use-cases.
>
I wasn't thinking of public methods, but protected. It would be the
responsibility of the developer extending the class to keep their code
up to date with the latest API.
As mentioned in another e-mail. I may be trying to hard. I'm not sure,
but I just had a realization that I may be able to include the
parameters for the 'delta' in the baseUrl of an HttpDataSource and
that would solve all of my problems.
Benjamin Liles
Lead Software App Developer
Texas A&M University General Libraries
PH 979.862.4948 ex 122
bliles@library.tamu.edu
Re: DataImportHandler XPathEntityProcessor DeltaImport Support
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Wed, Feb 25, 2009 at 8:02 PM, Benjamin Liles <bl...@library.tamu.edu>wrote:
> I was looking at adding support to the XPathEntityProcessor for a
> DeltaImport by adding a new parameter, deltaUrl, in order to support
> harvesting from an OAI-PMH server into SOLR. I wasn't planning on bothering
> with getting commit access to the repository so I was looking to extend the
> XPathEntityProcessor in my own package. I ran into a problem because most
> of the methods on the class are private instead of protected. I only want to
> override a methods and not all of them.
Cool, does OAI-PMH expose a method identify deltas and deletions?
Just a thought, would adding a deltaUrl confuse users because most APIs do
not support delta? Another option (for your use-case) could be to add
another root entity specific for delta import only. DIH supports indexing
one or more root entities independently of others.
> Should I just modify the XPathEntityProcessor to add the functionality and
> submit a patch instead of trying to write my own class that extends the
> XPathEntityProcessor?
>
> Was there a reason for making the methods be private?
Public methods come with a baggage because changes have to be
back-compatible across releases. But we can make methods public which can be
useful to a large number of use-cases.
--
Regards,
Shalin Shekhar Mangar.
Re: DataImportHandler XPathEntityProcessor DeltaImport Support
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
you can share the details on how you were identifying deltas and what
are the methods you would like to make protected.
delta import necessarily does not need the entityprocessor to support
it. you can easily do delta-import with a full-import .
http://wiki.apache.org/solr/DataImportHandlerFaq#head-0d5144d35dfdba4ba9e182a46ed71c15c3b5a1c6
On Wed, Feb 25, 2009 at 8:02 PM, Benjamin Liles <bl...@library.tamu.edu> wrote:
> I was looking at adding support to the XPathEntityProcessor for a
> DeltaImport by adding a new parameter, deltaUrl, in order to support
> harvesting from an OAI-PMH server into SOLR. I wasn't planning on bothering
> with getting commit access to the repository so I was looking to extend the
> XPathEntityProcessor in my own package. I ran into a problem because most
> of the methods on the class are private instead of protected. I only want to
> override a methods and not all of them.
>
> Should I just modify the XPathEntityProcessor to add the functionality and
> submit a patch instead of trying to write my own class that extends the
> XPathEntityProcessor?
>
> Was there a reason for making the methods be private?
>
> Benjamin Liles
> Lead Software App Developer
> Texas A&M University General Libraries
> PH 979.862.4948 ex 122
> bliles@library.tamu.edu
>
>
--
--Noble Paul