You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by brian4 <bq...@gmail.com> on 2013/08/23 23:54:49 UTC
Empty webpage metadata in IndexingFilter, but not empty in database
I am using Nutch 2.2.1 and hbase.
I am trying to create a plug-in to parse and index specific elements. I
store the parsed values in the webPage metadata field with a ParseFilter (I
verified this worked by retrieving the values right after that).
Then I try to retrieve the values from the metadata in an IndexingFilter,
however, when getting the metadata from the page it returns the metadata Map
but it is always empty so I cannot retrieve the values. However, if I look
at the actual table in HBase afterward, the metadata is fully populated with
all the values from the ParseFilter -it is not empty.
How come I can't retrieve the values in the IndexingFilter (why is it not
seeing the metadata and thinking the field is empty)? Are the pages being
passed to the indexing filter not the actual page as stored in HBase?
--
View this message in context: http://lucene.472066.n3.nabble.com/Empty-webpage-metadata-in-IndexingFilter-but-not-empty-in-database-tp4086419.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Empty webpage metadata in IndexingFilter, but not empty in database
Posted by Lewis John Mcgibbney <le...@gmail.com>.
hi Brian
there's no doubt that we should add this to plugin central.
do you have an interest in doing so?
good to hear you got it working
lewis
On Monday, August 26, 2013, brian4 <bq...@gmail.com> wrote:
> Figured out the issue was I was not explicitly including the metadata
field
> in the indexing filter, I had to do the following:
>
> private static final Collection<WebPage.Field> FIELDS = new
> HashSet<WebPage.Field>();
> static {
> FIELDS.add(WebPage.Field.METADATA);
> }
>
> @Override
> public Collection<WebPage.Field> getFields() {
> return FIELDS;
> }
>
> I don't think this is that well documented - at least I did not realize it
> was necessary for some time. Previous plugins I used were able to get
data
> from the "page" variable without explicitly setting fields, albeit basic
> data (baseUrl), and I was basing my plugin on another plugin that was also
> doing this incorrectly.
>
>
>
>
>
> --
> View this message in context:
http://lucene.472066.n3.nabble.com/Empty-webpage-metadata-in-IndexingFilter-but-not-empty-in-database-tp4086419p4086633.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
*Lewis*
Re: Empty webpage metadata in IndexingFilter, but not empty in
database
Posted by brian4 <bq...@gmail.com>.
Figured out the issue was I was not explicitly including the metadata field
in the indexing filter, I had to do the following:
private static final Collection<WebPage.Field> FIELDS = new
HashSet<WebPage.Field>();
static {
FIELDS.add(WebPage.Field.METADATA);
}
@Override
public Collection<WebPage.Field> getFields() {
return FIELDS;
}
I don't think this is that well documented - at least I did not realize it
was necessary for some time. Previous plugins I used were able to get data
from the "page" variable without explicitly setting fields, albeit basic
data (baseUrl), and I was basing my plugin on another plugin that was also
doing this incorrectly.
--
View this message in context: http://lucene.472066.n3.nabble.com/Empty-webpage-metadata-in-IndexingFilter-but-not-empty-in-database-tp4086419p4086633.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Empty webpage metadata in IndexingFilter, but not empty in
database
Posted by brian4 <bq...@gmail.com>.
Yes I am trying both and both are coming up empty. When I try to get the
value directly I get null (even though it matches what is stored according
to looking at the table in hbase shell, e.g., column=mtdt:h1,... for the
name "h1".
Here is what I use to check the metadata in the indexing filter - the
ByteBuffer returned is null, but when I directly retrieve the metadata is
not null but is empty, which I don't see how that is possible (I've checked
the HBase table after parsing and updating and before indexing and the
metadata field is definitely not empty).
ByteBuffer bb = page.getFromMetadata(new Utf8(targetField));
if (bb != null){
LOG.info("ByteBuffer retrieved: " + bb.toString());
}
Map<Utf8,ByteBuffer> metadata = (Map<Utf8,ByteBuffer>) page.get(22);
if(metadata==null){
LOG.info("metadata is null!");
} else{
LOG.info("metadata is not null, isempty: " + metadata.isEmpty() +" size: "
+ metadata.size());
}
--
View this message in context: http://lucene.472066.n3.nabble.com/Empty-webpage-metadata-in-IndexingFilter-but-not-empty-in-database-tp4086419p4086627.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Empty webpage metadata in IndexingFilter, but not empty in database
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Brian,
When getting your metadata from the WebPage are you obtaining the fulle mat
e.g. page.getMetadata() or are you trying to get a Key and obtain the
ByteBuffer value? e.g. page.getFromMetadata(Utf8 Key)?
The latter will return null if nothing is present which is normal but means
that the problem is therefore during the page.putToMetadata call.
It is strange that you are seeing data written to HBase but not when you
make the call.
thnaks
Lewis
On Friday, August 23, 2013, brian4 <bq...@gmail.com> wrote:
> I am using Nutch 2.2.1 and hbase.
>
> I am trying to create a plug-in to parse and index specific elements. I
> store the parsed values in the webPage metadata field with a ParseFilter
(I
> verified this worked by retrieving the values right after that).
>
> Then I try to retrieve the values from the metadata in an IndexingFilter,
> however, when getting the metadata from the page it returns the metadata
Map
> but it is always empty so I cannot retrieve the values. However, if I
look
> at the actual table in HBase afterward, the metadata is fully populated
with
> all the values from the ParseFilter -it is not empty.
>
> How come I can't retrieve the values in the IndexingFilter (why is it not
> seeing the metadata and thinking the field is empty)? Are the pages being
> passed to the indexing filter not the actual page as stored in HBase?
>
>
>
>
> --
> View this message in context:
http://lucene.472066.n3.nabble.com/Empty-webpage-metadata-in-IndexingFilter-but-not-empty-in-database-tp4086419.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
--
*Lewis*