You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Pritesh Patel <pr...@gmail.com> on 2014/03/06 20:51:20 UTC
Dataimport handler
I'm using the dataimporthandler to index data from a mysql DB. Been
running it just fine. I've been using full-imports. I'm now trying
implement the delta import functionality.
To implement the delta query, you need to be reading the last_index_time
from a properties file to know what new to index. So I'm using the
parameter:
{dataimporter.last_index_time} within my query.
The problem is when I use this, the date always is : "Thu Jan 01 00:00:00
UTC 1970". It's never actually reading the correct date stored in the
dataimport.properties file.
So my delta query does not work. Has anybody see this issue?
Seems like its always using the beginning date for epoch or unix timestamp
code 0.
--Pritesh
P.S. If you want to see the delta query, see below.
deltaQuery="SELECT node.nid from node where node.type = 'news' and
node.status = 1 and (node.changed >
UNIX_TIMESTAMP('${dataimporter.last_index_time}'jgkg) or node.created >
UNIX_TIMESTAMP('${dataimporter.last_index_time}'))"
deltaImportQuery="SELECT node.nid, node.vid, node.type, node.language,
node.title, node.uid, node.status,
FROM_UNIXTIME(node.created,'%Y-%m-%dT%TZ') as created,
FROM_UNIXTIME(node.changed,'%Y-%m-%dT%TZ') as changed, node.comment,
node.promote, node.moderate, node.sticky, node.tnid, node.translate,
content_type_news.field_image_credit_value,
content_type_news.field_image_caption_value,
content_type_news.field_subhead_value,
content_type_news.field_author_value,
content_type_news.field_dateline_value,
content_type_news.field_article_image_fid,
content_type_news.field_article_image_list,
content_type_news.field_article_image_data,
content_type_news.field_news_blurb_value,
content_type_news.field_news_blurb_format,
content_type_news.field_news_syndicate_value,
content_type_news.field_news_video_reference_nid,
content_type_news.field_news_inline_location_value,
content_type_news.field_article_contributor_nid,
content_type_news.field_news_title_value, page_title.page_title FROM node
LEFT JOIN content_type_news ON node.nid = content_type_news.nid LEFT JOIN
page_title ON node.nid = page_title.id where node.type = 'news' and
node.status = 1 and node.nid = '${deltaimport.delta.nid}'"
Re: Dataimport handler
Posted by Pritesh Patel <pr...@gmail.com>.
Ok. Apologies.
On Thu, Mar 6, 2014 at 10:14 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> Hi Pritesh,
>
> This list is for development related discussions. Please post this on
> solr-user.
>
> On Fri, Mar 7, 2014 at 1:21 AM, Pritesh Patel <pr...@gmail.com>
> wrote:
> > I'm using the dataimporthandler to index data from a mysql DB. Been
> running
> > it just fine. I've been using full-imports. I'm now trying implement the
> > delta import functionality.
> >
> > To implement the delta query, you need to be reading the last_index_time
> > from a properties file to know what new to index. So I'm using the
> > parameter:
> > {dataimporter.last_index_time} within my query.
> >
> > The problem is when I use this, the date always is : "Thu Jan 01 00:00:00
> > UTC 1970". It's never actually reading the correct date stored in the
> > dataimport.properties file.
> >
> > So my delta query does not work. Has anybody see this issue?
> >
> > Seems like its always using the beginning date for epoch or unix
> timestamp
> > code 0.
> >
> > --Pritesh
> >
> > P.S. If you want to see the delta query, see below.
> >
> > deltaQuery="SELECT node.nid from node where node.type = 'news' and
> > node.status = 1 and (node.changed >
> > UNIX_TIMESTAMP('${dataimporter.last_index_time}'jgkg) or node.created
> >
> > UNIX_TIMESTAMP('${dataimporter.last_index_time}'))"
> >
> > deltaImportQuery="SELECT node.nid, node.vid, node.type, node.language,
> > node.title, node.uid, node.status,
> > FROM_UNIXTIME(node.created,'%Y-%m-%dT%TZ') as created,
> > FROM_UNIXTIME(node.changed,'%Y-%m-%dT%TZ') as changed, node.comment,
> > node.promote, node.moderate, node.sticky, node.tnid, node.translate,
> > content_type_news.field_image_credit_value,
> > content_type_news.field_image_caption_value,
> > content_type_news.field_subhead_value,
> content_type_news.field_author_value,
> > content_type_news.field_dateline_value,
> > content_type_news.field_article_image_fid,
> > content_type_news.field_article_image_list,
> > content_type_news.field_article_image_data,
> > content_type_news.field_news_blurb_value,
> > content_type_news.field_news_blurb_format,
> > content_type_news.field_news_syndicate_value,
> > content_type_news.field_news_video_reference_nid,
> > content_type_news.field_news_inline_location_value,
> > content_type_news.field_article_contributor_nid,
> > content_type_news.field_news_title_value, page_title.page_title FROM node
> > LEFT JOIN content_type_news ON node.nid = content_type_news.nid LEFT JOIN
> > page_title ON node.nid = page_title.id where node.type = 'news' and
> > node.status = 1 and node.nid = '${deltaimport.delta.nid}'"
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Dataimport handler
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Pritesh,
This list is for development related discussions. Please post this on solr-user.
On Fri, Mar 7, 2014 at 1:21 AM, Pritesh Patel <pr...@gmail.com> wrote:
> I'm using the dataimporthandler to index data from a mysql DB. Been running
> it just fine. I've been using full-imports. I'm now trying implement the
> delta import functionality.
>
> To implement the delta query, you need to be reading the last_index_time
> from a properties file to know what new to index. So I'm using the
> parameter:
> {dataimporter.last_index_time} within my query.
>
> The problem is when I use this, the date always is : "Thu Jan 01 00:00:00
> UTC 1970". It's never actually reading the correct date stored in the
> dataimport.properties file.
>
> So my delta query does not work. Has anybody see this issue?
>
> Seems like its always using the beginning date for epoch or unix timestamp
> code 0.
>
> --Pritesh
>
> P.S. If you want to see the delta query, see below.
>
> deltaQuery="SELECT node.nid from node where node.type = 'news' and
> node.status = 1 and (node.changed >
> UNIX_TIMESTAMP('${dataimporter.last_index_time}'jgkg) or node.created >
> UNIX_TIMESTAMP('${dataimporter.last_index_time}'))"
>
> deltaImportQuery="SELECT node.nid, node.vid, node.type, node.language,
> node.title, node.uid, node.status,
> FROM_UNIXTIME(node.created,'%Y-%m-%dT%TZ') as created,
> FROM_UNIXTIME(node.changed,'%Y-%m-%dT%TZ') as changed, node.comment,
> node.promote, node.moderate, node.sticky, node.tnid, node.translate,
> content_type_news.field_image_credit_value,
> content_type_news.field_image_caption_value,
> content_type_news.field_subhead_value, content_type_news.field_author_value,
> content_type_news.field_dateline_value,
> content_type_news.field_article_image_fid,
> content_type_news.field_article_image_list,
> content_type_news.field_article_image_data,
> content_type_news.field_news_blurb_value,
> content_type_news.field_news_blurb_format,
> content_type_news.field_news_syndicate_value,
> content_type_news.field_news_video_reference_nid,
> content_type_news.field_news_inline_location_value,
> content_type_news.field_article_contributor_nid,
> content_type_news.field_news_title_value, page_title.page_title FROM node
> LEFT JOIN content_type_news ON node.nid = content_type_news.nid LEFT JOIN
> page_title ON node.nid = page_title.id where node.type = 'news' and
> node.status = 1 and node.nid = '${deltaimport.delta.nid}'"
--
Regards,
Shalin Shekhar Mangar.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org