You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "sowjanya@pointcross.com" <so...@pointcross.com> on 2013/08/09 14:12:06 UTC
Environment Timezone considered When using SolrJ
Hi,
When using SolrJ I've realized document dates are being modified according
to the environment UTC timezone.
I have indexed the large amount of data on date fileds of Solr (using Solr
3.3). While retrieving this date using the SolrJ into SolrDocumentList. The
original date value is modified.
Are we missing any setting at solrConfig. Any Help
--
View this message in context: http://lucene.472066.n3.nabble.com/Environment-Timezone-considered-When-using-SolrJ-tp4083497.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Environment Timezone considered When using SolrJ
Posted by Jack Krupansky <ja...@basetechnology.com>.
What specific evidence do you have that the dates have ben modified?
All Solr dates are specified using the "Z" suffix, meaning GMT.
-- Jack Krupansky
-----Original Message-----
From: sowjanya@pointcross.com
Sent: Friday, August 09, 2013 8:12 AM
To: solr-user@lucene.apache.org
Subject: Environment Timezone considered When using SolrJ
Hi,
When using SolrJ I've realized document dates are being modified according
to the environment UTC timezone.
I have indexed the large amount of data on date fileds of Solr (using Solr
3.3). While retrieving this date using the SolrJ into SolrDocumentList. The
original date value is modified.
Are we missing any setting at solrConfig. Any Help
--
View this message in context:
http://lucene.472066.n3.nabble.com/Environment-Timezone-considered-When-using-SolrJ-tp4083497.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Environment Timezone considered When using SolrJ
Posted by Chris Hostetter <ho...@fucit.org>.
You haven't mentioned what language your update client code is written in
... so i'm not sure what classes "DateTime" and "Convert" are and what
their default behavior is regarding timezones, but w/o that info it's
still pretty clear where your two mistakes are...
: string strStartDate = "26/5/2000 8:38:16 AM";
here's you have a string representation of a date and that string
doesn't contain any TimeZone information -- from this string alone, I (and
the computer) have no way of knowing what absolute moment in time you are
refering to -- is that 8:38:16 AM in *your* local time zone? in *my* local
timezone? in UTC? ... it's unclear.
which means when you ask some code to parse that date...
: DateTime dt = new DateTime();
:
: dt = Convert.ToDateTime(strStartDate);
...it has to make an implicit assumption about what timezone (ie:
coordinate system) that string representation is relative to -- it's
probably picking your local timezone.
Now, in this next bit, you ask your langauge to formate the date as a
string again -- but again you don't give it any instructions as to what
timzeon/coordinate system you want it to use -- so it is (mot likeley)
making the same assumption it did when it parsed your original date
(probably your local TZ, but it doesn't really matter -- whatever it
assumed before it's assuming again, and just outputing the same value with
the fields in differnet order)
You are then forcibly concatenanting a "Z" on the end....
: strStartDate = dt.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'");
: AddField(doc, "441", strStartDate); // posting to solr.
...by attaching that Z, you are telling Solr (and any other XML dateTIme
compliant parser) "this string representation of a date is in UTC" ... so
if, back at the top of this email, you really meant "8:38:16 AM in the UTC
time zone" and not ""8:38:16 AM in my local timezone" or "8:38:16 AM in
hoss'ss local timezone" then up to now, you've gotten really lucky and the
value solr knows about matches reallity, but only because you kludged it
that way, not because of well behaved parsing/ormatting logic...
: Query: http://localhost:8090/solrTest/select/?q=ID%3ASOLR1001&version=2.2&start=0&rows=10&indent=on&fl=441
...:
: <date name="441">2000-05-26T08:38:16Z</date>
Now at query time, when you use SolrJ you will get back a java Date object
representing the absolute moment in that that, when formatted relative the
UTC TZ coordinate system, is displayed as "08:38:16 AM UTC" on that date.
You are then using a simple "toString()" function in java to display the
response, since toString() is a very simplistic method for displaying Date
objects, and doesn't have any mechanism for you to tell it what format you
want the response string in, or what coordinate system (ie: timezone) you
want the string representation to be relateive to, it uses the platform
default timezone...
: ... docs=[SolrDocument[{441=Fri May 26 14:08:16 IST 2000}]]}}
...so aparenty, on the system where you are running this SolrJ query code,
the the java platform default TZ is "IST" which use UTC+05:30, so that
toString() representation does in fact refer to the same abstract moment
in time as the absolute UTC representation you see in the XML -- the fact
taht you are seeing "IST" in that output has nothing to do with Solr, or
SolrJ, it's 100% an artifact of your java setup and the fact that you
called "toString()" on a Date object (instead of using a Date formatter
that you explicitly told "format relative timezone XYZ"
>From the momenent you gave Solr the date string "2000-05-26T08:38:16Z" in
your indexing code, everything after that is 100% consistent, just
represented as strings in differnet timezone coordinate systems depending
on your plaform defaults. but whether or not you gave solr the correct
input string in the first place depends on what you ment by "08:38:16 AM"
-Hoss
RE: Environment Timezone considered When using SolrJ
Posted by "sowjanya@pointcross.com" <so...@pointcross.com>.
Chris,
Thanks for your reply. Please find the below process What we followed on Sample Test Solr:
Step1: Indexing data to Solr ( Schema looks like this)
<fieldType name="date" class="solr.TrieDateField" omitNorms="true" precisionStep="0" positionIncrementGap="0" />
<field name="441" CN="STARTDATE" type="date" indexed="true" stored="true" multiValued="false" />
Step2: Indexing code:
string strStartDate = "26/5/2000 8:38:16 AM";
DateTime dt = new DateTime();
dt = Convert.ToDateTime(strStartDate);
strStartDate = dt.ToString("yyyy'-'MM'-'dd'T'HH':'mm':'ss'.'fff'Z'");
AddField(doc, "441", strStartDate); // posting to solr.
Step3: Results from Browser
Query: http://localhost:8090/solrTest/select/?q=ID%3ASOLR1001&version=2.2&start=0&rows=10&indent=on&fl=441
Result:
<doc>
<date name="441">2000-05-26T08:38:16Z</date>
</doc>
Step4: Results from SolrJ
Code Snippet:
CommonsHttpSolrServer SOLRSERVER = new CommonsHttpSolrServer(strQueryURL);
qrResponse = SOLRSERVER.query(sqUserQuery, METHOD.POST);
Result:
{responseHeader={status=0,QTime=1,params={facet.mincount=1,facet.limit=-1,wt=javabin,rows=100000,version=2,fl=441,start=0,q=ID:SOLR1001}},response={numFound=1,start=0,docs=[SolrDocument[{441=Fri May 26 14:08:16 IST 2000}]]}}
We can see the Result in yellow marked coming (SolrJ) as in different format of time zone(original Date). I hope we have explained what is the issue that we are facing.
Thanks & Regards,
Sowjanya K
From: Chris Hostetter-3 [via Lucene] [mailto:ml-node+s472066n4083620h35@n3.nabble.com]
Sent: 09 August 2013 23:52
To: sowjanya@pointcross.com
Subject: Re: Environment Timezone considered When using SolrJ
: If you index a Java date object instead of the text format, it will be
: valid in the timezone at the client, and SolrJ will do timezone
: translation before sending to Solr. Solr will index/store the date in UTC.
this is missleading -- SolrJ doesn't do any sort of "timezone translation"
... a Java Date object represents an absolute fixed moment in time, there
is no timezone information in it. If your client code builds a Date
object from some intial string, then at the moment *you* parse hat string,
you are either explcitly or implicitly assuming some TZ information about
the string representation when converting it into an absolute moment in
time to build the Date object -- likewise, when executing a search and
fetching responses, SolrJ will give you back a Data object (representing
an absolute moment in time) and if *you* format that date as a string, you
are either explcitly or implicitly assuming some TZ information about the
string representation when converting it from a Date object (representing
that absolute moment in time)
Internally in SolrJ and Solr, Dates are always either Date objects
(representing absolute moments in times) or they are Strings or longs
representing that absolute moment in time relative to the UTC TZ
coordinate space.
Bottom line: if you are not seeing the correct "time" then some assumption
is probaly being violated somewhere in your client code as far as the
conversaion to/from string values -- if you gather up and look at:
1) your indexing code
2) a sample query URL done from the browser
3) the XML/json results from that sample query in your browser
4) your searching code excuting the same query in SolrJ
5) the output of your SolrJ search code
...the problem may become obvious, but if not is not, then sending all of
those things in a reply will help us point out where any discrepencies
might be.
-Hoss
_____
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/Environment-Timezone-considered-When-using-SolrJ-tp4083497p4083620.html
To unsubscribe from Environment Timezone considered When using SolrJ, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4083497&code=c293amFueWFAcG9pbnRjcm9zcy5jb218NDA4MzQ5N3w5MjQxMTAxNDE=>.
NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://lucene.472066.n3.nabble.com/Environment-Timezone-considered-When-using-SolrJ-tp4083497p4083947.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Environment Timezone considered When using SolrJ
Posted by Shawn Heisey <so...@elyograg.org>.
On 8/9/2013 12:20 PM, Chris Hostetter wrote:
>
> : If you index a Java date object instead of the text format, it will be
> : valid in the timezone at the client, and SolrJ will do timezone
> : translation before sending to Solr. Solr will index/store the date in UTC.
>
> this is missleading -- SolrJ doesn't do any sort of "timezone translation"
<snip>
Thanks for the clarification. Realities are often much more complex
than my experiences.
Shawn
Re: Environment Timezone considered When using SolrJ
Posted by Chris Hostetter <ho...@fucit.org>.
: If you index a Java date object instead of the text format, it will be
: valid in the timezone at the client, and SolrJ will do timezone
: translation before sending to Solr. Solr will index/store the date in UTC.
this is missleading -- SolrJ doesn't do any sort of "timezone translation"
... a Java Date object represents an absolute fixed moment in time, there
is no timezone information in it. If your client code builds a Date
object from some intial string, then at the moment *you* parse hat string,
you are either explcitly or implicitly assuming some TZ information about
the string representation when converting it into an absolute moment in
time to build the Date object -- likewise, when executing a search and
fetching responses, SolrJ will give you back a Data object (representing
an absolute moment in time) and if *you* format that date as a string, you
are either explcitly or implicitly assuming some TZ information about the
string representation when converting it from a Date object (representing
that absolute moment in time)
Internally in SolrJ and Solr, Dates are always either Date objects
(representing absolute moments in times) or they are Strings or longs
representing that absolute moment in time relative to the UTC TZ
coordinate space.
Bottom line: if you are not seeing the correct "time" then some assumption
is probaly being violated somewhere in your client code as far as the
conversaion to/from string values -- if you gather up and look at:
1) your indexing code
2) a sample query URL done from the browser
3) the XML/json results from that sample query in your browser
4) your searching code excuting the same query in SolrJ
5) the output of your SolrJ search code
...the problem may become obvious, but if not is not, then sending all of
those things in a reply will help us point out where any discrepencies
might be.
-Hoss
Re: Environment Timezone considered When using SolrJ
Posted by Shawn Heisey <so...@elyograg.org>.
On 8/9/2013 6:12 AM, sowjanya@pointcross.com wrote:
> When using SolrJ I've realized document dates are being modified according
> to the environment UTC timezone.
>
> I have indexed the large amount of data on date fileds of Solr (using Solr
> 3.3). While retrieving this date using the SolrJ into SolrDocumentList. The
> original date value is modified.
>
> Are we missing any setting at solrConfig. Any Help
If you are using a text date for indexing, which must be in a specific
format, then it will be interpreted as UTC, which Jack has already told you.
This is the format that Solr requires when the date for indexing is a
text string. This is also the format that you will see in raw responses
from Solr. The "Z" means Zulu, or UTC. There is no way to include a
timezone when using text strings:
2013-07-03T14:08:58Z
If you index a Java date object instead of the text format, it will be
valid in the timezone at the client, and SolrJ will do timezone
translation before sending to Solr. Solr will index/store the date in UTC.
Similarly, if you extract the date from the Solr response as a Date
object, you can see it in your local timezone. I just tried this with
some SolrJ code that looked like this:
SolrServer srv = new HttpSolrServer("server:port/solr/corename");
SolrQuery qry = new SolrQuery();
qry.setQuery("id:idvalue");
QueryResponse rsp = srv.query(qry);
Date dt = (Date) rsp.getResults().get(0).get("date");
System.out.println(dt);
Thanks,
Shawn