Problem with DataFormat and XML Serialise/Deserialise

[Adrian Wiesmann <aw...@somap.org>]

Hello

I guess I found a problem within the XML Serialisation/Deserialisation
code when working with Dates. But before posting a patch or bug in JIRA I
thought I'd make sure that I really found a problem and am not hunting
ghosts (the wrong ones :) ).

The scenario is the following:

Serialising a list of objects with the XMLEncoder:

> xmlEncoder.encode(myId, dataObjectList);

results in the date fields to be formatted like this:

"Tue Mar 13 20:56:11 CET 2007"

which basically represents this format:

"EEE MMM dd HH:mm:ss zzz yyyy"

That works like a charm. The resulting XML is well formatted.

The problem is then, when you try to read back such a deserialised list
with the help of the XMLDecoder into a DataContext:

> XMLDecoder.decodeList(buffer, dataContext);

Sometimes this resulted in our application in the following error:

> Unparseable date: "Tue Mar 13 20:56:11 CET 2007"

and first we thought this is an error with Windows because the error only
happened on that system. But after doing some testing I think we have
a locale problem:

The XMLDecoder decodes Dates like this:

> formatter = new SimpleDateFormat("EEE MMM dd HH:mm:ss zzz yyyy");
> formatter.parse("Tue Mar 13 20:56:11 CET 2007");

The problem is, that the SimpleDateFormat seems to use the locale of the
respective system, although we define the mask it should use in its
constructor! I tried to use non lenient functionality:

> formatter.setLenient(false);

but that didn't work as well. Which basically means the XMLDecoder and
XMLEncoder are not supporting different locales.

Can somebody confirm this behaviour? (And if yes, shouldn't we change the
format of the date/time as well as the formatting within the Decoder and
Encoder?)

Regards,
Adrian


P.S: This is some snippet to test the problem:

String strDate = "Tue Mar 13 20:56:11 CET 2007";
String strFormat = "EEE MMM dd HH:mm:ss zzz yyyy";
DateFormat formatter = new SimpleDateFormat(strFormat, Locale.CHINA);     
        

try {
	  System.out.println(formatter.parse(strDate));
} catch (ParseException e) {
  e.printStackTrace();
}

formatter = new SimpleDateFormat(strFormat, Locale.CHINA);        
formatter.setLenient(false);
try {
	  System.out.println(formatter.parse(strDate));
} catch (ParseException e) {
  e.printStackTrace();
}

formatter = new SimpleDateFormat(strFormat, Locale.US);         
try {
  System.out.println(formatter.parse(strDate));
} catch (ParseException e) {
  e.printStackTrace();
}

Re: Problem with DataFormat and XML Serialise/Deserialise

[Adrian Wiesmann <aw...@somap.org>]

> Do you have a few minutes to help turning this commit into patches  
> against 1.1 and 1.2:
> 
> http://svn.apache.org/viewvc?view=rev&revision=541595

Hmm, I will have to check out the old code then. I am only having version
3 configured for compilation. I plan to look into this on this friday.


> At the same time I've been looking to implement a similar algorithm  
> (outside Cayenne but based on Cayenne schema) for one of my projects,  
> and it is possible if you make some assumptions about the nature of  
> the schema changes (e.g. only support add/drop column, create/drop  
> table ops) and do not support porting the data (e.g. no "rename  
> column").

I guess I will have to delay our update mechanism for later but I am
currently thinking about something like this:

- Export table into XML via Cayenne
- Drop all constraints and foreign key relations
- Drop Table
- Recreate Table with new metadata
- Import table from XML via Cayenne
- Recreate all constraints, indexes and foreign key relations

I chose this strategy because of these thoughts:

- Simple changes to the data model can be managed by above strategy
without any further work needed. 
- More complex changes could be scripted (during the import we call a
class or some beanshell script or something) or the data can be changed
with some XSLT script before the import.
- I did not look to much into the constraints and foreign key relations
but creating a table is supported by the Cayenne DBAdapter which is a plus
when trying to support as many DBs as possible. So there would be no need
to do some difficult update/alter table algorithm.

What do you think?

Adrian

Re: Problem with DataFormat and XML Serialise/Deserialise

[Andrus Adamchik <an...@objectstyle.org>]

On May 25, 2007, at 12:50 PM, Adrian Wiesmann wrote:

> Hello Andrus
>
>> I guess I am going to add some failover strategy that
>> would try a previous formatter in case of parse exceptions..
>
> Sounds great. I had 2 thoughts about that:
>
> - Probably this should be "backported" to the stable version as  
> well. IIRC
> decoding/encoding was added somewhen after version 1.1?

+1

Do you have a few minutes to help turning this commit into patches  
against 1.1 and 1.2:

http://svn.apache.org/viewvc?view=rev&revision=541595


> Another sidenote: Have you (or anybody else) ever thought about  
> using the
> decoder/encoder plus the XML with the database structure (map.xml) for
> automatic DB update purposes (magically alter table etc)?

For data updates - sure this should work. For schema updates - it may  
work for certain cases, but generally this is a very complicated  
task. There were extensive discussions in the past about altering  
schema from CayenneModeler, and the consensus was that a generic  
implementation is hard and beyond Cayenne scope.

At the same time I've been looking to implement a similar algorithm  
(outside Cayenne but based on Cayenne schema) for one of my projects,  
and it is possible if you make some assumptions about the nature of  
the schema changes (e.g. only support add/drop column, create/drop  
table ops) and do not support porting the data (e.g. no "rename  
column").

Andrus

Re: Problem with DataFormat and XML Serialise/Deserialise

[Adrian Wiesmann <aw...@somap.org>]

Hello Andrus

> I guess I am going to add some failover strategy that  
> would try a previous formatter in case of parse exceptions..

Sounds great. I had 2 thoughts about that:

- Probably this should be "backported" to the stable version as well. IIRC
decoding/encoding was added somewhen after version 1.1?
- I was not that sure about how I should solve backwards compatibility
and since I was in the position to not have to think about old versions...
Thanks for adding it, looks good to me.


> On a side note - maybe we can improve encoder/decoder performance by  
> introducing some "context" object that internally stores reusable  
> thread-unsafe objects like formatters, instead of creating them for  
> every field?

Hmm, should work. But I am not sure if I know much about such concepts. I
am using the decoding/encoding for quite some special cases (distributable
snapshot files). Is there a more common use for the decoder/encoder within
Cayenne?

Another sidenote: Have you (or anybody else) ever thought about using the
decoder/encoder plus the XML with the database structure (map.xml) for
automatic DB update purposes (magically alter table etc)? 

Regards,
Adrian

Re: Problem with DataFormat and XML Serialise/Deserialise

[Andrus Adamchik <an...@objectstyle.org>]

Hi Adrian,

I think you correctly identified the problem (encoding date via a  
'toString()' method is definitely wrong), and looks like the patch  
attached to CAY-792 should fix it by not including any textual output  
in the encoded date. There's one problem though - our existing users  
may have stored XML that was encoded using the previous default  
formatter. I guess I am going to add some failover strategy that  
would try a previous formatter in case of parse exceptions..

On a side note - maybe we can improve encoder/decoder performance by  
introducing some "context" object that internally stores reusable  
thread-unsafe objects like formatters, instead of creating them for  
every field?

Thanks
Andrus


On May 22, 2007, at 8:58 PM, Adrian Wiesmann wrote:

> Hello
>
> I guess I found a problem within the XML Serialisation/Deserialisation
> code when working with Dates. But before posting a patch or bug in  
> JIRA I
> thought I'd make sure that I really found a problem and am not hunting
> ghosts (the wrong ones :) ).
>
> The scenario is the following:
>
> Serialising a list of objects with the XMLEncoder:
>
>> xmlEncoder.encode(myId, dataObjectList);
>
> results in the date fields to be formatted like this:
>
> "Tue Mar 13 20:56:11 CET 2007"
>
> which basically represents this format:
>
> "EEE MMM dd HH:mm:ss zzz yyyy"
>
> That works like a charm. The resulting XML is well formatted.
>
> The problem is then, when you try to read back such a deserialised  
> list
> with the help of the XMLDecoder into a DataContext:
>
>> XMLDecoder.decodeList(buffer, dataContext);
>
> Sometimes this resulted in our application in the following error:
>
>> Unparseable date: "Tue Mar 13 20:56:11 CET 2007"
>
> and first we thought this is an error with Windows because the  
> error only
> happened on that system. But after doing some testing I think we have
> a locale problem:
>
> The XMLDecoder decodes Dates like this:
>
>> formatter = new SimpleDateFormat("EEE MMM dd HH:mm:ss zzz yyyy");
>> formatter.parse("Tue Mar 13 20:56:11 CET 2007");
>
> The problem is, that the SimpleDateFormat seems to use the locale  
> of the
> respective system, although we define the mask it should use in its
> constructor! I tried to use non lenient functionality:
>
>> formatter.setLenient(false);
>
> but that didn't work as well. Which basically means the XMLDecoder and
> XMLEncoder are not supporting different locales.
>
> Can somebody confirm this behaviour? (And if yes, shouldn't we  
> change the
> format of the date/time as well as the formatting within the  
> Decoder and
> Encoder?)
>
> Regards,
> Adrian
>
>
> P.S: This is some snippet to test the problem:
>
> String strDate = "Tue Mar 13 20:56:11 CET 2007";
> String strFormat = "EEE MMM dd HH:mm:ss zzz yyyy";
> DateFormat formatter = new SimpleDateFormat(strFormat, Locale.CHINA);
>
>
> try {
> 	  System.out.println(formatter.parse(strDate));
> } catch (ParseException e) {
>   e.printStackTrace();
> }
>
> formatter = new SimpleDateFormat(strFormat, Locale.CHINA);
> formatter.setLenient(false);
> try {
> 	  System.out.println(formatter.parse(strDate));
> } catch (ParseException e) {
>   e.printStackTrace();
> }
>
> formatter = new SimpleDateFormat(strFormat, Locale.US);
> try {
>   System.out.println(formatter.parse(strDate));
> } catch (ParseException e) {
>   e.printStackTrace();
> }
>