You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@xerces.apache.org by "Daniel Turcanu (JIRA)" <xe...@xml.apache.org> on 2011/02/18 16:08:38 UTC
[jira] Created: (XERCESC-1959) serializeGrammars does not work
between 32 and 64 bit systems
serializeGrammars does not work between 32 and 64 bit systems
-------------------------------------------------------------
Key: XERCESC-1959
URL: https://issues.apache.org/jira/browse/XERCESC-1959
Project: Xerces-C++
Issue Type: Bug
Components: Validating Parser (XML Schema)
Affects Versions: 3.1.1
Environment: Win Vista 32 bit + VS 2010 express, Xerces-C 3.1.1 binary
Ubuntu 10.10 64 bit, Xerces-C 3.1 installed
Reporter: Daniel Turcanu
Serialization of schema grammar does not work between Windows 32 and Linux 64 bit. Serializing on one machine (with serializeGrammars) and deserializing on the other (with deserializeGrammars) fails ungracefully.
Serializing and deserializing on the same machine works fine.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
Re: [jira] Commented: (XERCESC-1959) serializeGrammars does not
workbetween 32 and 64 bit systems *
Posted by Stephen Williams <sc...@gmail.com>.
Serialized grammars are a good idea. Standardized and architecture neutral
serialized grammars are even better. I tried to get it in W3C EXI (and I
will, at some point, as an extension). It is even more important for EXI
than in general XML cases.
It really is not hard to make data formats like these portable and
architecturally neutral. Byte order can be handled in just a few lines of
code, for instance.
Stephen
On Wed, Feb 23, 2011 at 5:17 PM, David Bertoni (JIRA) <
xerces-c-dev@xml.apache.org> wrote:
> [image: Boxbe] <https://www.boxbe.com/overview> Junk Score: 1 out of 10
> (below your Auto Allow threshold <https://www.boxbe.com/mail-screening>) |
> Approve sender <https://www.boxbe.com/anno?tc=7045170930_887573393> | Block
> sender <https://www.boxbe.com/anno?tc=7045170930_887573393&disp=b> | Block
> domain <https://www.boxbe.com/anno?tc=7045170930_887573393&disp=b&dom>
>
>
> [
> https://issues.apache.org/jira/browse/XERCESC-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998651#comment-12998651]
>
> David Bertoni commented on XERCESC-1959:
> ----------------------------------------
>
> The serialization of grammars was never designed to be portable across
> machines. There are issues with 32-bit and 64-bit builds and big-endian and
> little-endian machines. For example:
>
> void XSerializeEngine::writeString(const XMLCh* const toWrite
> , const XMLSize_t bufferLen
> , bool toWriteBufLen)
> {
> if (toWrite)
> {
> if (toWriteBufLen)
> *this<<(unsigned long)bufferLen;
>
> XMLSize_t strLen = XMLString::stringLen(toWrite);
> *this<<(unsigned long)strLen;
>
> write(toWrite, strLen);
> }
> else
> {
> *this<<noDataFollowed;
> }
>
> }
>
> Note that XMLSize_t is being cast to unsigned long, which will be a 32-bit
> value using a 32-bit binary, and a 64-bit value using a 64-bit binary.
>
> The only way to make this work properly would be to store integrals as
> 64-bit values in both 32-bit and 64-bit binaries. And that still would not
> deal with any endianness issues.
>
> Is there some reason you can't keep different serialized grammars for
> 32-bit and 64-bit builds?
>
> > serializeGrammars does not work between 32 and 64 bit systems
> > -------------------------------------------------------------
> >
> > Key: XERCESC-1959
> > URL: https://issues.apache.org/jira/browse/XERCESC-1959
> > Project: Xerces-C++
> > Issue Type: Bug
> > Components: Validating Parser (XML Schema)
> > Affects Versions: 3.1.1
> > Environment: Win Vista 32 bit + VS 2010 express, Xerces-C 3.1.1
> binary
> > Ubuntu 10.10 64 bit, Xerces-C 3.1 installed
> > Reporter: Daniel Turcanu
> >
> > Serialization of schema grammar does not work between Windows 32 and
> Linux 64 bit. Serializing on one machine (with serializeGrammars) and
> deserializing on the other (with deserializeGrammars) fails ungracefully.
> > Serializing and deserializing on the same machine works fine.
>
> --
> This message is automatically generated by JIRA.
> -
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
> For additional commands, e-mail: c-dev-help@xerces.apache.org
>
>
>
--
--
Stephen D. Williams sdw@lig.net scienteer@gmail.com LinkedIn:
http://sdw.st/in
V:650-450-UNIX (8649) V:866.SDW.UNIX V:703.371.9362 F:703.995.0407
AIM:sdw Skype:StephenDWilliams Resume: http://sdw.st/gres
Personal: sdw.st facebook.com/sdwlig twitter.com/scienteer
[jira] Commented: (XERCESC-1959) serializeGrammars does not work
between 32 and 64 bit systems
Posted by "David Bertoni (JIRA)" <xe...@xml.apache.org>.
[ https://issues.apache.org/jira/browse/XERCESC-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998651#comment-12998651 ]
David Bertoni commented on XERCESC-1959:
----------------------------------------
The serialization of grammars was never designed to be portable across machines. There are issues with 32-bit and 64-bit builds and big-endian and little-endian machines. For example:
void XSerializeEngine::writeString(const XMLCh* const toWrite
, const XMLSize_t bufferLen
, bool toWriteBufLen)
{
if (toWrite)
{
if (toWriteBufLen)
*this<<(unsigned long)bufferLen;
XMLSize_t strLen = XMLString::stringLen(toWrite);
*this<<(unsigned long)strLen;
write(toWrite, strLen);
}
else
{
*this<<noDataFollowed;
}
}
Note that XMLSize_t is being cast to unsigned long, which will be a 32-bit value using a 32-bit binary, and a 64-bit value using a 64-bit binary.
The only way to make this work properly would be to store integrals as 64-bit values in both 32-bit and 64-bit binaries. And that still would not deal with any endianness issues.
Is there some reason you can't keep different serialized grammars for 32-bit and 64-bit builds?
> serializeGrammars does not work between 32 and 64 bit systems
> -------------------------------------------------------------
>
> Key: XERCESC-1959
> URL: https://issues.apache.org/jira/browse/XERCESC-1959
> Project: Xerces-C++
> Issue Type: Bug
> Components: Validating Parser (XML Schema)
> Affects Versions: 3.1.1
> Environment: Win Vista 32 bit + VS 2010 express, Xerces-C 3.1.1 binary
> Ubuntu 10.10 64 bit, Xerces-C 3.1 installed
> Reporter: Daniel Turcanu
>
> Serialization of schema grammar does not work between Windows 32 and Linux 64 bit. Serializing on one machine (with serializeGrammars) and deserializing on the other (with deserializeGrammars) fails ungracefully.
> Serializing and deserializing on the same machine works fine.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
[jira] Commented: (XERCESC-1959) serializeGrammars does not work
between 32 and 64 bit systems
Posted by "Boris Kolpackov (JIRA)" <xe...@xml.apache.org>.
[ https://issues.apache.org/jira/browse/XERCESC-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002969#comment-13002969 ]
Boris Kolpackov commented on XERCESC-1959:
------------------------------------------
Daniel, I agree the current implementation is not ideal so if you would like to come up with a patch, then that would be very welcome.
David, regarding backwards compatibility, there is a format version number stored with the binary representation. Now, the problem is that the version itself is stored in a non-portable manner. Luckily it is always 32-bit (unsigned int) so we can write 0xFFFFFFFF there for the new format which will make sure that any previous version (regardless of the endianness) will not compare equal.
> serializeGrammars does not work between 32 and 64 bit systems
> -------------------------------------------------------------
>
> Key: XERCESC-1959
> URL: https://issues.apache.org/jira/browse/XERCESC-1959
> Project: Xerces-C++
> Issue Type: Bug
> Components: Validating Parser (XML Schema)
> Affects Versions: 3.1.1
> Environment: Win Vista 32 bit + VS 2010 express, Xerces-C 3.1.1 binary
> Ubuntu 10.10 64 bit, Xerces-C 3.1 installed
> Reporter: Daniel Turcanu
>
> Serialization of schema grammar does not work between Windows 32 and Linux 64 bit. Serializing on one machine (with serializeGrammars) and deserializing on the other (with deserializeGrammars) fails ungracefully.
> Serializing and deserializing on the same machine works fine.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
[jira] Commented: (XERCESC-1959) serializeGrammars does not work
between 32 and 64 bit systems
Posted by "Daniel Turcanu (JIRA)" <xe...@xml.apache.org>.
[ https://issues.apache.org/jira/browse/XERCESC-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998783#comment-12998783 ]
Daniel Turcanu commented on XERCESC-1959:
-----------------------------------------
We work on Zorba XQuery engine (http://www.zorba-xquery.com), and we implemented a way to serialize the precompiled xquery code, so that the user doesn't have to compile his code every time. For serialization of schemas we use Xerces's internal serialization.
A user found this problem between 32 and 64 bit. Zorba can deal with this, because we serialize numbers as strings, so there is no problem with 32/64 or the endianess. If Xerces cannot solve this, then we will also have to limit machine interoperability.
> serializeGrammars does not work between 32 and 64 bit systems
> -------------------------------------------------------------
>
> Key: XERCESC-1959
> URL: https://issues.apache.org/jira/browse/XERCESC-1959
> Project: Xerces-C++
> Issue Type: Bug
> Components: Validating Parser (XML Schema)
> Affects Versions: 3.1.1
> Environment: Win Vista 32 bit + VS 2010 express, Xerces-C 3.1.1 binary
> Ubuntu 10.10 64 bit, Xerces-C 3.1 installed
> Reporter: Daniel Turcanu
>
> Serialization of schema grammar does not work between Windows 32 and Linux 64 bit. Serializing on one machine (with serializeGrammars) and deserializing on the other (with deserializeGrammars) fails ungracefully.
> Serializing and deserializing on the same machine works fine.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
[jira] [Commented] (XERCESC-1959) serializeGrammars does not work
between 32 and 64 bit systems
Posted by "Lee Doron (JIRA)" <xe...@xml.apache.org>.
[ https://issues.apache.org/jira/browse/XERCESC-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011556#comment-13011556 ]
Lee Doron commented on XERCESC-1959:
------------------------------------
The new format could include info on the writer's endianness and 32- vs. 64-bitness. The reader would check whether they match its native parameters -- if so, then it can read with no penalty in speed (and no unnecessary memory usage, if both are 32-bit machines). If not, then it would have to translate, for a slight speed penalty.
> serializeGrammars does not work between 32 and 64 bit systems
> -------------------------------------------------------------
>
> Key: XERCESC-1959
> URL: https://issues.apache.org/jira/browse/XERCESC-1959
> Project: Xerces-C++
> Issue Type: Bug
> Components: Validating Parser (XML Schema)
> Affects Versions: 3.1.1
> Environment: Win Vista 32 bit + VS 2010 express, Xerces-C 3.1.1 binary
> Ubuntu 10.10 64 bit, Xerces-C 3.1 installed
> Reporter: Daniel Turcanu
>
> Serialization of schema grammar does not work between Windows 32 and Linux 64 bit. Serializing on one machine (with serializeGrammars) and deserializing on the other (with deserializeGrammars) fails ungracefully.
> Serializing and deserializing on the same machine works fine.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org
[jira] Commented: (XERCESC-1959) serializeGrammars does not work
between 32 and 64 bit systems
Posted by "David Bertoni (JIRA)" <xe...@xml.apache.org>.
[ https://issues.apache.org/jira/browse/XERCESC-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999009#comment-12999009 ]
David Bertoni commented on XERCESC-1959:
----------------------------------------
Xerces-C serializes numbers into the stream that indicates things like string length, so it will still need to deal with endianness.
It's possible for this to be changed in a future, but we would then break compatibility with grammars that were serialized by previous releases,
> serializeGrammars does not work between 32 and 64 bit systems
> -------------------------------------------------------------
>
> Key: XERCESC-1959
> URL: https://issues.apache.org/jira/browse/XERCESC-1959
> Project: Xerces-C++
> Issue Type: Bug
> Components: Validating Parser (XML Schema)
> Affects Versions: 3.1.1
> Environment: Win Vista 32 bit + VS 2010 express, Xerces-C 3.1.1 binary
> Ubuntu 10.10 64 bit, Xerces-C 3.1 installed
> Reporter: Daniel Turcanu
>
> Serialization of schema grammar does not work between Windows 32 and Linux 64 bit. Serializing on one machine (with serializeGrammars) and deserializing on the other (with deserializeGrammars) fails ungracefully.
> Serializing and deserializing on the same machine works fine.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: c-dev-unsubscribe@xerces.apache.org
For additional commands, e-mail: c-dev-help@xerces.apache.org