You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-users@xerces.apache.org by Ganesh Pagade <ga...@gmail.com> on 2010/11/29 14:45:35 UTC

Convert C++ string to XMLCh* for XMLUTF8Transcoder

Hi,

I have a C++ string which is in UTF-8, which I am passing to
XMLString::transcode(). However XMLString::transcode() is expecting SJIS
(Japanese). So it returns garbage characters in XMLCh*.

So I tired using XMLUTF8Transcoder. However XMLUTF8Transcoder's transcodeTo()
expects XMLCh*.

How do I convert C++ string into XMLCh* without using the
XMLString::transcode(),
so that I can pass it to transcodeTo()?

Or how do I make XMLString::transcode() expect UTF-8?

Any suggestions/pointers would be highly appreciated.

Thanks,
-Ganesh

Re: Convert C++ string to XMLCh* for XMLUTF8Transcoder

Posted by pagade <ga...@gmail.com>.
Thanks John. I copied the TranscodeToStr and TranscodeFromStr functions from
3.1 library to 2.8 library and was able to use them successfully. It worked
just fine.

However does that mean there was no way to do this in 2.8 and prior versions
of Xerces? IOW, how can I achieve this in Xerces 2.8?

Thanks,
-Ganesh

On 29 Nov 2010, at 13:45, Ganesh Pagade wrote:

> Hi,
> 
> I have a C++ string which is in UTF-8, which I am passing to
> XMLString::transcode(). However XMLString::transcode() is expecting SJIS
> (Japanese). So it returns garbage characters in XMLCh*.
> 
> So I tired using XMLUTF8Transcoder. However XMLUTF8Transcoder's
> transcodeTo()
> expects XMLCh*.
> 
> How do I convert C++ string into XMLCh* without using the
> XMLString::transcode(),
> so that I can pass it to transcodeTo()?
> 
> Or how do I make XMLString::transcode() expect UTF-8?
> 
> Any suggestions/pointers would be highly appreciated.
> 
> Thanks,
> -Ganesh




-- 
View this message in context: http://old.nabble.com/Convert-C%2B%2B-string-to-XMLCh*-for-XMLUTF8Transcoder-tp30330507p30341747.html
Sent from the Xerces - C - Users mailing list archive at Nabble.com.


Re: Convert C++ string to XMLCh* for XMLUTF8Transcoder

Posted by pagade <ga...@gmail.com>.

If you were using a more recent version of Xerces-C, you could use the
TranscodeToStr and TranscodeFromStr classes to both do the transcoding and
manage the allocated memory, and lifetime of the resultant string. The code
above is basically cut and paste from that class.

Thanks John
-- 
View this message in context: http://old.nabble.com/Convert-C%2B%2B-string-to-XMLCh*-for-XMLUTF8Transcoder-tp30330507p30341691.html
Sent from the Xerces - C - Users mailing list archive at Nabble.com.

Re: Convert C++ string to XMLCh* for XMLUTF8Transcoder

Posted by John Snelson <jo...@snelson.org.uk>.
Hi Ganesh,

If you want to use the Xerces-C abstractions to do the transcoding for you, you want to take a look at the XMLTransService class in xerces/util/TransService.cpp. The API is a little obtuse, so here's a code snippet for how to use it:

    char *data = "..."; // in UTF-8
    size_t length = strlen(data);

    XMLTransService::Codes failReason;
    const XMLSize_t blockSize = 2048;

    XMLTranscoder* trans = XMLPlatformUtils::fgTransService->makeNewTranscoderFor("utf-8", failReason, blockSize, fMemoryManager);
    Janitor<XMLTranscoder> janTrans(trans);

    XMLSize_t allocSize = length + 1;
    fString = (XMLCh*)fMemoryManager->allocate(allocSize * sizeof(XMLCh));

    XMLSize_t csSize = length;
    ArrayJanitor<unsigned char> charSizes((unsigned char*)fMemoryManager->allocate(csSize * sizeof(unsigned char)),
                                          fMemoryManager);

    XMLSize_t bytesRead = 0;
    XMLSize_t bytesDone = 0;

    while(true) {
        fCharsWritten += trans->transcodeFrom(in + bytesDone, length - bytesDone,
                                              fString + fCharsWritten, allocSize - fCharsWritten,
                                              bytesRead, charSizes.get());
        bytesDone += bytesRead;
        if(bytesDone == length) break;

        allocSize *= 2;
        XMLCh *newBuf = (XMLCh*)fMemoryManager->allocate(allocSize * sizeof(XMLCh));
        memcpy(newBuf, fString, fCharsWritten);
        fMemoryManager->deallocate(fString);
        fString = newBuf;

        if((allocSize - fCharsWritten) > csSize) {
            csSize = allocSize - fCharsWritten;
            charSizes.reset((unsigned char*)fMemoryManager->allocate(csSize * sizeof(unsigned char)),
                            fMemoryManager);
        }
    }

    // null terminate
    if(fCharsWritten == allocSize) {
        allocSize += 1;
        XMLCh *newBuf = (XMLCh*)fMemoryManager->allocate(allocSize * sizeof(XMLCh));
        memcpy(newBuf, fString, fCharsWritten);
        fMemoryManager->deallocate(fString);
        fString = newBuf;
    }
    fString[fCharsWritten] = 0;

If you were using a more recent version of Xerces-C, you could use the TranscodeToStr and TranscodeFromStr classes to both do the transcoding and manage the allocated memory, and lifetime of the resultant string. The code above is basically cut and paste from that class.

John

On 29 Nov 2010, at 13:45, Ganesh Pagade wrote:

> Hi,
> 
> I have a C++ string which is in UTF-8, which I am passing to
> XMLString::transcode(). However XMLString::transcode() is expecting SJIS
> (Japanese). So it returns garbage characters in XMLCh*.
> 
> So I tired using XMLUTF8Transcoder. However XMLUTF8Transcoder's transcodeTo()
> expects XMLCh*.
> 
> How do I convert C++ string into XMLCh* without using the
> XMLString::transcode(),
> so that I can pass it to transcodeTo()?
> 
> Or how do I make XMLString::transcode() expect UTF-8?
> 
> Any suggestions/pointers would be highly appreciated.
> 
> Thanks,
> -Ganesh


Re: Convert C++ string to XMLCh* for XMLUTF8Transcoder

Posted by Ganesh Pagade <ga...@gmail.com>.
2010/11/29 Stanisław Findeisen <st...@gemius.pl>

> Hi Ganesh
>
>
> On 11/29/2010 02:45 PM, Ganesh Pagade wrote:
>
>> I have a C++ string which is in UTF-8, which I am passing to
>> XMLString::transcode(). However XMLString::transcode() is expecting SJIS
>> (Japanese). So it returns garbage characters in XMLCh*.
>>
>
> Which of the XMLString::transcode methods are you calling?
>
> I think this one:
> http://xerces.apache.org/xerces-c/apiDocs-2/classXMLString.html#392a86c9a310dff366bedf2330050c3bhas a buggy description (shouldn't it read: "Transcodes a string *from*
> native code-page"??) and the expected input string format is platform
> dependent.


Thanks STF for the reply. Yes that's the one I am using.


>
>
>  So I tired using XMLUTF8Transcoder. However XMLUTF8Transcoder's
>> transcodeTo()
>> expects XMLCh*.
>>
>> How do I convert C++ string into XMLCh* without using the
>> XMLString::transcode(),
>> so that I can pass it to transcodeTo()?
>>
>> Or how do I make XMLString::transcode() expect UTF-8?
>>
>> Any suggestions/pointers would be highly appreciated.
>>
>
> People on this list have told me (Sat, 24 Jul 2010 10:48:38 -0700) that
> XMLCh is UTF-16 with platform byte order. So converting char* to XMLCh*
> equals converting char* to UTF-16 (with platform byte order). You should be
> able to do it with iconv (which is POSIX.1-2001 conforming) or, if you need
> any more sophisticated string operations, you might want to use ICU:
> http://userguide.icu-project.org/strings .
>

I believe you are suggesting me to use the Iconv transcoder (which is
present in include\xercesc\util\Transcoders\Iconv).

I tired following code:
        IconvLCPTranscoder *trans = new IconvLCPTranscoder();

        xmlTransCodeTemp = trans->transcode(pathName.c_str()); // pathName
is my C++ string, UTF-8 containing Japanese characters.

However this gives me xmlTransCodeTemp as null.

Am I doing something wrong here or I misunderstood?

BTW, I am using Xerces 2.8 and working on Windows x64 with Visual Studio
2005.

STF
>

Re: Convert C++ string to XMLCh* for XMLUTF8Transcoder

Posted by Stanisław Findeisen <st...@gemius.pl>.
Hi Ganesh

On 11/29/2010 02:45 PM, Ganesh Pagade wrote:
> I have a C++ string which is in UTF-8, which I am passing to
> XMLString::transcode(). However XMLString::transcode() is expecting SJIS
> (Japanese). So it returns garbage characters in XMLCh*.

Which of the XMLString::transcode methods are you calling?

I think this one: http://xerces.apache.org/xerces-c/apiDocs-2/classXMLString.html#392a86c9a310dff366bedf2330050c3b has a buggy description (shouldn't it read: "Transcodes a string *from* native code-page"??) and the expected input string format is platform dependent.

> So I tired using XMLUTF8Transcoder. However XMLUTF8Transcoder's transcodeTo()
> expects XMLCh*.
>
> How do I convert C++ string into XMLCh* without using the
> XMLString::transcode(),
> so that I can pass it to transcodeTo()?
>
> Or how do I make XMLString::transcode() expect UTF-8?
>
> Any suggestions/pointers would be highly appreciated.

People on this list have told me (Sat, 24 Jul 2010 10:48:38 -0700) that XMLCh is UTF-16 with platform byte order. So converting char* to XMLCh* equals converting char* to UTF-16 (with platform byte order). You should be able to do it with iconv (which is POSIX.1-2001 conforming) or, if you need any more sophisticated string operations, you might want to use ICU: http://userguide.icu-project.org/strings .

STF