You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by LiuYan 刘研 <lo...@21cn.com> on 2010/11/17 15:06:19 UTC

Prevent mod_dav_svn REPORT log failure if files under $REPOS/db/revprops/ are not in UTF-8 encoding

Stefan Sperling <stsp <at> elego.de> writes:

> 
> On Wed, Nov 17, 2010 at 11:48:07AM +0000, LiuYan 刘研 wrote:
> > I converted CVSNT respositoy to SVN repository some days ago, and config 
> > mod_dav_svn in Apache http.conf.
> > 
> > 
> > When I try 'svn log http://192.168.105.4:8000/svn/cmcc', it shown some logs 
but 
> > failed/terminated at some revisions with a message like:
> > svn: REPORT of '/svn/cmcc/!svn/bc/1': 200 OK (http://192.168.105.4:8000)
> > 
> > 
> > I try to the svn:// protocol to get log of the failed revisions, it can get 
the 
> > log, but the log is not human-readable like the following:
> > svn log svn://192.168.105.4 -r 1
> > ------------------------------------------------------------------------
> > r1 | (no author) | 2005-11-10 17:28:50 +0800 (星期四, 10 十一月 2005) | 2 
lines
> > 
> > Standard project directories initialized by cvs2svn.?\211?\201 cvs2svn
> > ?\179?\245?\202?\188?\187?\175?\181?\196?\177?\234?\215?\188?\207?\238?\196?
> > \191?\206?\196?\188?\254?\188?\208
> > ------------------------------------------------------------------------
> > 
> > 
> > It seems a charset encoding problem of 'revprops' files because I 
> > set 'ctx.initial_project_commit_message' cvs2svn options config file to a 
> > string contains Chinese characters, and cvs2svn options config file itself 
is 
> > encode in GBK, and it seems cvs2svn does not converted the charset encoding 
of 
> > initial_project_commit_message to UTF-8.
> > 
> > 
> > Does this encoding issue can caused mod_dav_svn REPORT failed?
> 
> The revprops should be encoded in UTF-8.
> 
> You could write each revprop to a temporary file (get it with 'svn
> propget --revprop'), and convert that file to UTF-8 encoding.
> Then set the revprop again e.g. by using the svnadmin setrevprop command.
> 
> Stefan
> 
 
Thank you Stefan! And sorry for my poor English.

I've already converted these revprops files by operating them directly (I know 
it's a dangerous way) via a little script, now 'svn log 
http://svnserver/svnrepos/svnrepo' works fine.

My orginal idea is:
If this "svn: REPORT of " error is returned by mod_dav_svn, can it act friendly 
as the same as svnserve ?
I mean, if the revprops files are not in UTF-8 encoding, don't return REPORT 
Failed to client side, but still return the malformed log instead, so that the 
end user can notice IT'S WRONG, and can easily figure out WHY IT'S WRONG.

Liu Yan

Re: Prevent mod_dav_svn REPORT log failure if files under?$REPOS/db/revprops/ are not in UTF-8 encoding

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
DON'T DO IT THIS WAY.  Follow Stefan's advice upthread.

Re: Prevent mod_dav_svn REPORT log failure if files under $REPOS/db/revprops/ are not in UTF-8 encoding

Posted by LiuYan 刘研 <lo...@21cn.com>.
Daniel Shahaf <d.s <at> daniel.shahaf.name> writes:

> 
> LiuYan 刘研 wrote on Thu, Nov 18, 2010 at 02:53:37 +0000:
> > Daniel Shahaf <d.s <at> daniel.shahaf.name> writes:
> > 
> > > 
> > > Stefan Sperling wrote on Wed, Nov 17, 2010 at 18:13:44 +0100:
> > > > On Wed, Nov 17, 2010 at 03:06:19PM +0000, LiuYan 刘研 wrote:
> > > > > I mean, if the revprops files are not in UTF-8 encoding, don't return 
> > REPORT 
> > > 
> > > Small correction: it's meaningless to talk about the encoding of the
> > > revprop files; it's only meaningful to talk about the encoding of the
> > > value of a given property.
> > > 
> > > (At the revprop files level, the values are binary, and the rest of the
> > > data in those files is always ASCII.)
> > > 
> > > 
> > 
> > You're right Daniel, but in such situation, these revprop files can be 
treated 
> > as readable text files:
> 
> This is simply not true: if you apply 'iconv -f latin1 -t utf-8' to
> a revprop file, you will CORRUPT that revprop file.
> 
> 

You're right Daniel, simply apply an 'iconv' operation to a revprop file will 
CORRUPT it, there's data length value should be changed too.
So I wrote a small script to do the conversion as I mentioned at 
http://article.gmane.org/gmane.comp.version-control.subversion.user/101383

The script do the following operations:
1. find out the affected revprop files
2. change the svn:log value length from "V 85" to "V 98"
3. change/convert the svn:log value to a UTF-8 encoded string

Here's the small script, and be aware of this script file is in GBK encoding.

Administrator@CMTEL-SVR-HR-DB /cygdrive/d/SVNRepositories/repos/cmcc/db
$ cat fix-cvs2svn.sh
IFS=$'\n'
grep -i -r -n "Standard project directories initialized by cvs2svn" revprops/* 
| cut -d ":" -f 1 > affected_files.txt
#grep result sample
#0/1:8:Standard project directories initialized by cvs2svn.由 cvs2svn
#0/133:8:Standard project directories initialized by cvs2svn.由 cvs2svn

for file in `cat affected_files.txt`
do
        echo $file

        #${file:9}: strip out 'revprops/'
        dest_file="fix/${file:9}"

        cp --force --preserve --verbose $file "fix-backup/${file:9}"

        gawk 'FNR==7 {print "V 98"} FNR==8{print "Standard project directories 
initialized by cvs2svn."} FNR==9{print "由 cvs2svn 初始化的标准项目文件夹"} FNR<7 
|| FNR==10 {print $0}' $file | iconv --from GBK --to UTF-8 > "$dest_file"
done


Re: Prevent mod_dav_svn REPORT log failure if files under $REPOS/db/revprops/ are not in UTF-8 encoding

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
LiuYan 刘研 wrote on Thu, Nov 18, 2010 at 02:53:37 +0000:
> Daniel Shahaf <d.s <at> daniel.shahaf.name> writes:
> 
> > 
> > Stefan Sperling wrote on Wed, Nov 17, 2010 at 18:13:44 +0100:
> > > On Wed, Nov 17, 2010 at 03:06:19PM +0000, LiuYan 刘研 wrote:
> > > > I mean, if the revprops files are not in UTF-8 encoding, don't return 
> REPORT 
> > 
> > Small correction: it's meaningless to talk about the encoding of the
> > revprop files; it's only meaningful to talk about the encoding of the
> > value of a given property.
> > 
> > (At the revprop files level, the values are binary, and the rest of the
> > data in those files is always ASCII.)
> > 
> > 
> 
> You're right Daniel, but in such situation, these revprop files can be treated 
> as readable text files:

This is simply not true: if you apply 'iconv -f latin1 -t utf-8' to
a revprop file, you will CORRUPT that revprop file.

Re: Prevent mod_dav_svn REPORT log failure if files under $REPOS/db/revprops/ are not in UTF-8 encoding

Posted by LiuYan 刘研 <lo...@21cn.com>.
Daniel Shahaf <d.s <at> daniel.shahaf.name> writes:

> 
> Stefan Sperling wrote on Wed, Nov 17, 2010 at 18:13:44 +0100:
> > On Wed, Nov 17, 2010 at 03:06:19PM +0000, LiuYan 刘研 wrote:
> > > I mean, if the revprops files are not in UTF-8 encoding, don't return 
REPORT 
> 
> Small correction: it's meaningless to talk about the encoding of the
> revprop files; it's only meaningful to talk about the encoding of the
> value of a given property.
> 
> (At the revprop files level, the values are binary, and the rest of the
> data in those files is always ASCII.)
> 
> 

You're right Daniel, but in such situation, these revprop files can be treated 
as readable text files:
Administrator@CMTEL-SVR-HR-DB /cygdrive/d/SVNRepositories/repos/cmcc/db
$ file revprops/0/1 fix-backup/0/1
revprops/0/1:   UTF-8 Unicode text
fix-backup/0/1: ISO-8859 text

Thanks for the correction Daniel! (^_^)

And, I've post this to buglist here:
http://subversion.tigris.org/issues/show_bug.cgi?id=3753

Wish it can be accepted and fixed.

Liu Yan

Re: Prevent mod_dav_svn REPORT log failure if files under $REPOS/db/revprops/ are not in UTF-8 encoding

Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Stefan Sperling wrote on Wed, Nov 17, 2010 at 18:13:44 +0100:
> On Wed, Nov 17, 2010 at 03:06:19PM +0000, LiuYan 刘研 wrote:
> > I mean, if the revprops files are not in UTF-8 encoding, don't return REPORT 

Small correction: it's meaningless to talk about the encoding of the
revprop files; it's only meaningful to talk about the encoding of the
value of a given property.

(At the revprop files level, the values are binary, and the rest of the
data in those files is always ASCII.)

Re: Prevent mod_dav_svn REPORT log failure if files under $REPOS/db/revprops/ are not in UTF-8 encoding

Posted by Stefan Sperling <st...@elego.de>.
On Wed, Nov 17, 2010 at 03:06:19PM +0000, LiuYan 刘研 wrote:
> My orginal idea is:
> If this "svn: REPORT of " error is returned by mod_dav_svn, can it act friendly 
> as the same as svnserve ?
> I mean, if the revprops files are not in UTF-8 encoding, don't return REPORT 
> Failed to client side, but still return the malformed log instead, so that the 
> end user can notice IT'S WRONG, and can easily figure out WHY IT'S WRONG.
> 

I agree that mod_dav_svn's behaviour should match the behaviour of
svnserve in this case. I'm not sure why that isn't the case.
It's probably due to some slight difference in implementation.
If you want to file an issue about this in our issue tracker, feel free
to do so: http://subversion.tigris.org/issue-tracker.html

Thanks,
Stefan