You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@forrest.apache.org by Sjur Moshagen <sj...@mac.com> on 2006/09/25 14:19:45 UTC

Dispatcher i18n update + three more issues

Hello all,

AS seen in JIRA, I have found a set of errors in Dispatcher and  
Forrest regarding i18n, and submitted patches for them. With those  
patches, Dispatcher should be ready to go re i18n - almost:-) There  
are still three more issues for which I can't submit a patch (I think):

1) one file is wrongly named in whiteboard/plugins/ 
o.a.f.plugin.internal.dispatcher/messages/

ContractsMessages_us.xml

should most likely be:

ContractsMessages_en.xml

2) There should probably be a default/fallback ContractsMessages.xml  
file, which should really be the file above (ie the English one, if  
one wants to have English as the fallback language, which I expect  
one does). Not having one will at least cause some noise in the log  
files, and might also generate errors

3) when doing a forrest seed, the translation files in whiteboard/ 
plugins/o.a.f.plugin.internal.dispatcher/messages/ should be included  
in the src/documentation/translations/ directory, with a proper note  
on them being useful only with the dispatcher. Without translation  
files in this location, the standard "messages" (Last published,  
Search, etc.) will not be translated.

Best regards,
Sjur


Re: Dispatcher i18n update + three more issues

Posted by Thorsten Scherler <th...@wyona.com>.
Sorry, sluggish/no internet connection and lots of work, prevent me to
be a bigger help.

some remarks:

On Tue, 2006-09-26 at 08:56 +0300, Sjur Moshagen wrote:
> Den 25. sep. 2006 kl. 17.17 skrev Gav....:
> 
> >> 3) when doing a forrest seed, the translation files in whiteboard/
> >> plugins/o.a.f.plugin.internal.dispatcher/messages/ should be included
> >> in the src/documentation/translations/ directory, with a proper note
> >> on them being useful only with the dispatcher. Without translation
> >> files in this location, the standard "messages" (Last published,
> >> Search, etc.) will not be translated.
> >
> > So what is the best way to do this, do you mean just moving the  
> > files into
> > That location permanently and removing them from their current  
> > location, or
> > Doing some xslt magic to create them there from the originals at  
> > build time?
> 
> I would imagine an ant task to copy the files as part of the forrest  
> seed task.

http://issues.apache.org/jira/browse/FOR-799 
and
http://issues.apache.org/jira/browse/FOR-809

This issue are about removing such ant tasks (the last issue is about
skin related copy task). Maybe we should take the opportunity and
refactor this targets. Something like forrest seed-dispatcher and
forrest seed-skins both based on the fresh site and then copying the
specific stuff from the plugins.

> 
> > Have you tested these locally with your patches you made ?
> 
> Yes, and I found that the Dispatcher translation files only work if  
> placed in the project location (but I didn't test this  
> systematically, so please repeat it with your setup)

It is a long time when I added the i18n support to the dispatcher and
since then hardly reviewed it. 
http://svn.apache.org/viewvc/forrest/trunk/whiteboard/plugins/org.apache.forrest.plugin.internal.dispatcher/internal.xmap?view=markup
<map:transformer name="i18n" 
        src="org.apache.cocoon.transformation.I18nTransformer">
        <catalogues default="contracts">
          <catalogue id="other" name="OtherMessages" location="{lm:project.translations}" />
          <catalogue id="contracts" name="ContractsMessages" 
            location="{lm:project.translations}" />
        </catalogues>
        <cache-at-startup>true</cache-at-startup>
      </map:transformer>

I reckon the problem is because {lm:project.translations} is not looking
into the fallback translation of the dispatcher. 

http://svn.apache.org/viewvc/forrest/trunk/main/webapp/locationmap-i18n.xml?view=markup

<match pattern="project.translations">
        <location src="{properties:translations}" />
    </match>

Did you try to change the properties:translations in your forrest.properties(.xml) to point to 
http://svn.apache.org/viewvc/forrest/trunk/whiteboard/plugins/org.apache.forrest.themes.core/messages/

That should work, but here you see that we need to harmonize the i18n stuff of all plugins.

> 
> > But I think a new Jira issue should be made to remember to do  
> > these, maybe
> > as a subtask as one of the others? Sub-tasks don't get used often  
> > enough
> > IMHO and think that maybe the three you just opened could have been  
> > all
> > sub-tasks of one jira issue. But this is a reflection on our  
> > current methods
> > as
> > A whole rather than how you created them - everyone else would have  
> > done the
> > Same I think. This is a good example of sub-tasking, each of those  
> > three
> > issues combined with the 3 new issues you identify above would not  
> > work
> > without the others also being completed.
> 
> I'll try to create one super-issue and add the existing (and new)  
> issues as sub-issues to that one.
> 
> > Great work on all this i18n stuff !! :)
> 
> Thanks:-)

Yeah, thanks very much Sjur. I will try to catch up with all your work
but ATM I am not able. Sorry and keep up the good work.

salu2
-- 
Thorsten Scherler
COO Spain
Wyona Inc.  -  Open Source Content Management  -  Apache Lenya
http://www.wyona.com                   http://lenya.apache.org
thorsten.scherler@wyona.com                thorsten@apache.org


Re: Dispatcher i18n update + three more issues

Posted by Ross Gardler <rg...@apache.org>.
Sjur Moshagen wrote:
> Den 26. sep. 2006 kl. 09.07 skrev Sjur Moshagen:
> 
>> Den 26. sep. 2006 kl. 08.56 skrev Sjur Moshagen:
>>
>>> I'll try to create one super-issue and add the existing (and new) 
>>> issues as sub-issues to that one.
>>
>> Hm, I received the following error message when I tried to create a 
>> sub-task to my newly created super-issue 
>> http://issues.apache.org/jira/browse/FOR-938 :
>> Errors
>>
>> org.ofbiz.core.entity.GenericEntityException: while inserting: 
>> [GenericEntity:IssueLink][destination,12351006][linktype,10010][source,12351004][sequence,0][id,12313523] 
>> (SQL Exception while executing the following:INSERT INTO issuelink 
>> (ID, LINKTYPE, SOURCE, DESTINATION, SEQUENCE) VALUES (?, ?, ?, ?, ?) 
>> (Duplicate entry '12313523' for key 1))
> 
> It seems the issue was created, but the linking between it and its 
> superissue failed. At least there are no visible links between the 
> issues in JIRA.
> 
> I'll wait with adding the other issues until others have responded to 
> the error message above.

Jira is broken, please raise it at infra@ (probably best to do this via 
an issue). Issues with existing subtasks work OK, so it just looks like 
a problem with creting new subtasks.

Ross

Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
Den 26. sep. 2006 kl. 09.07 skrev Sjur Moshagen:

> Den 26. sep. 2006 kl. 08.56 skrev Sjur Moshagen:
>
>> I'll try to create one super-issue and add the existing (and new)  
>> issues as sub-issues to that one.
>
> Hm, I received the following error message when I tried to create a  
> sub-task to my newly created super-issue http://issues.apache.org/ 
> jira/browse/FOR-938 :
> Errors
>
> org.ofbiz.core.entity.GenericEntityException: while inserting:  
> [GenericEntity:IssueLink][destination,12351006][linktype,10010] 
> [source,12351004][sequence,0][id,12313523] (SQL Exception while  
> executing the following:INSERT INTO issuelink (ID, LINKTYPE,  
> SOURCE, DESTINATION, SEQUENCE) VALUES (?, ?, ?, ?, ?) (Duplicate  
> entry '12313523' for key 1))

It seems the issue was created, but the linking between it and its  
superissue failed. At least there are no visible links between the  
issues in JIRA.

I'll wait with adding the other issues until others have responded to  
the error message above.

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
Den 26. sep. 2006 kl. 08.56 skrev Sjur Moshagen:

> I'll try to create one super-issue and add the existing (and new)  
> issues as sub-issues to that one.

Hm, I received the following error message when I tried to create a  
sub-task to my newly created super-issue http://issues.apache.org/ 
jira/browse/FOR-938 :
Errors

org.ofbiz.core.entity.GenericEntityException: while inserting:  
[GenericEntity:IssueLink][destination,12351006][linktype,10010] 
[source,12351004][sequence,0][id,12313523] (SQL Exception while  
executing the following:INSERT INTO issuelink (ID, LINKTYPE, SOURCE,  
DESTINATION, SEQUENCE) VALUES (?, ?, ?, ?, ?) (Duplicate entry  
'12313523' for key 1))

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
Den 25. sep. 2006 kl. 17.17 skrev Gav....:

>> 3) when doing a forrest seed, the translation files in whiteboard/
>> plugins/o.a.f.plugin.internal.dispatcher/messages/ should be included
>> in the src/documentation/translations/ directory, with a proper note
>> on them being useful only with the dispatcher. Without translation
>> files in this location, the standard "messages" (Last published,
>> Search, etc.) will not be translated.
>
> So what is the best way to do this, do you mean just moving the  
> files into
> That location permanently and removing them from their current  
> location, or
> Doing some xslt magic to create them there from the originals at  
> build time?

I would imagine an ant task to copy the files as part of the forrest  
seed task.

> Have you tested these locally with your patches you made ?

Yes, and I found that the Dispatcher translation files only work if  
placed in the project location (but I didn't test this  
systematically, so please repeat it with your setup)

> But I think a new Jira issue should be made to remember to do  
> these, maybe
> as a subtask as one of the others? Sub-tasks don't get used often  
> enough
> IMHO and think that maybe the three you just opened could have been  
> all
> sub-tasks of one jira issue. But this is a reflection on our  
> current methods
> as
> A whole rather than how you created them - everyone else would have  
> done the
> Same I think. This is a good example of sub-tasking, each of those  
> three
> issues combined with the 3 new issues you identify above would not  
> work
> without the others also being completed.

I'll try to create one super-issue and add the existing (and new)  
issues as sub-issues to that one.

> Great work on all this i18n stuff !! :)

Thanks:-)

Sjur


applying patches

Posted by David Crossley <cr...@apache.org>.
Gav.... wrote:
> > From: Ross Gardler
> > Gav.... wrote:
> > 
> > >> 2) There should probably be a default/fallback ContractsMessages.xml
> > >> file, which should really be the file above (ie the English one, if
> > >> one wants to have English as the fallback language, which I expect
> > >> one does). Not having one will at least cause some noise in the log
> > >> files, and might also generate errors
> > >
> > > Ok, so exactly the same file, no changes apart from the file name ?
> > >
> > > No sure how others do it, but I would just copy the file, give it the
> > > New name and then add it to our svn with 'svn add filename.xml' and then
> > > Commit it. So, again no patch needed.
> > 
> > To copy a file in svn do "svn cp SRC_FILE DEST_FILE" this has the
> > advantage of keeping the SVN history for the file.
> 
> Ok Sounds good. How do you go about applying patches locally then? 
> Suppose I'm downloading Sjurs internal.xmap patch, normally I just
> Overwrite the current version with the patched version, is there a
> Better way?

Yes there is. It is better to ask the contributor to provide a
patch rather than just the full file. This ensures that it is
patched against the correct version of the file.
http://forrest.apache.org/contrib.html#patch

Otherwise you might clobber other recent changes to the file.

Then you apply the patch with
patch -p0 < ~/Desktop/internal.xmap.diff

If that is not possible then take the full file.

As with all patches or files you need to first convert the
file to the correct line-endings. Sjur has UNIX and you are
on Windows.

Then do the following ...

cd $FORREST_HOME
svn up
cp ~/Desktop/internal.xmap whiteboard/plugins/org.apache.forrest.plugin.internal.dispatcher/internal.xmap
svn diff whiteboard/plugins/org.apache.forrest.plugin.internal.dispatcher/internal.xmap
... make sure it is what you expect. Then test and commit.

-David

Re: i18n & LM

Posted by Sjur Moshagen <sj...@mac.com>.
[Changed the subject to reflect the topic at han]

Den 27. sep. 2006 kl. 12.13 skrev Ross Gardler:

> Sjur Moshagen wrote:
>> [WARNING - long e-mail]
>
> (not any more ;-)

:-)

>> This is a lot of work, and I will not be able to do all of this,  
>> both in terms of time and knowledge. The first question is easy  
>> enough though:
>> WDYT?
>
> The solution to the first three use cases involves a value in a  
> session.  There is no concept of a session in a static build so how  
> would this play with the CLI?

In the given scenario it should not have any impact. The use of a  
session value is to create a more pleasent user experience by  
"remembering" the selected locale (=store locale in session). When  
running the CLI one would just traverse all links in the normal way,  
and by the proposed changes, also the language override links, since  
these are now regular links. This way all, all files, localised or  
not, will be included in the final, static site.

Use case 3) is irrelevant for the CLI scenario, I believe.

That is: I see no problem with the session and the CLI.

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Ross Gardler <rg...@apache.org>.
Sjur Moshagen wrote:
> [WARNING - long e-mail]

(not any more ;-)

> This is a lot of work, and I will not be able to do all of this, both in 
> terms of time and knowledge. The first question is easy enough though:
> 
> WDYT?

The solution to the first three use cases involves a value in a session. 
  There is no concept of a session in a static build so how would this 
play with the CLI?

Ross

Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
[WARNING - long e-mail]

Den 26. sep. 2006 kl. 12.37 skrev Ross Gardler:

> This brings me back to the idea of using the locationmap for locale  
> matching rather than Cocoons components, which have their problems  
> in our use case. Something like (be warned, I've not fully thought  
> this through and do not have any i18n sites, I'm not at all sure  
> this will work in the real world):
>
> <match pattern="**.xml">
>     <location src="{project:xdocs}/{1}.{2}.xml/>
> </location>
>
> <match pattern="**.*.xml">
>   <select>
>     <location src="{project:xdocs}/{1}.{2}.xml/>
>     <location src="{project:xdocs}/{1}.{project:locale.default}.xml/>
>   </select>
> </location>

I have been thinking about the LM solution a bit, and here are some  
thoughts (some of these may be unneeded, but I do not understand or  
know the LM, thus this is as well a way for me to clear my thoughts):

Goal: change the localised URL from the 'index.html?locale=XY'  
pattern to something along 'index.XY.html when overriding browser  
locale settings.

I see no point in removing the present i18n processing, replacing it  
completely with LM; instead I hope we can use the LM to overcome the  
URL problem above. Thus, the Cocoon components and the LM should be  
able to coexist. They both have their use cases (see discussion below).

There are four possible combinations of URLs and files:

1 - URL is localised - file is localised
2 - URL is localised - file is not
3 - URL is not localised - file is
4 - URL is not localised - file is not

1 - URL is localised - file is localised
========================================

Use case: the user has selected a language override from a list, or  
has manually edited the URL to try other languages

Assumption: there is a match between requested locale and available  
locale (non-match is covered by case 2)

Actions:
* return the corresponding localised file, using LM
* use the identified locale for all i18n processing of menus, tabs  
and messages
* make sure the non-localised URL is given the menu and tab  
processing (otherwise the menu system breaks down)
* store the identified locale in the session (such that the next page  
requested will be given in the same locale, if available) (this might  
be dependent upon configuration)

2 - URL is localised - file is not
==================================

Use cases:
1 - the user has selected a language override from a list
2 - the user has manually edited the URL to try other languages

In the first use case it means that the localised URL corresponds to  
the fallback file.
In the second case it most often means that there is no corresponding  
localised file - we need to gracefully handle this situation

Actions:
* check the content of the fall-back file for @xml:lang
** if the fallback document corresponds to the requested locale,  
return it
** if not, fall back to regular negotiation using the browser  
settings - optionally throw an error saying that the document isn't  
available in the requested language (which one to use could depend on  
a setting)
* use the identified locale for all i18n processing of menus, tabs  
and messages
* make sure the non-localised URL is given the menu and tab  
processing (otherwise the menu system breaks down)
* store the identified locale in the session (such that the next page  
requested will be given in the same locale, if available) (this might  
be dependent upon configuration)

3 - URL is not localised - file is
==================================

Use cases:
1 - the user is entering the site - traditional negotiation should apply
2 - the user is entering a new page after a locale selection - the  
locale should be picked from the session if available

1) is what we do today with the Cocoon modules, 2) is just a  
variation of 1.

Actions:
* return the corresponding localised file (Cocoon i18n components)
* use the identified locale for all i18n processing of menus, tabs  
and messages
* store the identified locale in the session (such that the next page  
requested will be given in the same locale, if available) (this might  
be dependent upon configuration)

4 - URL is not localised - file is not
======================================

Use case: this is the fallback case - no match found, return the  
default/fallback.

Actions:
* return the fallback file (Cocoon i18n components)
* use the fallback locale for all i18n processing of menus, tabs and  
messages, to give a consistent view of the page (all elements in the  
same language) (should this be depending on a setting, thus allowing  
menus etc to be in the user's preferred language, even though the  
document is not?)
* do NOT store the document locale in the session - the next time the  
user returns to a page in the preferred locale, that one should be used

It is combinations 1 & 2 that are problematic from a URL POV, and  
this is where we will need the LM.

Work to be done:
- figure out whether the solution sketched here will actually work,  
or whether another approach should be taken
- enhance all document lookups with a locationmap variant for cases  
where the URL is of pattern **.*.* ({0} = document ref, {1} = locale,  
{2} = extension) - this will likely collide with existing patterns  
for internal matches, and needs thorough consideration
- figure out how to set the session locale from a LM/i18n match
- adapt the menu and tab processing (this is probably a subset of the  
first task)

This is a lot of work, and I will not be able to do all of this, both  
in terms of time and knowledge. The first question is easy enough  
though:

WDYT?

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
Den 26. sep. 2006 kl. 15.00 skrev Thorsten Scherler:

>> Using the locationmap is fine, but not until content negotiation is
>> working.
>
>
> Just want to remark that the LocaleAction should as well work in  
> the lm.

Thanks for pointing it out - whatever can be done to make both the  
webapp and the CLI happy running an i18n site is A Good Thing (tm).

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Thorsten Scherler <th...@apache.org>.
On Tue, 2006-09-26 at 14:49 +0300, Sjur Moshagen wrote:
> Den 26. sep. 2006 kl. 12.37 skrev Ross Gardler:
> 
> >> I can open http://localhost:8888/index.en.html and have my page in  
> >> English. The problem with this solution is that it does not work  
> >> nicely in a webapp environment (e.g. forrest run), because this  
> >> locale encoding does not communicate with the locale modules in  
> >> Cocoon (LocaleAction, LocaleMatcher, I18nTransformer). Thus, all  
> >> menus and tabs are/will most likely be in a different language. In  
> >> addition, it's location in the menu is lost.
> >
> > This brings me back to the idea of using the locationmap for locale  
> > matching rather than Cocoons components, which have their problems  
> > in our use case. Something like (be warned, I've not fully thought  
> > this through and do not have any i18n sites, I'm not at all sure  
> > this will work in the real world):
> >
> > <match pattern="**.xml">
> >     <location src="{project:xdocs}/{1}.{2}.xml/>
> > </location>
> >
> > <match pattern="**.*.xml">
> >   <select>
> >     <location src="{project:xdocs}/{1}.{2}.xml/>
> >     <location src="{project:xdocs}/{1}.{project:locale.default}.xml/>
> >   </select>
> > </location>
> 
> This looses completely the concept of content negotiation, which is  
> exactly what the Cocoon modules provide. And this is really the core  
> of the problem: when running in a webapp environment, you want  
> negotiation, and ways to override the negotiated result, whereas in a  
> static build you don't want negotiation at all, just all files  
> generated to the requested output format(s).
> 
> Using the locationmap is fine, but not until content negotiation is  
> working.


Just want to remark that the LocaleAction should as well work in the lm.

salu2

> 
> I must admit I never tried the CLI during my development of the  
> language override menu, since my own usecase is solely 'forrest run'.  
> But we do need to generate a static version of our site as well, thus  
> this is also important to us.
> 
> >> The problem with static builds is a known one in that you can't  
> >> build all locales in the same run.
> >
> > Yes, but if we have an url with foo.html?locale=en then no static  
> > site build will work due to the way Cocoon encodes the url  
> > parameters in the filename. Unless you have found away this.
> 
> No. Now that I finally tried a static build, I have found lots of  
> errors durng the build, all of them belonging to two types:
> 
> X [0]                                     wordsdoc/search/ 
> index.html    BROKEN:  
> org.apache.cocoon.environment.commandline.CommandLineRequest.getLocales( 
> ) method not yet implemented!
> 
> X [0]                                     /doc/ling/corpus-conversion- 
> tech.html BROKEN: dispatcherError: 500 - Internal server error
> The contract "siteinfo-meta-navigation" has thrown thrown an  
> exception by resolving raw data from "cocoon://doc/ling/corpus- 
> conversion-tech.navigation.xml".
> 
> dispatcherErrorStack:
> org.apache.excalibur.source.SourceNotFoundException: Exception during  
> processing of cocoon://doc/ling/corpus-conversion-tech.navigation.xml
> 
> The first error is clear enough - no direct support for locales in  
> the CLI.
> 
> The other one is harder - I already saw this error when running  
> forrest run, but I could not debug it. It seems to be a problem with  
> HTML source files only, and it is NOT related to dispatcher, despite  
> the text in the error message. It originates somewhere in the  
> **book.xml processing, breaking down in a sitemap match for no  
> appearent reason. More on this later.
> 
> >>> The global language default should be set in a config file.
> >> Agree, I just don't know how to do it.
> >
> > It depends if you want to make your work compatible with 0.7  
> > version of Forrest. If you do then see [1], but for an easier way  
> > in 0.8 see [2]
> 
> 0.8 is fine with us.
> 
> > Note both of these are issues to document this process, so ask  
> > questions if you don't understand.
> 
> Ok, I will.
> 
> > Ross
> >
> > [1] http://issues.apache.org/jira/browse/FOR-777
> > [2] http://issues.apache.org/jira/browse/FOR-588
> 
> Sjur
> 
-- 
thorsten

"Together we stand, divided we fall!" 
Hey you (Pink Floyd)


Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
Den 26. sep. 2006 kl. 12.37 skrev Ross Gardler:

>> I can open http://localhost:8888/index.en.html and have my page in  
>> English. The problem with this solution is that it does not work  
>> nicely in a webapp environment (e.g. forrest run), because this  
>> locale encoding does not communicate with the locale modules in  
>> Cocoon (LocaleAction, LocaleMatcher, I18nTransformer). Thus, all  
>> menus and tabs are/will most likely be in a different language. In  
>> addition, it's location in the menu is lost.
>
> This brings me back to the idea of using the locationmap for locale  
> matching rather than Cocoons components, which have their problems  
> in our use case. Something like (be warned, I've not fully thought  
> this through and do not have any i18n sites, I'm not at all sure  
> this will work in the real world):
>
> <match pattern="**.xml">
>     <location src="{project:xdocs}/{1}.{2}.xml/>
> </location>
>
> <match pattern="**.*.xml">
>   <select>
>     <location src="{project:xdocs}/{1}.{2}.xml/>
>     <location src="{project:xdocs}/{1}.{project:locale.default}.xml/>
>   </select>
> </location>

This looses completely the concept of content negotiation, which is  
exactly what the Cocoon modules provide. And this is really the core  
of the problem: when running in a webapp environment, you want  
negotiation, and ways to override the negotiated result, whereas in a  
static build you don't want negotiation at all, just all files  
generated to the requested output format(s).

Using the locationmap is fine, but not until content negotiation is  
working.

I must admit I never tried the CLI during my development of the  
language override menu, since my own usecase is solely 'forrest run'.  
But we do need to generate a static version of our site as well, thus  
this is also important to us.

>> The problem with static builds is a known one in that you can't  
>> build all locales in the same run.
>
> Yes, but if we have an url with foo.html?locale=en then no static  
> site build will work due to the way Cocoon encodes the url  
> parameters in the filename. Unless you have found away this.

No. Now that I finally tried a static build, I have found lots of  
errors durng the build, all of them belonging to two types:

X [0]                                     wordsdoc/search/ 
index.html    BROKEN:  
org.apache.cocoon.environment.commandline.CommandLineRequest.getLocales( 
) method not yet implemented!

X [0]                                     /doc/ling/corpus-conversion- 
tech.html BROKEN: dispatcherError: 500 - Internal server error
The contract "siteinfo-meta-navigation" has thrown thrown an  
exception by resolving raw data from "cocoon://doc/ling/corpus- 
conversion-tech.navigation.xml".

dispatcherErrorStack:
org.apache.excalibur.source.SourceNotFoundException: Exception during  
processing of cocoon://doc/ling/corpus-conversion-tech.navigation.xml

The first error is clear enough - no direct support for locales in  
the CLI.

The other one is harder - I already saw this error when running  
forrest run, but I could not debug it. It seems to be a problem with  
HTML source files only, and it is NOT related to dispatcher, despite  
the text in the error message. It originates somewhere in the  
**book.xml processing, breaking down in a sitemap match for no  
appearent reason. More on this later.

>>> The global language default should be set in a config file.
>> Agree, I just don't know how to do it.
>
> It depends if you want to make your work compatible with 0.7  
> version of Forrest. If you do then see [1], but for an easier way  
> in 0.8 see [2]

0.8 is fine with us.

> Note both of these are issues to document this process, so ask  
> questions if you don't understand.

Ok, I will.

> Ross
>
> [1] http://issues.apache.org/jira/browse/FOR-777
> [2] http://issues.apache.org/jira/browse/FOR-588

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Ross Gardler <rg...@apache.org>.
Sjur Moshagen wrote:
> Den 26. sep. 2006 kl. 10.34 skrev Ross Gardler:
> 
>> Does this new i18n solution require a ?locale=xx in the URL in any 
>> situations? If so then this is not a good solution. It will break the 
>> static build of the site since "?" is not a valid character in 
>> filenames, Cocoon therefore encodes this which in turn breaks links.
>>
>> You need to make the language selection part of the url, i.e.
>>
>> http://foo.com/bar.en.html
>>
>> or
>>
>> http://foo.com/en/bar.xml
> 
> I can open http://localhost:8888/index.en.html and have my page in 
> English. The problem with this solution is that it does not work nicely 
> in a webapp environment (e.g. forrest run), because this locale encoding 
> does not communicate with the locale modules in Cocoon (LocaleAction, 
> LocaleMatcher, I18nTransformer). Thus, all menus and tabs are/will most 
> likely be in a different language. In addition, it's location in the 
> menu is lost.

This brings me back to the idea of using the locationmap for locale 
matching rather than Cocoons components, which have their problems in 
our use case. Something like (be warned, I've not fully thought this 
through and do not have any i18n sites, I'm not at all sure this will 
work in the real world):

<match pattern="**.xml">
     <location src="{project:xdocs}/{1}.{2}.xml/>
</location>

<match pattern="**.*.xml">
   <select>
     <location src="{project:xdocs}/{1}.{2}.xml/>
     <location src="{project:xdocs}/{1}.{project:locale.default}.xml/>
   </select>
</location>

> The problem with static builds is a known one in that you can't build 
> all locales in the same run. 

Yes, but if we have an url with foo.html?locale=en then no static site 
build will work due to the way Cocoon encodes the url parameters in the 
filename. Unless you have found away this.

>> The global language default should be set in a config file.
> 
> Agree, I just don't know how to do it.

It depends if you want to make your work compatible with 0.7 version of 
Forrest. If you do then see [1], but for an easier way in 0.8 see [2]

Note both of these are issues to document this process, so ask questions 
if you don't understand.

Ross

[1] http://issues.apache.org/jira/browse/FOR-777
[2] http://issues.apache.org/jira/browse/FOR-588

Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
Den 26. sep. 2006 kl. 10.34 skrev Ross Gardler:

> Does this new i18n solution require a ?locale=xx in the URL in any  
> situations? If so then this is not a good solution. It will break  
> the static build of the site since "?" is not a valid character in  
> filenames, Cocoon therefore encodes this which in turn breaks links.
>
> You need to make the language selection part of the url, i.e.
>
> http://foo.com/bar.en.html
>
> or
>
> http://foo.com/en/bar.xml

I can open http://localhost:8888/index.en.html and have my page in  
English. The problem with this solution is that it does not work  
nicely in a webapp environment (e.g. forrest run), because this  
locale encoding does not communicate with the locale modules in  
Cocoon (LocaleAction, LocaleMatcher, I18nTransformer). Thus, all  
menus and tabs are/will most likely be in a different language. In  
addition, it's location in the menu is lost.

The problem with static builds is a known one in that you can't build  
all locales in the same run. But my colleague is building the site  
just fine, running the CLI once for each locale IIHUC. I'll ask him  
to fill in the details/correct me. Thus, it works, although not  
optimally.

>> although I am a little unsure of the impact of the fact that en_US  
>> is defined as the global default language. It could mean that for  
>> that special case you would get the _en version without writing ? 
>> locale=en.
>
> The global language default should be set in a config file.

Agree, I just don't know how to do it.

>> That is, you can just delete it, and the strings will appear in  
>> English if there's no match. The only benefit of having it there  
>> is to serve as an example for new translations.
>
> And to allow people to override the default settings for those  
> strings, even in English.
>
> I'm -1 on deleting it for that reason and for the two reasons  
> (documentation and human readability) that Sjur provides.

That's fine, it does not do any harm having it there. Just rename it  
to the correct language code:-)

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Ross Gardler <rg...@apache.org>.
Sjur Moshagen wrote:
> Den 26. sep. 2006 kl. 02.14 skrev Gav....:
> 
>>> In this case shouldn't the fallback be defined as a specific language,
>>> i.e. English. In which case the fallback should be
>>> "ContractsMessages_en.xml" shouldn't it (as opposed to
>>> "ContractMessages.xml").
>>
>> Sounds like the best way, haven't tested Sjurs patches yet, but hopefully
>> Then that wont mean having ?locale=en in the url for the default.

Does this new i18n solution require a ?locale=xx in the URL in any 
situations? If so then this is not a good solution. It will break the 
static build of the site since "?" is not a valid character in 
filenames, Cocoon therefore encodes this which in turn breaks links.

You need to make the language selection part of the url, i.e.

http://foo.com/bar.en.html

or

http://foo.com/en/bar.xml

Sorry, I've not been following the i18n work or looked at these patches, 
i'm just trying to pick up on this threa now.

> Actually, Ross' suggestion would require just that, 

See above.

> although I am a 
> little unsure of the impact of the fact that en_US is defined as the 
> global default language. It could mean that for that special case you 
> would get the _en version without writing ?locale=en.

The global language default should be set in a config file.

> That is, you can just delete it, and the strings will appear in English 
> if there's no match. The only benefit of having it there is to serve as 
> an example for new translations.

And to allow people to override the default settings for those strings, 
even in English.

I'm -1 on deleting it for that reason and for the two reasons 
(documentation and human readability) that Sjur provides.

Ross

Re: Dispatcher i18n update + three more issues

Posted by Sjur Moshagen <sj...@mac.com>.
Den 26. sep. 2006 kl. 02.14 skrev Gav....:

>> In this case shouldn't the fallback be defined as a specific  
>> language,
>> i.e. English. In which case the fallback should be
>> "ContractsMessages_en.xml" shouldn't it (as opposed to
>> "ContractMessages.xml").
>
> Sounds like the best way, haven't tested Sjurs patches yet, but  
> hopefully
> Then that wont mean having ?locale=en in the url for the default.

Actually, Ross' suggestion would require just that, although I am a  
little unsure of the impact of the fact that en_US is defined as the  
global default language. It could mean that for that special case you  
would get the _en version without writing ?locale=en.

In the general case it works like this:

- when there's no locale match, the ContractsMessages.xml (without  
_en) will be used
- if there is a locale match, the match will be used
- if there's no match and no default/fallback file (like  
ContractsMessages.xml), the strings will be left untranslated

That means that in cases where the input strings (those to be  
translated) are themselves perfect strings in one language (say,  
English), *there's no need to do translation at all*. Which means  
that all of the following files are superfluous:

- ContractsMessages_en.xml
- ContractsMessages_us.xml (the original, with misspelled locale)
- ContractsMessages.xml

That is, you can just delete it, and the strings will appear in  
English if there's no match. The only benefit of having it there is  
to serve as an example for new translations.

Beware that when the input strings are NOT human readable by most  
people (like ISO 639 language codes), it is necessary to have at  
least the fallback file to ensure that the input strings are always  
translated into something understandable in some language.

Sjur


Re: Dispatcher i18n update + three more issues

Posted by Ross Gardler <rg...@apache.org>.
Gav.... wrote:
> 
>> -----Original Message-----
>> From: Ross Gardler [mailto:rgardler@apache.org]
>> Sent: Monday, 25 September 2006 10:54 PM
>> To: dev@forrest.apache.org
>> Subject: Re: Dispatcher i18n update + three more issues
>>
>> Gav.... wrote:
>>
>>>> 2) There should probably be a default/fallback ContractsMessages.xml
>>>> file, which should really be the file above (ie the English one, if
>>>> one wants to have English as the fallback language, which I expect
>>>> one does). Not having one will at least cause some noise in the log
>>>> files, and might also generate errors
>>> Ok, so exactly the same file, no changes apart from the file name ?
>>>
>>> No sure how others do it, but I would just copy the file, give it the
>>> New name and then add it to our svn with 'svn add filename.xml' and then
>>> Commit it. So, again no patch needed.
>> To copy a file in svn do "svn cp SRC_FILE DEST_FILE" this has the
>> advantage of keeping the SVN history for the file.
> 
> Ok Sounds good. How do you go about applying patches locally then? 
> Suppose I'm downloading Sjurs internal.xmap patch, normally I just
> Overwrite the current version with the patched version, is there a
> Better way?

Just apply the patches as notmal. SVN is smart enough to do the copy and 
modification at the same time.

You can use the command line to apply a patch to a different file if 
you've copied a file across. You'll have to look that one up i the docs 
though, I can't remember the syntax.

Ross

RE: Dispatcher i18n update + three more issues

Posted by "Gav...." <br...@brightontown.com.au>.

> -----Original Message-----
> From: Ross Gardler [mailto:rgardler@apache.org]
> Sent: Monday, 25 September 2006 10:54 PM
> To: dev@forrest.apache.org
> Subject: Re: Dispatcher i18n update + three more issues
> 
> Gav.... wrote:
> 
> >> 2) There should probably be a default/fallback ContractsMessages.xml
> >> file, which should really be the file above (ie the English one, if
> >> one wants to have English as the fallback language, which I expect
> >> one does). Not having one will at least cause some noise in the log
> >> files, and might also generate errors
> >
> > Ok, so exactly the same file, no changes apart from the file name ?
> >
> > No sure how others do it, but I would just copy the file, give it the
> > New name and then add it to our svn with 'svn add filename.xml' and then
> > Commit it. So, again no patch needed.
> 
> To copy a file in svn do "svn cp SRC_FILE DEST_FILE" this has the
> advantage of keeping the SVN history for the file.

Ok Sounds good. How do you go about applying patches locally then? 
Suppose I'm downloading Sjurs internal.xmap patch, normally I just
Overwrite the current version with the patched version, is there a
Better way?

> 
> However, be wary of copying files (as opposed to moving). If you have
> two files that are identical, and are likely to remain identical, as is
> the case here, this is usually an indicator of doing something the wrong
> way.
> 
> In this case shouldn't the fallback be defined as a specific language,
> i.e. English. In which case the fallback should be
> "ContractsMessages_en.xml" shouldn't it (as opposed to
> "ContractMessages.xml").

Sounds like the best way, haven't tested Sjurs patches yet, but hopefully
Then that wont mean having ?locale=en in the url for the default.

> 
> ...
> 
> > I guess once the jira issues you created are applied then we'll be able
> to
> > Test these out also.
> >
> > I'm going to download your patches and try them out anyway.
> 
> Remember you are committer Gav. If you find they work then commit them,
> don't wait for others to test them as well.

No problem, will do as soon as I can.

Gav...

> 
> > But I think a new Jira issue should be made to remember to do these,
> maybe
> > as a subtask as one of the others? Sub-tasks don't get used often enough
> > IMHO
> 
> +1
> 
> > Great work on all this i18n stuff !! :)
> 
> +1000
> 
> Ross
> 
> 
> --
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.1.405 / Virus Database: 268.12.8/455 - Release Date: 9/22/2006




Re: Dispatcher i18n update + three more issues

Posted by Ross Gardler <rg...@apache.org>.
Gav.... wrote:

>> 2) There should probably be a default/fallback ContractsMessages.xml
>> file, which should really be the file above (ie the English one, if
>> one wants to have English as the fallback language, which I expect
>> one does). Not having one will at least cause some noise in the log
>> files, and might also generate errors
> 
> Ok, so exactly the same file, no changes apart from the file name ?
> 
> No sure how others do it, but I would just copy the file, give it the
> New name and then add it to our svn with 'svn add filename.xml' and then
> Commit it. So, again no patch needed.

To copy a file in svn do "svn cp SRC_FILE DEST_FILE" this has the 
advantage of keeping the SVN history for the file.

However, be wary of copying files (as opposed to moving). If you have 
two files that are identical, and are likely to remain identical, as is 
the case here, this is usually an indicator of doing something the wrong 
way.

In this case shouldn't the fallback be defined as a specific language, 
i.e. English. In which case the fallback should be 
"ContractsMessages_en.xml" shouldn't it (as opposed to 
"ContractMessages.xml").

...

> I guess once the jira issues you created are applied then we'll be able to
> Test these out also.
> 
> I'm going to download your patches and try them out anyway.

Remember you are committer Gav. If you find they work then commit them, 
don't wait for others to test them as well.

> But I think a new Jira issue should be made to remember to do these, maybe
> as a subtask as one of the others? Sub-tasks don't get used often enough
> IMHO

+1

> Great work on all this i18n stuff !! :)

+1000

Ross

RE: Dispatcher i18n update + three more issues

Posted by "Gav...." <br...@brightontown.com.au>.

> -----Original Message-----
> From: Sjur Moshagen [mailto:sjurnm@mac.com]
> Sent: Monday, 25 September 2006 8:20 PM
> To: dev@forrest.apache.org
> Subject: Dispatcher i18n update + three more issues
> 
> Hello all,
> 
> AS seen in JIRA, I have found a set of errors in Dispatcher and
> Forrest regarding i18n, and submitted patches for them. With those
> patches, Dispatcher should be ready to go re i18n - almost:-) There
> are still three more issues for which I can't submit a patch (I think):
> 
> 1) one file is wrongly named in whiteboard/plugins/
> o.a.f.plugin.internal.dispatcher/messages/
> 
> ContractsMessages_us.xml
> 
> should most likely be:
> 
> ContractsMessages_en.xml

For svn this would would be a 'svn mv' command I think to rename it so no
Patch needed just an issue to note to do it.

> 
> 2) There should probably be a default/fallback ContractsMessages.xml
> file, which should really be the file above (ie the English one, if
> one wants to have English as the fallback language, which I expect
> one does). Not having one will at least cause some noise in the log
> files, and might also generate errors

Ok, so exactly the same file, no changes apart from the file name ?

No sure how others do it, but I would just copy the file, give it the
New name and then add it to our svn with 'svn add filename.xml' and then
Commit it. So, again no patch needed.

> 
> 3) when doing a forrest seed, the translation files in whiteboard/
> plugins/o.a.f.plugin.internal.dispatcher/messages/ should be included
> in the src/documentation/translations/ directory, with a proper note
> on them being useful only with the dispatcher. Without translation
> files in this location, the standard "messages" (Last published,
> Search, etc.) will not be translated.

So what is the best way to do this, do you mean just moving the files into
That location permanently and removing them from their current location, or
Doing some xslt magic to create them there from the originals at build time?

Have you tested these locally with your patches you made ?

I guess once the jira issues you created are applied then we'll be able to
Test these out also.

I'm going to download your patches and try them out anyway.

But I think a new Jira issue should be made to remember to do these, maybe
as a subtask as one of the others? Sub-tasks don't get used often enough
IMHO and think that maybe the three you just opened could have been all
sub-tasks of one jira issue. But this is a reflection on our current methods
as
A whole rather than how you created them - everyone else would have done the
Same I think. This is a good example of sub-tasking, each of those three
issues combined with the 3 new issues you identify above would not work
without the others also being completed.

Great work on all this i18n stuff !! :)

Gav...

> 
> Best regards,
> Sjur
> 
> 
> 
> --
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.1.405 / Virus Database: 268.12.8/455 - Release Date: 9/22/2006