You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@openoffice.apache.org by bu...@apache.org on 2016/03/09 13:57:02 UTC

[Issue 126863] New: en_AU.dic has UTF-8 errors

https://bz.apache.org/ooo/show_bug.cgi?id=126863

          Issue ID: 126863
        Issue Type: DEFECT
           Summary: en_AU.dic has UTF-8 errors
           Product: General
           Version: 4.2.0-dev
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: Normal
          Priority: P5 (lowest)
         Component: spell checking
          Assignee: issues@openoffice.apache.org
          Reporter: ioot@yahoo.com

In regards to the en_AU.dic extension for Australian spelling, a number of
spellings were corrupted. This appears to have occurred due to incorrect
conversion to/from UTF-8 during adding new words or in the editing process in
2008, but these errors persist to the current version of the en_AU.dic. I would
fix these errors myself but surely there is a maintainer to contact in regards
to this issue? Has it occurred with other dictionaries?

Two options I see, delete all entries with characters that are not Australian
English, or change all those bad characters to good ones. Noting that the
character ï¿½ implies error, not a particular character. In other words we see
variants such as piï¿½ata (should be piñata) and clichï¿½ (should be cliché).

I tracked this down through various versions of the en_AU.dic
http://extensions.services.openoffice.org/en/project/AustralianDictionary

Here is some analysis of version and line numbers of 2 words as they changed
over time. This problem is rife in the newest version of en_AU.dic, with at
least 211 occurrences of the ¿ character, which indicates a failed conversion.
The word cliche, for example, is misrepresented over time in different ways.
Note that many words with the ï¿½ character in the en_AU.dic file never
appeared correctly, although this example for the word cliché was originally
correct but was corrupted over time.

Version 2016.03.01 (Newest)
1700: clichï¿½/SM

Version: 2010.03.16
1700: clichï¿½/SM

Version: 2008.11.25
1700: clich�/SM

Version: 2008.10.3
1702: cliché/MS
1703: clichï¿½/SM

Version: 1.0.0
1523: cliché/MS

With reference to files at:
http://extensions.services.openoffice.org/en/project/english-dictionaries-apache-openoffice
http://extensions.services.openoffice.org/en/project/AustralianDictionary

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

--- Comment #8 from Matthias Seidel <ms...@apache.org> ---
Hi,

thank you  for the confirmation.

I want to close some old issues here. ;-)

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

Tom Anderson <io...@yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ioot@yahoo.com

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

--- Comment #7 from Tom Anderson <io...@yahoo.com> ---
(In reply to Matthias Seidel from comment #6)
> I assume this is fixed, at least I couldn't find these errors in the latest
> en_AU dictionary...
> 
> If you disagree, feel free to reopen.

I also checked and the errors are not there. All the words with the corrupted
accents are now without any accents, e.g. like "cliché/MS" is now "cliche/MDS".
Should work just fine.

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

--- Comment #3 from Andrea Pescetti <pe...@apache.org> ---
Words to be fixed are about 200. The following search lists all "suspicious"
entries, each with its line number in the en_AU.dic file. It is likely that
some smart find/replace can fix the file easily.

$ grep -n -v -e "^[a-zA-Z0-9/'-\. \!]*$" en_AU.dic 
1104:bourï¿½e
1110:boutonniï¿½re/SM
1626:chï¿½telaine/SM
1700:clichï¿½/SM
1701:clichï¿½d
1713:cloisonnï¿½/M
1882:Concepciï¿½n/M
2083:coulï¿½e/SM
2202:crï¿½pe/SM
2468:derriï¿½re/S
2533:diamantï¿½
2604:discothï¿½que/SM
2647:dï¿½modï¿½
2721:dï¿½paysï¿½e
2761:Dï¿½sseldorf
2762:d'ï¿½tre
3010:entrecï¿½te/SM
3011:entrepï¿½t/S
3786:glacï¿½/DGS
4919:kindergï¿½rtner/SM
5209:littï¿½rateur/S
5331:macramï¿½/MS
5458:matinï¿½e/S
5461:maï¿½tre
5652:ï¿½migrï¿½/S
5730:mï¿½nage
5985:nï¿½e
6047:nï¿½gligï¿½
6196:Nounï¿½a
6759:piï¿½on/S
7123:purï¿½eing
7480:rï¿½gime/SM
7540:rï¿½le/MS
8002:shouldâve
8190:Sï¿½o
8208:soignï¿½
8209:soirï¿½e/SM
8731:Tannhï¿½user/M
9119:tï¿½te-bï¿½che
9120:tï¿½te-ï¿½-tï¿½te
10250:appliquï¿½/SMG
10251:appliquï¿½d
10381:attachï¿½/MS
10808:Bogotï¿½/M
11388:chï¿½teau/MS
11445:ï¿½clat/M
11660:confrï¿½re/MS
11819:cortï¿½ge/SM
11896:crï¿½che/MS
11940:cruditï¿½s
12044:Danaï¿½
12075:dï¿½colletage/S
12076:dï¿½coupage
12391:divorcï¿½
12392:divorcï¿½e/SM
12472:dï¿½paysï¿½
12477:dï¿½railleur/SM
12505:dï¿½tente/S
12924:exposï¿½/SM
12954:faï¿½ade/MS
13301:Franï¿½oise/M
13687:Grï¿½newald/M
13702:Gï¿½teborg/M
14541:jalapeï¿½o/S
15118:lycï¿½e
15228:manquï¿½/M
15294:matï¿½riel/MS
15527:mï¿½lï¿½e/SM
15528:mï¿½moire
15663:mï¿½tier/S
15784:naï¿½vety/S
16016:Noumï¿½a/M
16370:passï¿½/M
16824:premiï¿½re/DMGS
16834:pï¿½res/F
16995:purï¿½e/DSM
17101:raison d'ï¿½tre
17465:rï¿½sumï¿½/S
17491:Rï¿½union/M
17801:seï¿½orita/SM
18913:touchï¿½
19393:voilï¿½
19772:abbï¿½/S
19872:adiï¿½s
20700:blasï¿½
21017:cafï¿½/SM
21838:crï¿½pey
21978:dï¿½but/S
21980:dï¿½colletï¿½
22316:dï¿½nouement
22492:Dvorï¿½k/M
22929:Fabergï¿½/M
23037:fiancï¿½/MS
23275:Franï¿½ois
23589:grï¿½ce
23932:Hï¿½loise/M
24445:jardiniï¿½re/MS
24494:Josï¿½/M
24703:lamï¿½
24717:ï¿½lan/M
24952:Lomï¿½/M
25096:Mallarmï¿½/M
25383:mightâve
25709:naivetï¿½/MS
25745:naï¿½vetï¿½/S
26162:outrï¿½
27135:recherchï¿½
27421:rosï¿½
27443:rouï¿½/SM
27541:sï¿½ance/MS
27776:seï¿½ora/SM
27777:Seï¿½ora/M
28142:soupï¿½on/SM
28900:Tomï¿½/M
29414:vis-ï¿½-vis
30136:animï¿½
30291:arï¿½te/MS
30341:Asunciï¿½n/M
30549:Bartï¿½k/M
30579:bï¿½che
30854:bouclï¿½
30935:bric-ï¿½-brac
31646:compï¿½re/M
31723:consommï¿½/S
32070:dï¿½classï¿½
32071:dï¿½classï¿½e
32073:dï¿½cor/MS
32369:dï¿½jï¿½
32406:doppelgï¿½nger
32633:Elysï¿½e/M
32762:Esterhï¿½zy/M
32920:faï¿½ence/S
33018:fiancï¿½e/MS
33269:frappï¿½
33422:Gï¿½del/M
33484:Gewï¿½rztraminer
33731:habituï¿½/SM
34305:ingï¿½nue/S
35087:Lumiï¿½re/M
35494:Mï¿½nchhausen/M
35719:naï¿½veness
36676:portiï¿½re/SM
36721:prï¿½cis/dMS
36869:protï¿½gï¿½/SM
36870:protï¿½gï¿½e/S
36903:pï¿½tï¿½/M
37244:repï¿½chage
37552:sautï¿½/GSD
37596:Schrï¿½dinger/M
37754:seï¿½ores
38976:Tï¿½rshavn/M
39263:Velï¿½squez/M
39308:vicuï¿½a/S
39889:aide-mï¿½moire
40658:ï»¿blowhole
40864:bï¿½te/S
40865:bï¿½tise
41015:canapï¿½/S
41322:ï¿½clair/MS
41360:clientï¿½le/M
41490:communiquï¿½/SM
41741:coupï¿½/SM
41840:croï¿½ton/SM
41853:Cï¿½te
41960:dï¿½bï¿½cle/MS
41962:dï¿½butante/MS
41963:dï¿½colletï¿½e
42395:dï¿½shabillï¿½'s
42678:entrï¿½e/S
43040:fï¿½hn
43041:fï¿½hrer/SM
43108:flambï¿½/DSG
43315:fï¿½te/SM
43385:ï¿½gar
43397:garï¿½on/MS
43668:Gruyï¿½re
44086:howâd
45140:maï¿½ana/M
45216:manï¿½ge/GDS
45346:Mï¿½bius
45604:mï¿½lange
45803:nï¿½
45854:naï¿½ve/Y
45881:negligï¿½e/SM
46200:olï¿½
46501:passï¿½e
46688:piï¿½ata/S
47063:Provenï¿½al
47489:risquï¿½
47872:seï¿½or/M
48226:soufflï¿½/SM
49086:tï¿½te
49797:Yaoundï¿½/M

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

Matthias Seidel <ms...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mseidel@apache.org

--- Comment #5 from Matthias Seidel <ms...@apache.org> ---
Are these fixes in the latest en_AU dictionary?

If yes, can we close this issue?

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

marcoagpinto <ma...@mail.telepac.pt> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jza@oooes.org,
                   |                            |pescetti@apache.org

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

Marcus <ma...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |marcoagpinto@mail.telepac.p
                   |                            |t

--- Comment #1 from Marcus <ma...@apache.org> ---
@Marco:
Please can you have a look? Maybe you can explain and solve the issue. Thanks.

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

--- Comment #4 from marcoagpinto <ma...@mail.telepac.pt> ---
Created attachment 85357
  --> https://bz.apache.org/ooo/attachment.cgi?id=85357&action=edit
en_AU - accents fixes

Here is the fixed .DIC .

I auto-replaced all corrupted characters with an "é" and then had to check the
entire .DIC because over 90% of é's were other characters with accents.

It seems that the corrupted symbol was all the same.

Please tell me if you find any invalid words with accents.

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

marcoagpinto <ma...@mail.telepac.pt> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Latest|---                         |4.1.2
    Confirmation in|                            |
             Status|UNCONFIRMED                 |CONFIRMED
     Ever confirmed|0                           |1

--- Comment #2 from marcoagpinto <ma...@mail.telepac.pt> ---
I have opened the .AFF + .DIC and I confirm that there are corrupted words.

The person who converted the .DIC to UTF-8 probably did something wrong in the
procedure.

I am the maintainer of the English dictionaries but I only add words to the
British one.

The other dictionaries are only packed by me in the monthly OXT.

@Tom, is there any chance you could fix the words since you know what to search
for?

If you do it, please also update the .AFF and README with your name and state
that you fixed the issue.

Then, ZIP and upload the files here and in the monthly update it will be fixed
for everyone.

Thanks!

:-)

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

Matthias Seidel <ms...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|CONFIRMED                   |RESOLVED
         Resolution|---                         |FIXED

--- Comment #6 from Matthias Seidel <ms...@apache.org> ---
I assume this is fixed, at least I couldn't find these errors in the latest
en_AU dictionary...

If you disagree, feel free to reopen.

-- 
You are receiving this mail because:
You are the assignee for the issue.

[Issue 126863] en_AU.dic has UTF-8 errors

Posted by bu...@apache.org.

https://bz.apache.org/ooo/show_bug.cgi?id=126863

Matthias Seidel <ms...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

-- 
You are receiving this mail because:
You are the assignee for the issue.