You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@royale.apache.org by Greg Dove <gr...@gmail.com> on 2023/04/05 20:47:25 UTC

Fwd: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Nice find, Josh.

I was intending to look into some similar issue with resource*
.properties *files
as well.
We have definitely had the need to set utf8 file.encoding in
JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
royale compiler (JS in particular, but I will try to check if royale swf is
at odds also), in some work over the last 12-18 months.
This is just a mention, for your awareness, not a request for you to do
anything. I hope to find some time to spend on Royale work in the 2nd half
of April, so I will try to look into it then.

---------- Forwarded message ---------
From: <jo...@apache.org>
Date: Thu, Apr 6, 2023 at 8:31 AM
Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
reading plain text files for [Embed] metadata, read as UTF-8 instead of
system default
To: commits@royale.apache.org <co...@royale.apache.org>


This is an automated email from the ASF dual-hosted git repository.

joshtynjala pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/royale-compiler.git


The following commit(s) were added to refs/heads/develop by this push:
     new 5669b4714 JSRoyaleEmitter: when reading plain text files for
[Embed] metadata, read as UTF-8 instead of system default
5669b4714 is described below

commit 5669b4714118c1e2c2edf0e01360a39002e78033
Author: Josh Tynjala <jo...@apache.org>
AuthorDate: Wed Apr 5 13:31:34 2023 -0700

    JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
read as UTF-8 instead of system default

    This matches the behavior of reading other text files, and avoids the
need for JAVA_TOOL_OPTIONS with file.encoding
---
 .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java     |
2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
index 10bc04b4e..2f67e14f5 100644
---
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
+++
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
@@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
implements IJSRoyaleEmitter
             File file = new File(FilenameNormalization.normalize(source));
                try {
                String newlineReplacement = "\\\\n";
-                               String s = FileUtils.readFileToString(file);
+                               String s = FileUtils.readFileToString(file,
"UTF-8");
                    s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
                    s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
                    s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Josh Tynjala <jo...@bowlerhat.dev>.
I ended up skipping the compiler option. Instead, [Embed] metadata with
mimeType="text/plain" can now optionally specify a charset attribute.

[Embed(source="file.txt",mimeType="text/plain",charset="CP1250")]
public var file:String;

To me, this makes more sense than a compiler option, since non-utf8
encoding should be rare. The vast majority of projects will be more likely
to have almost all embedded text files encoded as utf8, and only a small
number in a different encoding, for special circumstances.

--
Josh Tynjala
Bowler Hat LLC <https://bowlerhat.dev>


On Wed, Apr 5, 2023 at 4:06 PM Josh Tynjala <jo...@bowlerhat.dev>
wrote:

> Maybe a new compiler option that will let folks choose an encoding would
> be best. Default to a sane UTF-8, and let people opt into playing with
> weird encodings, if they need it.
>
> --
> Josh Tynjala
> Bowler Hat LLC <https://bowlerhat.dev>
>
>
> On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <ah...@adobe.com.invalid>
> wrote:
>
>> There is a chance this will break someone on Windows where the file
>> system is not UTF8 and the data includes Windows-encoded characters that
>> reference a file or something like that.
>>
>> IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
>> make a change that can't be overridden.
>>
>> -Alex
>>
>> On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <mailto:
>> greg.dove@gmail.com>> wrote:
>>
>>
>> EXTERNAL: Use caution when clicking on links or opening attachments.
>>
>>
>>
>>
>> Nice find, Josh.
>>
>>
>> I was intending to look into some similar issue with resource*
>> .properties *files
>> as well.
>> We have definitely had the need to set utf8 file.encoding in
>> JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
>> royale compiler (JS in particular, but I will try to check if royale swf
>> is
>> at odds also), in some work over the last 12-18 months.
>> This is just a mention, for your awareness, not a request for you to do
>> anything. I hope to find some time to spend on Royale work in the 2nd half
>> of April, so I will try to look into it then.
>>
>>
>> ---------- Forwarded message ---------
>> From: <joshtynjala@apache.org <ma...@apache.org>>
>> Date: Thu, Apr 6, 2023 at 8:31 AM
>> Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
>> reading plain text files for [Embed] metadata, read as UTF-8 instead of
>> system default
>> To: commits@royale.apache.org <ma...@royale.apache.org> <
>> commits@royale.apache.org <ma...@royale.apache.org>>
>>
>>
>>
>>
>> This is an automated email from the ASF dual-hosted git repository.
>>
>>
>> joshtynjala pushed a commit to branch develop
>> in repository https://gitbox.apache.org/repos/asf/royale-compiler.git <
>> https://gitbox.apache.org/repos/asf/royale-compiler.git>
>>
>>
>>
>>
>> The following commit(s) were added to refs/heads/develop by this push:
>> new 5669b4714 JSRoyaleEmitter: when reading plain text files for
>> [Embed] metadata, read as UTF-8 instead of system default
>> 5669b4714 is described below
>>
>>
>> commit 5669b4714118c1e2c2edf0e01360a39002e78033
>> Author: Josh Tynjala <joshtynjala@apache.org <mailto:
>> joshtynjala@apache.org>>
>> AuthorDate: Wed Apr 5 13:31:34 2023 -0700
>>
>>
>> JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
>> read as UTF-8 instead of system default
>>
>>
>> This matches the behavior of reading other text files, and avoids the
>> need for JAVA_TOOL_OPTIONS with file.encoding
>> ---
>> .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
>> 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>>
>> diff --git
>>
>> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>>
>> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>> index 10bc04b4e..2f67e14f5 100644
>> ---
>>
>> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>> +++
>>
>> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>> @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
>> implements IJSRoyaleEmitter
>> File file = new File(FilenameNormalization.normalize(source));
>> try {
>> String newlineReplacement = "\\\\n";
>> - String s = FileUtils.readFileToString(file);
>> + String s = FileUtils.readFileToString(file,
>> "UTF-8");
>> s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
>> s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
>> s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
>>
>>
>>
>>

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Greg Dove <gr...@gmail.com>.
Pretty sure that if you compiled the same MXML file with Flex and Royale,
there's no chance the SWF will be byte-for-byte the same.  You have a
better chance with an AS-only project, but I think there were different AST
and output strategies in each compiler.
-> yeah, understood. This is not at that level at all. It's only about the
bytes representing the strings and the resulting runtime string value. I
think it could simply be checked by outputting the character code sequence
from a swf generated by the Flex compiler swf version and using that for
matching the character code sequence of the same string in the Royale
compiler version. But I am not sure it will be needed to do all this if we
simply have the config setting that Josh is suggesting.



On Thu, Apr 6, 2023 at 2:35 PM Alex Harui <ah...@adobe.com.invalid> wrote:

> Pretty sure that if you compiled the same MXML file with Flex and Royale,
> there's no chance the SWF will be byte-for-byte the same.  You have a
> better chance with an AS-only project, but I think there were different AST
> and output strategies in each compiler.
>
> Of course, I could be wrong,
> -Alex
>
> On 4/5/23, 7:21 PM, "Greg Dove" <greg.dove@gmail.com <mailto:
> greg.dove@gmail.com>> wrote:
>
>
> EXTERNAL: Use caution when clicking on links or opening attachments.
>
>
>
>
> @Josh:
> That sounds good to me, and makes things easier compared to what I was
> considering.
>
>
>
>
> @Alex:
> I was going to compare the behavior of the most recent Flex compiler and
> the Royale compiler to understand/avoid issues, but obviously only with swf
> vs. swf.
> Then check for any difference in royale compiler swf vs. js.
>
>
> For Resource properties files I had only found one very. old 'known issue'
> mention from the Flex 2 compiler which said:
>
>
> - All strings in properties files must be Latin-1 or UTF-8 encoded.
>
>
> I don't have experience using anything other than utf-8 throughout the Flex
> years in these cases. The Flex 2 issue comment above seems to be quite
> restrictive for properties files. However that may have been 'fixed' in
> Flex 3 or Flex 4 compilers to allow more.
> I do suspect that even in more recent compilers it ought to have been utf-8
> by default though, I believe that is what we were seeing for the migration
> project I was involved with, where the utf-8 encoding setting was needed on
> the Royale compiler but not on the old Flex one.. My intention however was
> to try to verify that before making any changes. If we do what Josh is
> suggesting then I would skip that work and just go with the explicit (but
> default Utf8) config setting.
>
>
>
>
>
>
> On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala <joshtynjala@bowlerhat.dev
> <ma...@bowlerhat.dev>>
> wrote:
>
>
> > Maybe a new compiler option that will let folks choose an encoding would
> be
> > best. Default to a sane UTF-8, and let people opt into playing with weird
> > encodings, if they need it.
> >
> > --
> > Josh Tynjala
> > Bowler Hat LLC <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbowlerhat.dev%2F&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G81YOIv0fhkDO29mlR%2FZbNqcpy38abo61LK18azzooA%3D&reserved=0>
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbowlerhat.dev%2F&amp;data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=G81YOIv0fhkDO29mlR%2FZbNqcpy38abo61LK18azzooA%3D&amp;reserved=0&gt
> ;>
> >
> >
> > On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <aharui@adobe.com.inva
> <ma...@adobe.com.inva>lid>
> > wrote:
> >
> > > There is a chance this will break someone on Windows where the file
> > system
> > > is not UTF8 and the data includes Windows-encoded characters that
> > reference
> > > a file or something like that.
> > >
> > > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
> > > make a change that can't be overridden.
> > >
> > > -Alex
> > >
> > > On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <mailto:
> greg.dove@gmail.com> <mailto:
> > > greg.dove@gmail.com <ma...@gmail.com>>> wrote:
> > >
> > >
> > > EXTERNAL: Use caution when clicking on links or opening attachments.
> > >
> > >
> > >
> > >
> > > Nice find, Josh.
> > >
> > >
> > > I was intending to look into some similar issue with resource*
> > > .properties *files
> > > as well.
> > > We have definitely had the need to set utf8 file.encoding in
> > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
> > > royale compiler (JS in particular, but I will try to check if royale
> swf
> > is
> > > at odds also), in some work over the last 12-18 months.
> > > This is just a mention, for your awareness, not a request for you to do
> > > anything. I hope to find some time to spend on Royale work in the 2nd
> > half
> > > of April, so I will try to look into it then.
> > >
> > >
> > > ---------- Forwarded message ---------
> > > From: <joshtynjala@apache.org <ma...@apache.org> <mailto:
> joshtynjala@apache.org <ma...@apache.org>>>
> > > Date: Thu, Apr 6, 2023 at 8:31 AM
> > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter:
> when
> > > reading plain text files for [Embed] metadata, read as UTF-8 instead of
> > > system default
> > > To: commits@royale.apache.org <ma...@royale.apache.org>
> <mailto:commits@royale.apache.org <ma...@royale.apache.org>> <
> > > commits@royale.apache.org <ma...@royale.apache.org> <mailto:
> commits@royale.apache.org <ma...@royale.apache.org>>>
> > >
> > >
> > >
> > >
> > > This is an automated email from the ASF dual-hosted git repository.
> > >
> > >
> > > joshtynjala pushed a commit to branch develop
> > > in repository
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&amp;data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&amp;reserved=0>
> <
> > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0>
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&amp;data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&amp;reserved=0&gt
> ;>
> > >
> > >
> > >
> > >
> > > The following commit(s) were added to refs/heads/develop by this push:
> > > new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> > > [Embed] metadata, read as UTF-8 instead of system default
> > > 5669b4714 is described below
> > >
> > >
> > > commit 5669b4714118c1e2c2edf0e01360a39002e78033
> > > Author: Josh Tynjala <joshtynjala@apache.org <mailto:
> joshtynjala@apache.org> <mailto:
> > > joshtynjala@apache.org <ma...@apache.org>>>
> > > AuthorDate: Wed Apr 5 13:31:34 2023 -0700
> > >
> > >
> > > JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> > > read as UTF-8 instead of system default
> > >
> > >
> > > This matches the behavior of reading other text files, and avoids the
> > > need for JAVA_TOOL_OPTIONS with file.encoding
> > > ---
> > > .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> > > 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > >
> > > diff --git
> > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > index 10bc04b4e..2f67e14f5 100644
> > > ---
> > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > +++
> > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends
> JSGoogEmitter
> > > implements IJSRoyaleEmitter
> > > File file = new File(FilenameNormalization.normalize(source));
> > > try {
> > > String newlineReplacement = "\\\\n";
> > > - String s = FileUtils.readFileToString(file);
> > > + String s = FileUtils.readFileToString(file,
> > > "UTF-8");
> > > s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> > > s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> > > s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
> > >
> > >
> > >
> > >
> >
>
>
>
>

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Alex Harui <ah...@adobe.com.INVALID>.
Pretty sure that if you compiled the same MXML file with Flex and Royale, there's no chance the SWF will be byte-for-byte the same.  You have a better chance with an AS-only project, but I think there were different AST and output strategies in each compiler.

Of course, I could be wrong,
-Alex

On 4/5/23, 7:21 PM, "Greg Dove" <greg.dove@gmail.com <ma...@gmail.com>> wrote:


EXTERNAL: Use caution when clicking on links or opening attachments.




@Josh:
That sounds good to me, and makes things easier compared to what I was
considering.




@Alex:
I was going to compare the behavior of the most recent Flex compiler and
the Royale compiler to understand/avoid issues, but obviously only with swf
vs. swf.
Then check for any difference in royale compiler swf vs. js.


For Resource properties files I had only found one very. old 'known issue'
mention from the Flex 2 compiler which said:


- All strings in properties files must be Latin-1 or UTF-8 encoded.


I don't have experience using anything other than utf-8 throughout the Flex
years in these cases. The Flex 2 issue comment above seems to be quite
restrictive for properties files. However that may have been 'fixed' in
Flex 3 or Flex 4 compilers to allow more.
I do suspect that even in more recent compilers it ought to have been utf-8
by default though, I believe that is what we were seeing for the migration
project I was involved with, where the utf-8 encoding setting was needed on
the Royale compiler but not on the old Flex one.. My intention however was
to try to verify that before making any changes. If we do what Josh is
suggesting then I would skip that work and just go with the explicit (but
default Utf8) config setting.






On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala <joshtynjala@bowlerhat.dev <ma...@bowlerhat.dev>>
wrote:


> Maybe a new compiler option that will let folks choose an encoding would be
> best. Default to a sane UTF-8, and let people opt into playing with weird
> encodings, if they need it.
>
> --
> Josh Tynjala
> Bowler Hat LLC <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbowlerhat.dev%2F&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G81YOIv0fhkDO29mlR%2FZbNqcpy38abo61LK18azzooA%3D&reserved=0> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbowlerhat.dev%2F&amp;data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=G81YOIv0fhkDO29mlR%2FZbNqcpy38abo61LK18azzooA%3D&amp;reserved=0&gt;>
>
>
> On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <aharui@adobe.com.inva <ma...@adobe.com.inva>lid>
> wrote:
>
> > There is a chance this will break someone on Windows where the file
> system
> > is not UTF8 and the data includes Windows-encoded characters that
> reference
> > a file or something like that.
> >
> > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
> > make a change that can't be overridden.
> >
> > -Alex
> >
> > On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <ma...@gmail.com> <mailto:
> > greg.dove@gmail.com <ma...@gmail.com>>> wrote:
> >
> >
> > EXTERNAL: Use caution when clicking on links or opening attachments.
> >
> >
> >
> >
> > Nice find, Josh.
> >
> >
> > I was intending to look into some similar issue with resource*
> > .properties *files
> > as well.
> > We have definitely had the need to set utf8 file.encoding in
> > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
> > royale compiler (JS in particular, but I will try to check if royale swf
> is
> > at odds also), in some work over the last 12-18 months.
> > This is just a mention, for your awareness, not a request for you to do
> > anything. I hope to find some time to spend on Royale work in the 2nd
> half
> > of April, so I will try to look into it then.
> >
> >
> > ---------- Forwarded message ---------
> > From: <joshtynjala@apache.org <ma...@apache.org> <mailto:joshtynjala@apache.org <ma...@apache.org>>>
> > Date: Thu, Apr 6, 2023 at 8:31 AM
> > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
> > reading plain text files for [Embed] metadata, read as UTF-8 instead of
> > system default
> > To: commits@royale.apache.org <ma...@royale.apache.org> <mailto:commits@royale.apache.org <ma...@royale.apache.org>> <
> > commits@royale.apache.org <ma...@royale.apache.org> <mailto:commits@royale.apache.org <ma...@royale.apache.org>>>
> >
> >
> >
> >
> > This is an automated email from the ASF dual-hosted git repository.
> >
> >
> > joshtynjala pushed a commit to branch develop
> > in repository https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0 <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&amp;data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&amp;reserved=0> <
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&reserved=0> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Froyale-compiler.git&amp;data=05%7C01%7Caharui%40adobe.com%7Cadd14bc9c46d4e843d3308db3645ae5d%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C638163445130547736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=v%2FPjMQ6GCV9gMZ9Y8vsQ%2BKrB5mGoff7hd7%2B27WBNED4%3D&amp;reserved=0&gt;>
> >
> >
> >
> >
> > The following commit(s) were added to refs/heads/develop by this push:
> > new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> > [Embed] metadata, read as UTF-8 instead of system default
> > 5669b4714 is described below
> >
> >
> > commit 5669b4714118c1e2c2edf0e01360a39002e78033
> > Author: Josh Tynjala <joshtynjala@apache.org <ma...@apache.org> <mailto:
> > joshtynjala@apache.org <ma...@apache.org>>>
> > AuthorDate: Wed Apr 5 13:31:34 2023 -0700
> >
> >
> > JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> > read as UTF-8 instead of system default
> >
> >
> > This matches the behavior of reading other text files, and avoids the
> > need for JAVA_TOOL_OPTIONS with file.encoding
> > ---
> > .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> > 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> >
> > diff --git
> >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > index 10bc04b4e..2f67e14f5 100644
> > ---
> >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > +++
> >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
> > implements IJSRoyaleEmitter
> > File file = new File(FilenameNormalization.normalize(source));
> > try {
> > String newlineReplacement = "\\\\n";
> > - String s = FileUtils.readFileToString(file);
> > + String s = FileUtils.readFileToString(file,
> > "UTF-8");
> > s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> > s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> > s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
> >
> >
> >
> >
>

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Greg Dove <gr...@gmail.com>.
Oh, that makes it clear that UTF-8 is expected, and would presumably also
account for utf-8 working by default (i.e without any configuration) in
Flex compiler, in terms of aiming for consistency.
OK, I will look into that next week some time, to see if I can address that
too.

Thanks,
-Greg


On Fri, Apr 7, 2023 at 7:50 AM Josh Tynjala <jo...@bowlerhat.dev>
wrote:

> According to the compiler docs, .properties files are assumed to be UTF-8.
>
>
> https://github.com/apache/royale-compiler/blob/apache-royale-0.9.9/compiler/src/main/java/org/apache/royale/compiler/internal/resourcebundles/PropertiesFileParser.java#L98-L100
>
> --
> Josh Tynjala
> Bowler Hat LLC <https://bowlerhat.dev>
>
>
> On Wed, Apr 5, 2023 at 7:21 PM Greg Dove <gr...@gmail.com> wrote:
>
> > @Josh:
> > That sounds good to me, and makes things easier compared to what I was
> > considering.
> >
> >
> > @Alex:
> > I was going to compare the behavior of the most recent Flex compiler and
> > the Royale compiler to understand/avoid issues, but obviously only with
> swf
> > vs. swf.
> > Then check for any difference in royale compiler swf vs. js.
> >
> > For Resource properties files I had only found one very. old 'known
> issue'
> > mention from the Flex 2 compiler which said:
> >
> >    - All strings in properties files must be Latin-1 or UTF-8 encoded.
> >
> > I don't have experience using anything other than utf-8 throughout the
> Flex
> > years in these cases. The Flex 2 issue comment above seems to be quite
> > restrictive for properties files. However that may have been 'fixed' in
> > Flex 3 or Flex 4 compilers to allow more.
> > I do suspect that even in more recent compilers it ought to have been
> utf-8
> > by default though, I believe that is what we were seeing for the
> migration
> > project I was involved with, where the utf-8 encoding setting was needed
> on
> > the Royale compiler but not on the old Flex one.. My intention however
> was
> > to try to verify that before making any changes. If we do what Josh is
> > suggesting then I would skip that work and just go with the explicit (but
> > default Utf8) config setting.
> >
> >
> >
> > On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala <jo...@bowlerhat.dev>
> > wrote:
> >
> > > Maybe a new compiler option that will let folks choose an encoding
> would
> > be
> > > best. Default to a sane UTF-8, and let people opt into playing with
> weird
> > > encodings, if they need it.
> > >
> > > --
> > > Josh Tynjala
> > > Bowler Hat LLC <https://bowlerhat.dev>
> > >
> > >
> > > On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <ah...@adobe.com.invalid>
> > > wrote:
> > >
> > > > There is a chance this will break someone on Windows where the file
> > > system
> > > > is not UTF8 and the data includes Windows-encoded characters that
> > > reference
> > > > a file or something like that.
> > > >
> > > > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we
> don't
> > > > make a change that can't be overridden.
> > > >
> > > > -Alex
> > > >
> > > > On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <mailto:
> > > > greg.dove@gmail.com>> wrote:
> > > >
> > > >
> > > > EXTERNAL: Use caution when clicking on links or opening attachments.
> > > >
> > > >
> > > >
> > > >
> > > > Nice find, Josh.
> > > >
> > > >
> > > > I was intending to look into some similar issue with resource*
> > > > .properties *files
> > > > as well.
> > > > We have definitely had the need to set utf8 file.encoding in
> > > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did
> for
> > > > royale compiler (JS in particular, but I will try to check if royale
> > swf
> > > is
> > > > at odds also), in some work over the last 12-18 months.
> > > > This is just a mention, for your awareness, not a request for you to
> do
> > > > anything. I hope to find some time to spend on Royale work in the 2nd
> > > half
> > > > of April, so I will try to look into it then.
> > > >
> > > >
> > > > ---------- Forwarded message ---------
> > > > From: <joshtynjala@apache.org <ma...@apache.org>>
> > > > Date: Thu, Apr 6, 2023 at 8:31 AM
> > > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter:
> > when
> > > > reading plain text files for [Embed] metadata, read as UTF-8 instead
> of
> > > > system default
> > > > To: commits@royale.apache.org <ma...@royale.apache.org> <
> > > > commits@royale.apache.org <ma...@royale.apache.org>>
> > > >
> > > >
> > > >
> > > >
> > > > This is an automated email from the ASF dual-hosted git repository.
> > > >
> > > >
> > > > joshtynjala pushed a commit to branch develop
> > > > in repository
> https://gitbox.apache.org/repos/asf/royale-compiler.git
> > <
> > > > https://gitbox.apache.org/repos/asf/royale-compiler.git>
> > > >
> > > >
> > > >
> > > >
> > > > The following commit(s) were added to refs/heads/develop by this
> push:
> > > > new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> > > > [Embed] metadata, read as UTF-8 instead of system default
> > > > 5669b4714 is described below
> > > >
> > > >
> > > > commit 5669b4714118c1e2c2edf0e01360a39002e78033
> > > > Author: Josh Tynjala <joshtynjala@apache.org <mailto:
> > > > joshtynjala@apache.org>>
> > > > AuthorDate: Wed Apr 5 13:31:34 2023 -0700
> > > >
> > > >
> > > > JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> > > > read as UTF-8 instead of system default
> > > >
> > > >
> > > > This matches the behavior of reading other text files, and avoids the
> > > > need for JAVA_TOOL_OPTIONS with file.encoding
> > > > ---
> > > > .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> > > > 2 +-
> > > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > >
> > > > diff --git
> > > >
> > > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > >
> > > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > > index 10bc04b4e..2f67e14f5 100644
> > > > ---
> > > >
> > > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > > +++
> > > >
> > > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > > @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends
> > JSGoogEmitter
> > > > implements IJSRoyaleEmitter
> > > > File file = new File(FilenameNormalization.normalize(source));
> > > > try {
> > > > String newlineReplacement = "\\\\n";
> > > > - String s = FileUtils.readFileToString(file);
> > > > + String s = FileUtils.readFileToString(file,
> > > > "UTF-8");
> > > > s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> > > > s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> > > > s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Josh Tynjala <jo...@bowlerhat.dev>.
According to the compiler docs, .properties files are assumed to be UTF-8.

https://github.com/apache/royale-compiler/blob/apache-royale-0.9.9/compiler/src/main/java/org/apache/royale/compiler/internal/resourcebundles/PropertiesFileParser.java#L98-L100

--
Josh Tynjala
Bowler Hat LLC <https://bowlerhat.dev>


On Wed, Apr 5, 2023 at 7:21 PM Greg Dove <gr...@gmail.com> wrote:

> @Josh:
> That sounds good to me, and makes things easier compared to what I was
> considering.
>
>
> @Alex:
> I was going to compare the behavior of the most recent Flex compiler and
> the Royale compiler to understand/avoid issues, but obviously only with swf
> vs. swf.
> Then check for any difference in royale compiler swf vs. js.
>
> For Resource properties files I had only found one very. old 'known issue'
> mention from the Flex 2 compiler which said:
>
>    - All strings in properties files must be Latin-1 or UTF-8 encoded.
>
> I don't have experience using anything other than utf-8 throughout the Flex
> years in these cases. The Flex 2 issue comment above seems to be quite
> restrictive for properties files. However that may have been 'fixed' in
> Flex 3 or Flex 4 compilers to allow more.
> I do suspect that even in more recent compilers it ought to have been utf-8
> by default though, I believe that is what we were seeing for the migration
> project I was involved with, where the utf-8 encoding setting was needed on
> the Royale compiler but not on the old Flex one.. My intention however was
> to try to verify that before making any changes. If we do what Josh is
> suggesting then I would skip that work and just go with the explicit (but
> default Utf8) config setting.
>
>
>
> On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala <jo...@bowlerhat.dev>
> wrote:
>
> > Maybe a new compiler option that will let folks choose an encoding would
> be
> > best. Default to a sane UTF-8, and let people opt into playing with weird
> > encodings, if they need it.
> >
> > --
> > Josh Tynjala
> > Bowler Hat LLC <https://bowlerhat.dev>
> >
> >
> > On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <ah...@adobe.com.invalid>
> > wrote:
> >
> > > There is a chance this will break someone on Windows where the file
> > system
> > > is not UTF8 and the data includes Windows-encoded characters that
> > reference
> > > a file or something like that.
> > >
> > > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
> > > make a change that can't be overridden.
> > >
> > > -Alex
> > >
> > > On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <mailto:
> > > greg.dove@gmail.com>> wrote:
> > >
> > >
> > > EXTERNAL: Use caution when clicking on links or opening attachments.
> > >
> > >
> > >
> > >
> > > Nice find, Josh.
> > >
> > >
> > > I was intending to look into some similar issue with resource*
> > > .properties *files
> > > as well.
> > > We have definitely had the need to set utf8 file.encoding in
> > > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
> > > royale compiler (JS in particular, but I will try to check if royale
> swf
> > is
> > > at odds also), in some work over the last 12-18 months.
> > > This is just a mention, for your awareness, not a request for you to do
> > > anything. I hope to find some time to spend on Royale work in the 2nd
> > half
> > > of April, so I will try to look into it then.
> > >
> > >
> > > ---------- Forwarded message ---------
> > > From: <joshtynjala@apache.org <ma...@apache.org>>
> > > Date: Thu, Apr 6, 2023 at 8:31 AM
> > > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter:
> when
> > > reading plain text files for [Embed] metadata, read as UTF-8 instead of
> > > system default
> > > To: commits@royale.apache.org <ma...@royale.apache.org> <
> > > commits@royale.apache.org <ma...@royale.apache.org>>
> > >
> > >
> > >
> > >
> > > This is an automated email from the ASF dual-hosted git repository.
> > >
> > >
> > > joshtynjala pushed a commit to branch develop
> > > in repository https://gitbox.apache.org/repos/asf/royale-compiler.git
> <
> > > https://gitbox.apache.org/repos/asf/royale-compiler.git>
> > >
> > >
> > >
> > >
> > > The following commit(s) were added to refs/heads/develop by this push:
> > > new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> > > [Embed] metadata, read as UTF-8 instead of system default
> > > 5669b4714 is described below
> > >
> > >
> > > commit 5669b4714118c1e2c2edf0e01360a39002e78033
> > > Author: Josh Tynjala <joshtynjala@apache.org <mailto:
> > > joshtynjala@apache.org>>
> > > AuthorDate: Wed Apr 5 13:31:34 2023 -0700
> > >
> > >
> > > JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> > > read as UTF-8 instead of system default
> > >
> > >
> > > This matches the behavior of reading other text files, and avoids the
> > > need for JAVA_TOOL_OPTIONS with file.encoding
> > > ---
> > > .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> > > 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > >
> > > diff --git
> > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > index 10bc04b4e..2f67e14f5 100644
> > > ---
> > >
> > >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > +++
> > >
> > >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > > @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends
> JSGoogEmitter
> > > implements IJSRoyaleEmitter
> > > File file = new File(FilenameNormalization.normalize(source));
> > > try {
> > > String newlineReplacement = "\\\\n";
> > > - String s = FileUtils.readFileToString(file);
> > > + String s = FileUtils.readFileToString(file,
> > > "UTF-8");
> > > s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> > > s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> > > s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
> > >
> > >
> > >
> > >
> >
>

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Greg Dove <gr...@gmail.com>.
@Josh:
That sounds good to me, and makes things easier compared to what I was
considering.


@Alex:
I was going to compare the behavior of the most recent Flex compiler and
the Royale compiler to understand/avoid issues, but obviously only with swf
vs. swf.
Then check for any difference in royale compiler swf vs. js.

For Resource properties files I had only found one very. old 'known issue'
mention from the Flex 2 compiler which said:

   - All strings in properties files must be Latin-1 or UTF-8 encoded.

I don't have experience using anything other than utf-8 throughout the Flex
years in these cases. The Flex 2 issue comment above seems to be quite
restrictive for properties files. However that may have been 'fixed' in
Flex 3 or Flex 4 compilers to allow more.
I do suspect that even in more recent compilers it ought to have been utf-8
by default though, I believe that is what we were seeing for the migration
project I was involved with, where the utf-8 encoding setting was needed on
the Royale compiler but not on the old Flex one.. My intention however was
to try to verify that before making any changes. If we do what Josh is
suggesting then I would skip that work and just go with the explicit (but
default Utf8) config setting.



On Thu, Apr 6, 2023 at 11:07 AM Josh Tynjala <jo...@bowlerhat.dev>
wrote:

> Maybe a new compiler option that will let folks choose an encoding would be
> best. Default to a sane UTF-8, and let people opt into playing with weird
> encodings, if they need it.
>
> --
> Josh Tynjala
> Bowler Hat LLC <https://bowlerhat.dev>
>
>
> On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <ah...@adobe.com.invalid>
> wrote:
>
> > There is a chance this will break someone on Windows where the file
> system
> > is not UTF8 and the data includes Windows-encoded characters that
> reference
> > a file or something like that.
> >
> > IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
> > make a change that can't be overridden.
> >
> > -Alex
> >
> > On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <mailto:
> > greg.dove@gmail.com>> wrote:
> >
> >
> > EXTERNAL: Use caution when clicking on links or opening attachments.
> >
> >
> >
> >
> > Nice find, Josh.
> >
> >
> > I was intending to look into some similar issue with resource*
> > .properties *files
> > as well.
> > We have definitely had the need to set utf8 file.encoding in
> > JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
> > royale compiler (JS in particular, but I will try to check if royale swf
> is
> > at odds also), in some work over the last 12-18 months.
> > This is just a mention, for your awareness, not a request for you to do
> > anything. I hope to find some time to spend on Royale work in the 2nd
> half
> > of April, so I will try to look into it then.
> >
> >
> > ---------- Forwarded message ---------
> > From: <joshtynjala@apache.org <ma...@apache.org>>
> > Date: Thu, Apr 6, 2023 at 8:31 AM
> > Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
> > reading plain text files for [Embed] metadata, read as UTF-8 instead of
> > system default
> > To: commits@royale.apache.org <ma...@royale.apache.org> <
> > commits@royale.apache.org <ma...@royale.apache.org>>
> >
> >
> >
> >
> > This is an automated email from the ASF dual-hosted git repository.
> >
> >
> > joshtynjala pushed a commit to branch develop
> > in repository https://gitbox.apache.org/repos/asf/royale-compiler.git <
> > https://gitbox.apache.org/repos/asf/royale-compiler.git>
> >
> >
> >
> >
> > The following commit(s) were added to refs/heads/develop by this push:
> > new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> > [Embed] metadata, read as UTF-8 instead of system default
> > 5669b4714 is described below
> >
> >
> > commit 5669b4714118c1e2c2edf0e01360a39002e78033
> > Author: Josh Tynjala <joshtynjala@apache.org <mailto:
> > joshtynjala@apache.org>>
> > AuthorDate: Wed Apr 5 13:31:34 2023 -0700
> >
> >
> > JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> > read as UTF-8 instead of system default
> >
> >
> > This matches the behavior of reading other text files, and avoids the
> > need for JAVA_TOOL_OPTIONS with file.encoding
> > ---
> > .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> > 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> >
> > diff --git
> >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > index 10bc04b4e..2f67e14f5 100644
> > ---
> >
> >
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > +++
> >
> >
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> > @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
> > implements IJSRoyaleEmitter
> > File file = new File(FilenameNormalization.normalize(source));
> > try {
> > String newlineReplacement = "\\\\n";
> > - String s = FileUtils.readFileToString(file);
> > + String s = FileUtils.readFileToString(file,
> > "UTF-8");
> > s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> > s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> > s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
> >
> >
> >
> >
>

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Josh Tynjala <jo...@bowlerhat.dev>.
Maybe a new compiler option that will let folks choose an encoding would be
best. Default to a sane UTF-8, and let people opt into playing with weird
encodings, if they need it.

--
Josh Tynjala
Bowler Hat LLC <https://bowlerhat.dev>


On Wed, Apr 5, 2023 at 3:58 PM Alex Harui <ah...@adobe.com.invalid> wrote:

> There is a chance this will break someone on Windows where the file system
> is not UTF8 and the data includes Windows-encoded characters that reference
> a file or something like that.
>
> IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't
> make a change that can't be overridden.
>
> -Alex
>
> On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <mailto:
> greg.dove@gmail.com>> wrote:
>
>
> EXTERNAL: Use caution when clicking on links or opening attachments.
>
>
>
>
> Nice find, Josh.
>
>
> I was intending to look into some similar issue with resource*
> .properties *files
> as well.
> We have definitely had the need to set utf8 file.encoding in
> JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
> royale compiler (JS in particular, but I will try to check if royale swf is
> at odds also), in some work over the last 12-18 months.
> This is just a mention, for your awareness, not a request for you to do
> anything. I hope to find some time to spend on Royale work in the 2nd half
> of April, so I will try to look into it then.
>
>
> ---------- Forwarded message ---------
> From: <joshtynjala@apache.org <ma...@apache.org>>
> Date: Thu, Apr 6, 2023 at 8:31 AM
> Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
> reading plain text files for [Embed] metadata, read as UTF-8 instead of
> system default
> To: commits@royale.apache.org <ma...@royale.apache.org> <
> commits@royale.apache.org <ma...@royale.apache.org>>
>
>
>
>
> This is an automated email from the ASF dual-hosted git repository.
>
>
> joshtynjala pushed a commit to branch develop
> in repository https://gitbox.apache.org/repos/asf/royale-compiler.git <
> https://gitbox.apache.org/repos/asf/royale-compiler.git>
>
>
>
>
> The following commit(s) were added to refs/heads/develop by this push:
> new 5669b4714 JSRoyaleEmitter: when reading plain text files for
> [Embed] metadata, read as UTF-8 instead of system default
> 5669b4714 is described below
>
>
> commit 5669b4714118c1e2c2edf0e01360a39002e78033
> Author: Josh Tynjala <joshtynjala@apache.org <mailto:
> joshtynjala@apache.org>>
> AuthorDate: Wed Apr 5 13:31:34 2023 -0700
>
>
> JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
> read as UTF-8 instead of system default
>
>
> This matches the behavior of reading other text files, and avoids the
> need for JAVA_TOOL_OPTIONS with file.encoding
> ---
> .../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
> 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>
> diff --git
>
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
>
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> index 10bc04b4e..2f67e14f5 100644
> ---
>
> a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> +++
>
> b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
> @@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
> implements IJSRoyaleEmitter
> File file = new File(FilenameNormalization.normalize(source));
> try {
> String newlineReplacement = "\\\\n";
> - String s = FileUtils.readFileToString(file);
> + String s = FileUtils.readFileToString(file,
> "UTF-8");
> s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
> s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
> s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");
>
>
>
>

Re: [royale-compiler] branch develop updated: JSRoyaleEmitter: when reading plain text files for [Embed] metadata, read as UTF-8 instead of system default

Posted by Alex Harui <ah...@adobe.com.INVALID>.
There is a chance this will break someone on Windows where the file system is not UTF8 and the data includes Windows-encoded characters that reference a file or something like that.

IIRC, that's why I decided to live with JAVA_TOOL_OPTIONS, so we don't make a change that can't be overridden.

-Alex

On 4/5/23, 1:47 PM, "Greg Dove" <greg.dove@gmail.com <ma...@gmail.com>> wrote:


EXTERNAL: Use caution when clicking on links or opening attachments.




Nice find, Josh.


I was intending to look into some similar issue with resource*
.properties *files
as well.
We have definitely had the need to set utf8 file.encoding in
JAVA_TOOL_OPTIONS to get that to work the same as flex compiler did for
royale compiler (JS in particular, but I will try to check if royale swf is
at odds also), in some work over the last 12-18 months.
This is just a mention, for your awareness, not a request for you to do
anything. I hope to find some time to spend on Royale work in the 2nd half
of April, so I will try to look into it then.


---------- Forwarded message ---------
From: <joshtynjala@apache.org <ma...@apache.org>>
Date: Thu, Apr 6, 2023 at 8:31 AM
Subject: [royale-compiler] branch develop updated: JSRoyaleEmitter: when
reading plain text files for [Embed] metadata, read as UTF-8 instead of
system default
To: commits@royale.apache.org <ma...@royale.apache.org> <commits@royale.apache.org <ma...@royale.apache.org>>




This is an automated email from the ASF dual-hosted git repository.


joshtynjala pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/royale-compiler.git <https://gitbox.apache.org/repos/asf/royale-compiler.git>




The following commit(s) were added to refs/heads/develop by this push:
new 5669b4714 JSRoyaleEmitter: when reading plain text files for
[Embed] metadata, read as UTF-8 instead of system default
5669b4714 is described below


commit 5669b4714118c1e2c2edf0e01360a39002e78033
Author: Josh Tynjala <joshtynjala@apache.org <ma...@apache.org>>
AuthorDate: Wed Apr 5 13:31:34 2023 -0700


JSRoyaleEmitter: when reading plain text files for [Embed] metadata,
read as UTF-8 instead of system default


This matches the behavior of reading other text files, and avoids the
need for JAVA_TOOL_OPTIONS with file.encoding
---
.../royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java |
2 +-
1 file changed, 1 insertion(+), 1 deletion(-)


diff --git
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
index 10bc04b4e..2f67e14f5 100644
---
a/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
+++
b/compiler-jx/src/main/java/org/apache/royale/compiler/internal/codegen/js/royale/JSRoyaleEmitter.java
@@ -1097,7 +1097,7 @@ public class JSRoyaleEmitter extends JSGoogEmitter
implements IJSRoyaleEmitter
File file = new File(FilenameNormalization.normalize(source));
try {
String newlineReplacement = "\\\\n";
- String s = FileUtils.readFileToString(file);
+ String s = FileUtils.readFileToString(file,
"UTF-8");
s = s.replaceAll("\n", "__NEWLINE_PLACEHOLDER__");
s = s.replaceAll("\r", "__CR_PLACEHOLDER__");
s = s.replaceAll("\t", "__TAB_PLACEHOLDER__");