You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "David Ciemiewicz (JIRA)" <ji...@apache.org> on 2009/04/20 18:29:47 UTC

[jira] Created: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
--------------------------------------------------------------------------------------------------

                 Key: PIG-771
                 URL: https://issues.apache.org/jira/browse/PIG-771
             Project: Pig
          Issue Type: Bug
            Reporter: David Ciemiewicz


PigDump does not properly output Chinese UTF8 characters.

The reason for this is that the function Tuple.toString() is called.

DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.

Instead, I think that the code should be changed instead to call the new DataType.toString() function.

{code}
    @Override
    public String toString() {
        StringBuilder sb = new StringBuilder();
        sb.append('(');
        for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
            Object d = it.next();
            if(d != null) {
                if(d instanceof Map) {
                    sb.append(DataType.mapToString((Map<Object, Object>)d));
                } else {
                    sb.append(DataType.toString(d));  // <<< Change this one line
                    if(d instanceof Long) {
                        sb.append("L");
                    } else if(d instanceof Float) {
                        sb.append("F");
                    }
                }
            } else {
                sb.append("");
            }
            if (it.hasNext())
                sb.append(",");
        }
        sb.append(')');
        return sb.toString();
    }
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703383#action_12703383 ] 

David Ciemiewicz edited comment on PIG-771 at 4/27/09 2:20 PM:
---------------------------------------------------------------

Daniel,

Thanks.  locale reported LANG=POSIX

I used locale -a to list the locales and then did:

export LANG=en_US.utf8

Then I got the correct PigDump output.

I found that also setting LESSCHARSET to utf-8 was valuable as well.

For bash users:

export LANG=en_US.utf8
export LESSCHARSET=utf-8


It would be useful if dump/PigDump() had a warning which indicated to the user if LANG=POSIX, then asian language characters may not display properly.  Something like:

{code}if (function_which_returns_local_setting_which_I_dont_know_name_of().equals("POSIX"))  {
    System.out.println("WARNING: dump will not properly display multibyte UTF-8 characters when\n" + 
    "environment variable LANG=\"POSIX\".  Try setting your environment variable LANG=en_US.utf8.\n"  +
    "See locale -a for other possible values.");
}{code}

      was (Author: ciemo):
    Daniel,

Thanks.  locale reported LANG=POSIX

I used locale -a to list the locales and then did:

export LANG=en_US.utf8

Then I got the correct PigDump output.

I found that also setting LESSCHARSET to utf-8 was valuable as well.

For bash users:

export LANG=en_US.utf8
export LESSCHARSET=utf-8


It would be useful if dump/PigDump() had a warning which indicated to the user if LANG=POSIX, then asian language characters may not display properly.  Something like:

{code}if (function_which_returns_local_setting_which_I_dont_know_name_of().equals("POSIX"))  {
    System.out.println("WARNING: dump will not properly display multibyte UTF-8 characters when\n" + 
    environment variable LANG=\"POSIX\".  Try setting your environment variable LANG=en_US.utf8.\n"  + "See locale -a for other possible values.")
}{code}
  
> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703165#action_12703165 ] 

Daniel Dai commented on PIG-771:
--------------------------------

Seems it works fine on my computer either dump it on the screen or store it into a file. My OS encoding is UTF8. Here is a log:

grunt> daniel@ubuntu-daniel1:~/pig$ cat chinese.txt
中文测试
daniel@ubuntu-daniel1:~/pig$ java -jar pig.jar
2009-04-27 11:14:10,889 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2009-04-27 11:14:11,046 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> A = load '/home/daniel/pig/chinese.txt' using PigStorage() as (str: chararray);
grunt> dump A;
2009-04-27 11:14:18,767 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2009-04-27 11:14:18,774 [Thread-4] WARN  org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2009-04-27 11:14:19,289 [Thread-10] INFO  org.apache.hadoop.mapred.MapTask - numReduceTasks: 0
2009-04-27 11:14:19,364 [Thread-10] INFO  org.apache.hadoop.mapred.LocalJobRunner -
2009-04-27 11:14:19,368 [Thread-10] INFO  org.apache.hadoop.mapred.TaskRunner - Task 'attempt_local_0001_m_000000_0' done.
2009-04-27 11:14:19,372 [Thread-10] INFO  org.apache.hadoop.mapred.TaskRunner - Saved output of task 'attempt_local_0001_m_000000_0' to file:/tmp/temp-480716160/tmp2138119751
2009-04-27 11:14:24,220 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2009-04-27 11:14:24,222 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2009-04-27 11:14:24,222 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
(中文测试)
grunt>


> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703383#action_12703383 ] 

David Ciemiewicz edited comment on PIG-771 at 4/27/09 2:19 PM:
---------------------------------------------------------------

Daniel,

Thanks.  locale reported LANG=POSIX

I used locale -a to list the locales and then did:

export LANG=en_US.utf8

Then I got the correct PigDump output.

I found that also setting LESSCHARSET to utf-8 was valuable as well.

For bash users:

export LANG=en_US.utf8
export LESSCHARSET=utf-8


It would be useful if dump/PigDump() had a warning which indicated to the user if LANG=POSIX, then asian language characters may not display properly.  Something like:

{code}if (function_which_returns_local_setting_which_I_dont_know_name_of().equals("POSIX"))  {
    System.out.println("WARNING: dump will not properly display multibyte UTF-8 characters when\n" + 
    environment variable LANG=\"POSIX\".  Try setting your environment variable LANG=en_US.utf8.\n"  + "See locale -a for other possible values.")
}{code}

      was (Author: ciemo):
    Daniel,

Thanks.  locale reported LANG=POSIX

I used locale -a to list the locales and then did:

export LANG=en_US.utf8

Then I got the correct PigDump output.

I found that also setting LESSCHARSET to utf-8 was valuable as well.

For bash users:

export LANG=en_US.utf8
export LESSCHARSET=utf-8


It would be useful if dump/PigDump() had a warning which indicated to the user if LANG=POSIX, then asian language characters may not display properly.  Something like:

{code}if (function_which_returns_local_setting_which_I_dont_know_name_of().equals("POSIX"))  {
    System.out.println("WARNING: dump will not properly display multibyte UTF-8 characters when environment variable LANG=\"POSIX\".  Try setting your environment variable LANG=en_US.utf8.  See locale -a for other possible values.")
}{code}
  
> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703492#action_12703492 ] 

Daniel Dai commented on PIG-771:
--------------------------------

It is good to put a warning message. Instead of checking if LANG=POSIX, we can check whether LANG=UTF8. We can see correct character only when LANG=UTF8.

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703155#action_12703155 ] 

David Ciemiewicz commented on PIG-771:
--------------------------------------

Take a file of UTF-8 Chinese characters (ch.txt).  Load it and dump it.

{code}A = load 'ch.txt' using PigStorage() as (str: chararray);

dump A;

store A into 'ch.out' using PigStorage();{code}



> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich resolved PIG-771.
--------------------------------

    Fix Version/s: 0.7.0
       Resolution: Fixed

PigDump is no longer supported

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>             Fix For: 0.7.0
>
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700853#action_12700853 ] 

David Ciemiewicz commented on PIG-771:
--------------------------------------

Actually, now that I think about it, there is a bit of inconsistency in the complex data type toString functions:

DefaultTuple.toString()
SingleTupleBag.toString()
DefaultAbstractBag.toString() // uses StringBuffer rather than StringBuilder.  Should it?

Would it not be better to be consistent and use one path for all toString() conversions?

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703293#action_12703293 ] 

Daniel Dai commented on PIG-771:
--------------------------------

David, can you check what is the language setting on that computer? Use the command "locale|grep LANG". Thanks

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Issue Comment Edited: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703383#action_12703383 ] 

David Ciemiewicz edited comment on PIG-771 at 4/27/09 2:18 PM:
---------------------------------------------------------------

Daniel,

Thanks.  locale reported LANG=POSIX

I used locale -a to list the locales and then did:

export LANG=en_US.utf8

Then I got the correct PigDump output.

I found that also setting LESSCHARSET to utf-8 was valuable as well.

For bash users:

export LANG=en_US.utf8
export LESSCHARSET=utf-8


It would be useful if dump/PigDump() had a warning which indicated to the user if LANG=POSIX, then asian language characters may not display properly.  Something like:

{code}if (function_which_returns_local_setting_which_I_dont_know_name_of().equals("POSIX"))  {
    System.out.println("WARNING: dump will not properly display multibyte UTF-8 characters when environment variable LANG=\"POSIX\".  Try setting your environment variable LANG=en_US.utf8.  See locale -a for other possible values.")
}{code}

      was (Author: ciemo):
    Daniel,

Thanks.  locale reported LANG=POSIX

I used locale -a to list the locales and then did:

export LANG=en_US.utf8

Then I got the correct PigDump output.

I found that also setting LESSCHARSET to utf-8 was valuable as well.

For bash users:

export LANG=en_US.utf8
export LESSCHARSET=utf-8


It would be useful if dump/PigDump() had a warning which indicated to the user if LANG=POSIX, then asian language characters may not display properly.  Something like:

if (function_which_returns_local_setting_which_I_dont_know_name_of().equals("POSIX")) {
    System.out.println("WARNING: dump will not properly display multibyte UTF-8 characters when environment variable LANG=\"POSIX\".  Try setting your environment variable LANG=en_US.utf8.  See locale -a for other possible values.")
}
  
> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703010#action_12703010 ] 

Daniel Dai commented on PIG-771:
--------------------------------

Hi, David,
In my mind PigDump should deal with UTF8 characters correctly. Can you describe the situation in which PigDump fail? What is your OS encoding?

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700845#action_12700845 ] 

David Ciemiewicz commented on PIG-771:
--------------------------------------

I was going to submit a patch for this one line change, but I discovered in compiling the code that DataType.toString(d) throws an ExecException.

Oddly, DataType.mapToString DOES NOT throw any Exceptions which is inconsistent with the other DataType.to... functions.

I am not sure how to best implement the try / catch / throw for this particular case.

Also, in doing the code review of DataType.mapToString(...) I discovered that it will also have problems with correctly dumping the data contained within it because it too uses Object.toString() on opaque data handles.

So, the code for DataType.mapToString(...) should also use DataType.toString(Object);

But now I witness a recursion problem.  DataType.toString(Object) does not work for complex types.  So maps of maps will not be recursed properly.

So DataType.toString(Object) should probably be enhanced to work on Maps as well.

But now we have another problem ... PigDump wants to append L and F for Long values and Float values.  But this won't work for nested structures.

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703219#action_12703219 ] 

David Ciemiewicz commented on PIG-771:
--------------------------------------

I'm just using Mac OS terminal to connect to a RHEL-4 gateway server to a RHEL-4 grid.

I changed the code to use PigDump() storage format for the STORE statement and reran the code, trying to eliminate the terminal aspect.  Pig itself is writing the question marks ('?', 0x3f).

{code}
-bash-3.00$ cat ch2.pig
A = load 'ch.txt' using PigStorage() as (str: chararray);
store A into 'ch.dmp' using PigDump();

-bash-3.00$ hadoop fs -cat ch.dmp/*
(????)

-bash-3.00$ hadoop fs -cat ch.dmp/* | od -xc
0000000 3f28 3f3f 293f 000a
          (   ?   ?   ?   ?   )  \n  \0
0000007
{code}

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703195#action_12703195 ] 

David Ciemiewicz commented on PIG-771:
--------------------------------------

Very strange.  I can display UTF8 chinese characters in my Mac OS Terminal window.  Only dump has a problem.

Here's the transcript of what I did.  If you look, you'll see:

{code}
-bash-3.00$ cat > ch.txt
中文测试

-bash-3.00$ file ch.txt
ch.txt: UTF-8 Unicode text

-bash-3.00$ cat ch.txt
中文测试

-bash-3.00$ cat ch.pig
A = load 'ch.txt' using PigStorage() as (str: chararray);
dump A;
store A into 'ch.out' using PigStorage();

-bash-3.00$ pig -exectype local ch.pig
USING: /grid/0/gs/pig/current
2009-04-27 16:15:16,314 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-04-27 16:15:16,315 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(????)
2009-04-27 16:15:16,339 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-04-27 16:15:16,339 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!

-bash-3.00$ cat ch.out
中文测试

-bash-3.00$ pig -exectype local 
USING: /grid/0/gs/pig/current
grunt> A = load 'ch.txt' using PigStorage() as (str: chararray);
grunt> dump A;
2009-04-27 16:16:51,786 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-04-27 16:16:51,786 [main] INFO  org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(????)
grunt> 
{code}

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703205#action_12703205 ] 

Daniel Dai commented on PIG-771:
--------------------------------

I am using Ubuntu. Maybe this problem is on Mac only. I don't know much about Mac, is there a language setting? If it is, can you change it to UTF8? 

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-771) PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??

Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703383#action_12703383 ] 

David Ciemiewicz commented on PIG-771:
--------------------------------------

Daniel,

Thanks.  locale reported LANG=POSIX

I used locale -a to list the locales and then did:

export LANG=en_US.utf8

Then I got the correct PigDump output.

I found that also setting LESSCHARSET to utf-8 was valuable as well.

For bash users:

export LANG=en_US.utf8
export LESSCHARSET=utf-8


It would be useful if dump/PigDump() had a warning which indicated to the user if LANG=POSIX, then asian language characters may not display properly.  Something like:

if (function_which_returns_local_setting_which_I_dont_know_name_of().equals("POSIX")) {
    System.out.println("WARNING: dump will not properly display multibyte UTF-8 characters when environment variable LANG=\"POSIX\".  Try setting your environment variable LANG=en_US.utf8.  See locale -a for other possible values.")
}

> PigDump does not properly output Chinese UTF8 characters - they are displayed as question marks ??
> --------------------------------------------------------------------------------------------------
>
>                 Key: PIG-771
>                 URL: https://issues.apache.org/jira/browse/PIG-771
>             Project: Pig
>          Issue Type: Bug
>            Reporter: David Ciemiewicz
>
> PigDump does not properly output Chinese UTF8 characters.
> The reason for this is that the function Tuple.toString() is called.
> DefaultTuple implements Tuple.toString() and it calls Object.toString() on the opaque object d.
> Instead, I think that the code should be changed instead to call the new DataType.toString() function.
> {code}
>     @Override
>     public String toString() {
>         StringBuilder sb = new StringBuilder();
>         sb.append('(');
>         for (Iterator<Object> it = mFields.iterator(); it.hasNext();) {
>             Object d = it.next();
>             if(d != null) {
>                 if(d instanceof Map) {
>                     sb.append(DataType.mapToString((Map<Object, Object>)d));
>                 } else {
>                     sb.append(DataType.toString(d));  // <<< Change this one line
>                     if(d instanceof Long) {
>                         sb.append("L");
>                     } else if(d instanceof Float) {
>                         sb.append("F");
>                     }
>                 }
>             } else {
>                 sb.append("");
>             }
>             if (it.hasNext())
>                 sb.append(",");
>         }
>         sb.append(')');
>         return sb.toString();
>     }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.