You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2009/03/02 10:22:12 UTC
[jira] Created: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
-----------------------------------------------------------------------------------------------------
Key: PIG-693
URL: https://issues.apache.org/jira/browse/PIG-693
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Viraj Bhat
Fix For: types_branch
Consider the following Pig Script
{code}
register myudf.jar;
A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
B = foreach A {
dec = myudf.URLDECODE('hello');
str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
generate
dec,
str1,
str2;
};
describe B;
dump B;
{code}
where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
{code}
describe B;
{code}
returns the following:
B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
{code}
dump B;
{code}
returns
(hello,[\u0000-\u0020],hello)
The result should be:
There is a workaround for the same,
{code}
register myudf.jar;
A = load 'one.txt' using PigStorage() as ( one: int );
B = foreach A {
dec = myudf.URLDECODE('hello');
generate
dec as dec,
myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
};
describe B;
dump B;
{code}
where
{code}
dump B;
{code}
returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682344#action_12682344 ]
Thejas M Nair commented on PIG-693:
-----------------------------------
Re: Index: src/org/apache/pig/impl/plan/OperatorPlan.java
I will make the changes you suggested in comments 2,3,4,5 in a new patch.
Thanks for reviewing the code and for the comments.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-693:
---------------------------
Attachment: one.txt
Test input file to start execution
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Fix For: types_branch
>
> Attachments: one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683561#action_12683561 ]
Thejas M Nair commented on PIG-693:
-----------------------------------
Re:
bq. Changing name() will have implications on existing test cases. Its better to use node.getClass().geSimpletName(). Its more user friendly than using the node.toString() which provides details which cannot be understood by users.
I have created a separate JIRA - PIG-726 to make node.toString() more user friendly. After than change, it will be more user friendly than printing class name. I will address that JIRA soon.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681857#action_12681857 ]
Santhosh Srinivasan commented on PIG-693:
-----------------------------------------
I will be reviewing this patch.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-693:
------------------------------
Attachment: 693.2.patch
Updated patch incorporating changes suggested in code review .
OperatorPlan.java
- using Integer for size instead of string
- removed imports of LogicalOperator and subclasses
- moved special handling of LOProject to patchInputReference method in RemoveRedundantOperators.java
- removeAndReconnectMultiSucc() takes into account other successors of A.
TestEvalPipeline2.java
- Using org.apache.pig.test.utils.Idenity instead of IdenityUDF.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.2.patch, 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681960#action_12681960 ]
Thejas M Nair commented on PIG-693:
-----------------------------------
Re: Review comments on 693.patch
Re: Overall comment: The private member variables are not part of the interface, so removing them should be fine. I can add deprecated versions of the constructor and set methods. But is it OK to keep even deprecated versions of them around if we not using the input parameters passed , or in case of set methods they are no-op ?
Re Complete package name used in Index: src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/LogToPhyTranslationVisitor.java ,
There are two ExpressionOperator classes used in there, from - org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.ExpressionOperator; and org.apache.pig.impl.logicalLayer.ExpressionOperator. Either of those have to be used with full package name. The one logicalLayer one is only used once, so I am using full name for it.
Re: Index: src/org/apache/pig/impl/logicalLayer/validators/TypeCheckingVisitor.java ,
If i use node.getClass().getSimpleName(), it will print names like LOOr, LOBinCond which are also not very user friendly. I think we should change the name() function in these classes to not print the scope.
Re: Index: src/org/apache/pig/impl/plan/OperatorPlan.java
1. Not sure if i understood what you meant there . The following code will not work -
Integer i = null;
System.out.println(i.toString());
( I am reviewing rest of the suggestions.)
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681944#action_12681944 ]
Santhosh Srinivasan commented on PIG-693:
-----------------------------------------
Review comments for 693.utest.patch
Index: test/org/apache/pig/test/TestEvalPipeline2.java
==========================================
Use org.apache.pig.test.utils.Idenity instead of creating another IdenityUDF.
{code}
+ static public class IdentityUdf
+ extends EvalFunc<Tuple> {
+
+ // Expects 3 chararrays in input tuple and returns the 3 chararrays in
+ // output tuple
+ @Override
+ public Tuple exec(Tuple input) throws IOException {
+ if (input == null || input.size() == 0)
+ return null;
+ return input ;
+
+ }
+
+ }
{code}
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan reassigned PIG-693:
---------------------------------------
Assignee: Thejas M Nair
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Fix For: types_branch
>
> Attachments: one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-693:
------------------------------
Attachment: 693.utest.patch
Unit test for this bug.
This patch is in addition to the patch file attached earlier.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-693:
---------------------------
Attachment: URLDECODE.java
Eval UDF
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Fix For: types_branch
>
> Attachments: URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683660#action_12683660 ]
Thejas M Nair commented on PIG-693:
-----------------------------------
Created PIG-727 to address exception creation in OperatorPlan.java
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683943#action_12683943 ]
Thejas M Nair commented on PIG-693:
-----------------------------------
Patch file 693.2.patch replaces all earlier patch files.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.2.patch, 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-693:
------------------------------
Patch Info: [Patch Available]
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-693:
---------------------------
Attachment: REPLACEALL.java
EVAL UDF
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Fix For: types_branch
>
> Attachments: one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682340#action_12682340 ]
Santhosh Srinivasan commented on PIG-693:
-----------------------------------------
1. If you deprecate the constructors then you need to re-instate the private member variables. You can ignore this comment considering the fact that all uses of this class are inside Pig for now.
2. Ok. Its fine for now.
3. Changing name() will have implications on existing test cases. Its better to use node.getClass().geSimpletName(). Its more user friendly than using the node.toString() which provides details which cannot be understood by users.
4. Do not use i.toString(). Just use the following:
{code}
Integer i = null;
System.out.println(i);
{code}
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681913#action_12681913 ]
Santhosh Srinivasan commented on PIG-693:
-----------------------------------------
Review comments on 693.patch
Overall comment:
Instead of removing the private member variables and the associated constructors, its better to mark the constructor and the other methods as deprecated. If there are developers who are using these sources then compiler warning messages will help them move the code instead of forcing them to make immediate changes.
Specific comments:
Index: src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/LogToPhyTranslationVisitor.java
=====================================================================================
Is it required to have the complete package name?
{code}
+ List<org.apache.pig.impl.logicalLayer.ExpressionOperator> fromList = func.getArguments();
{code}
Index: src/org/apache/pig/impl/logicalLayer/RemoveRedundantOperators.java
=============================================================
1. Can you add an error code and an appropriate message to the VisitorException ?
{code}
+ throw new VisitorException(pe.getMessage(),pe);
{code}
Index: src/org/apache/pig/impl/logicalLayer/validators/TypeCheckingVisitor.java
===================================================================
1. Instead of node use node.getClass().getSimpleName(). Otherwise, users will see a huge string with contains the operator key and scope. This is useful for developers but not for users.
{code}
+ String msg = "Problem with inserting cast operator for " + node + " in plan.";
{code}
Index: src/org/apache/pig/impl/plan/OperatorPlan.java
===========================================
1. Use Integer instead of string. Integer, Long, etc. can handle null and toString will ensure that null is printed as "null".
{code}
+ String size = " null ";
+ if(preds != null)
+ size = ((Integer)preds.size()).toString();
{code}
2. I see that you are following the convention of the source for throwing exceptions. It will be good if you can include an error code, not log the error and just throw the exception.
{code}
+ PlanException pe = new PlanException("Attempt to remove " +
+ " and reconnect for node with " + size +
+ " predecessors.");
+ log.error(pe.getMessage());
+ throw pe;
{code}
{code}
+ //checking pre-requisite conditions
+ if (C == null || C.size() == 0) {
+ PlanException pe = new PlanException("Attempt to remove " +
+ " and reconnect for node with no successors.");
+ log.error(pe.getMessage());
+ throw pe;
+ }
{code}
3. OperatorPlan is a generic class and should not hold any references to LogicalOperator, ExpressionOperator or LOProject. In addition, the last section of removeAndReconnectMultiSucc checks for LOProject. This should instead go into the patchInputReference method in RemoveRedundantOperators.java.
{code}
+import org.apache.pig.impl.logicalLayer.ExpressionOperator;
+import org.apache.pig.impl.logicalLayer.LOProject;
+import org.apache.pig.impl.logicalLayer.LogicalOperator;
{code}
{code}
+ //special handling of LOProject because its getExpression() does
+ // need not be same as getPredecessors(LOProject)
+ if(c instanceof LOProject){
+ LOProject lop = (LOProject)c;
+ if(B == lop.getExpression()){
+ lop.setExpression((LogicalOperator)B);
+ }
+ }
{code}
4. The removeAndReconnectMultiSucc does not take into account successors of A. If A has other successors apart from B then the following code will remove the edge from A to the other successors.
{code}
+ // Do A.succesors = B.successors(ie C)
+ mFromEdges.removeKey(A);
+ mFromEdges.put(A,C);
{code}
5. Minor comment, the use of variables A, B, C should be changed to nodeA, nodeB, nodeC to distinguish between types (like E) and instances (like A, B, C). Its a little bit confusing.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan resolved PIG-693.
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Patch has been committed. Thanks for the fix Thejas. Good work on cleaning up the type checker.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.2.patch, 693.patch, 693.utest.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-693) Parameter to UDF which is an alias
returned in another UDF in nested foreach causes incorrect results
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-693:
------------------------------
Attachment: 693.patch
- All subclasses of LogicalOperator(LOP) (except LOConst,LOProject) no longer maintain a local reference of their inputs. getOperand()/getLhs()/getArguments() etc will access the predecessor in the graph each time. This will avoid problems caused by predecessor and input arguments reference going out of sync.
- In above classes, the functions for setting the input arguments have been removed. The constructor also no longer takes arguments as input.
- LogToPhyTranslationVisitor - Checking if getPredecessor() is null before calling getOperand().getType(). (Fixes a negative test case failure).
- TypeCheckingVisitor - Insertion of casts in LOP s uses the OperatorPlan.insertBetween() function. Common logic used to insert cast in LOPs have been moved to new insertCast(..) function.
- LOConst does not have above change because its input is contained within the object, not on the graph. LOProject does not change because input need not be predecessor in graph if input is relational operator.
> Parameter to UDF which is an alias returned in another UDF in nested foreach causes incorrect results
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-693
> URL: https://issues.apache.org/jira/browse/PIG-693
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 1.0.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Attachments: 693.patch, one.txt, REPLACEALL.java, URLDECODE.java
>
>
> Consider the following Pig Script
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int ); --use this dummy file to start execution
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> str1 = myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' '); -- ERROR
> str2 = myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ');
> generate
> dec,
> str1,
> str2;
> };
> describe B;
> dump B;
> {code}
> where one.txt is a file which contains number one (1) for starting execution of the Pig script!!
> {code}
> describe B;
> {code}
> returns the following:
> B: {urldecode_9: chararray,replaceall_urldecode_10_11: chararray,replaceall_12: chararray}
> {code}
> dump B;
> {code}
> returns
> (hello,[\u0000-\u0020],hello)
> The result should be:
> There is a workaround for the same,
> {code}
> register myudf.jar;
> A = load 'one.txt' using PigStorage() as ( one: int );
> B = foreach A {
> dec = myudf.URLDECODE('hello');
> generate
> dec as dec,
> myudf.REPLACEALL(dec, '[\\u0000-\\u0020]', ' ') as str1,
> myudf.REPLACEALL('hello', '[\\u0000-\\u0020]', ' ') as str2;
> };
> describe B;
> dump B;
> {code}
> where
> {code}
> dump B;
> {code}
> returns (hello,hello,hello)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.