You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2008/08/01 23:23:32 UTC

[jira] Created: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Certain parse errors force exit from grunt (interactive mode)
-------------------------------------------------------------

                 Key: PIG-353
                 URL: https://issues.apache.org/jira/browse/PIG-353
             Project: Pig
          Issue Type: Bug
    Affects Versions: types_branch
            Reporter: Olga Natkovich
             Fix For: types_branch


I tried the following load:

A = load 'foo' as (B: bag{T: tuple(I: int)});

and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623423#action_12623423 ] 

Olga Natkovich commented on PIG-353:
------------------------------------

Santhosh, the original problem seems to be gone. But the following case does not work:

B = load 'foo' as (B: bag{(I: int)});

Is this a valid syntax?

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625205#action_12625205 ] 

Olga Natkovich commented on PIG-353:
------------------------------------

this seems like a regression. why did we change that?

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santhosh Srinivasan updated PIG-353:
------------------------------------

    Patch Info: [Patch Available]

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624836#action_12624836 ] 

Santhosh Srinivasan commented on PIG-353:
-----------------------------------------

If colon (:) is used in the schema, the type has to be specified. The following should work (not sure about describe though). I have removed the ':' after the identifier. The other option is to specify the type - pair: tuple, curr: tuple, next: tuple

{code}

A = load 'data' using PigStorage() as (
x: chararray,
y: int,
z: chararray
);

B = group A by ( x );

C = foreach B generate
FOO(A) as pair (
curr ( user, seq, query ),
next ( user, seq, query )
);

describe C;

{code}

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich resolved PIG-353.
--------------------------------

    Resolution: Fixed

Patch committed. Thanks, Santhosh for contributing!

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch, pig_353.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santhosh Srinivasan updated PIG-353:
------------------------------------

    Attachment: bag_schema_bag_constant_as_clause.patch

Patch fixes the following issues:

1. Grunt parser can disambiguate '{' amongst the foreach nesting, bag constant and bag schema declaration

2. The AS clause in conjunction with generate works. Fully qualified schema specified with the AS clause is set correctly.


When a column is aliased in the generate statement, the schema should be fully qualified. The patch failed testSUM1 in TestTypeChecking.java. A column cast to type long was aliased as age. Since the type information for age was not specified, bytearray is inferred as the type. Casts to bytearray are not supported. As a result, the type checker flagged this as an error. In order to fix, the alias had to be fully qualified, i.e., age:long

{code}

//old code
LogicalPlan plan1 = planTester.buildPlan("b = foreach a generate (long)age as age, (int)gpa as gpa;") ;

//modified code
LogicalPlan plan1 = planTester.buildPlan("b = foreach a generate (long)age as age:long, (int)gpa as gpa:int;") ;
{code}

Unit tests that still fail are:

    [junit] Running org.apache.pig.test.TestBuiltin
    [junit] Tests run: 23, Failures: 1, Errors: 2, Time elapsed: 14.297 sec
    [junit] Test org.apache.pig.test.TestBuiltin FAILED

    [junit] Running org.apache.pig.test.TestEvalPipeline
    [junit] Tests run: 9, Failures: 1, Errors: 0, Time elapsed: 145.656 sec
    [junit] Test org.apache.pig.test.TestEvalPipeline FAILED

    [junit] Running org.apache.pig.test.TestFilterOpNumeric
    [junit] Tests run: 8, Failures: 0, Errors: 1, Time elapsed: 57.466 sec
    [junit] Test org.apache.pig.test.TestFilterOpNumeric FAILED

    [junit] Running org.apache.pig.test.TestMRCompiler
    [junit] Tests run: 16, Failures: 1, Errors: 0, Time elapsed: 0.37 sec
    [junit] Test org.apache.pig.test.TestMRCompiler FAILED

    [junit] Running org.apache.pig.test.TestStoreOld
    [junit] Tests run: 3, Failures: 0, Errors: 2, Time elapsed: 40.966 sec
    [junit] Test org.apache.pig.test.TestStoreOld FAILED

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santhosh Srinivasan updated PIG-353:
------------------------------------

    Attachment: pig_353.patch

Removed the restriction of specifying the type information after colon ':' for a bag or a tuple. 

Unit tests that still fail are:

    [junit] Running org.apache.pig.test.TestBuiltin
    [junit] Tests run: 23, Failures: 1, Errors: 2, Time elapsed: 14.824 sec
    [junit] Test org.apache.pig.test.TestBuiltin FAILED

    [junit] Running org.apache.pig.test.TestEvalPipeline
    [junit] Tests run: 9, Failures: 1, Errors: 0, Time elapsed: 150.556 sec
    [junit] Test org.apache.pig.test.TestEvalPipeline FAILED

    [junit] Running org.apache.pig.test.TestFilterOpNumeric
    [junit] Tests run: 8, Failures: 0, Errors: 1, Time elapsed: 57.101 sec
    [junit] Test org.apache.pig.test.TestFilterOpNumeric FAILED

    [junit] Running org.apache.pig.test.TestMRCompiler
    [junit] Tests run: 16, Failures: 1, Errors: 0, Time elapsed: 0.445 sec
    [junit] Test org.apache.pig.test.TestMRCompiler FAILED

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch, pig_353.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623340#action_12623340 ] 

Santhosh Srinivasan commented on PIG-353:
-----------------------------------------

Pig-349 (https://issues.apache.org/jira/browse/PIG-349) is a duplicate of this issue and has been marked so.

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623565#action_12623565 ] 

Santhosh Srinivasan commented on PIG-353:
-----------------------------------------

B = load 'foo' as (B: bag{(I: int)}); is not valid syntax. Each element in the schema has to have a name. The valid syntax would be one of:

B = load 'foo' as (B: bag{t (I: int)});

OR

B = load 'foo' as (B: bag{t: tuple(I: int)});

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich reassigned PIG-353:
----------------------------------

    Assignee: Santhosh Srinivasan

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich updated PIG-353:
-------------------------------

    Status: Open  (was: Patch Available)

seems like the patch still has issues that need to be resolved

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Olga Natkovich updated PIG-353:
-------------------------------

    Status: Patch Available  (was: Open)

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santhosh Srinivasan updated PIG-353:
------------------------------------

    Attachment: bag_schema_in_pigscript_parser.patch

The attached patch addresses the issue of declaring bag schemas using grunt. PigScriptParser.java was modified to have an additional lexical state for schemas. The opening curly brace '{' in the bag schema was being treated as the delimiter of a nested block. With the additional lexical state the schema definition is tokenized appropriately.

Unit test cases that still fail are:

    [junit] Running org.apache.pig.test.TestBuiltin
    [junit] Tests run: 23, Failures: 1, Errors: 2, Time elapsed: 14.305 sec
    [junit] Test org.apache.pig.test.TestBuiltin FAILED

    [junit] Running org.apache.pig.test.TestEvalPipeline
    [junit] Tests run: 9, Failures: 1, Errors: 0, Time elapsed: 195.352 sec
    [junit] Test org.apache.pig.test.TestEvalPipeline FAILED

    [junit] Running org.apache.pig.test.TestFilterOpNumeric
    [junit] Tests run: 8, Failures: 0, Errors: 1, Time elapsed: 57.163 sec
    [junit] Test org.apache.pig.test.TestFilterOpNumeric FAILED

    [junit] Running org.apache.pig.test.TestMRCompiler
    [junit] Tests run: 16, Failures: 1, Errors: 0, Time elapsed: 0.353 sec
    [junit] Test org.apache.pig.test.TestMRCompiler FAILED

    [junit] Running org.apache.pig.test.TestStoreOld
    [junit] Tests run: 3, Failures: 0, Errors: 2, Time elapsed: 39.973 sec
    [junit] Test org.apache.pig.test.TestStoreOld FAILED

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (PIG-353) Certain parse errors force exit from grunt (interactive mode)

Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624475#action_12624475 ] 

Olga Natkovich commented on PIG-353:
------------------------------------

Santhosh, I applied the patch and it made several cases work but the following still gives errors

A = load 'data' using PigStorage() as (
        x: chararray,
        y: int,
        z: chararray
        );

B = group A by ( x );

C = foreach B generate
        FOO(A) as pair: (
                curr: ( user, seq, query ),
                next: ( user, seq, query )
        );

describe C;

> Certain parse errors force exit from grunt (interactive mode)
> -------------------------------------------------------------
>
>                 Key: PIG-353
>                 URL: https://issues.apache.org/jira/browse/PIG-353
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>            Assignee: Santhosh Srinivasan
>             Fix For: types_branch
>
>         Attachments: bag_schema_bag_constant_as_clause.patch, bag_schema_in_pigscript_parser.patch
>
>
> I tried the following load:
> A = load 'foo' as (B: bag{T: tuple(I: int)});
> and it gave me a huge stack and through me out of grunt

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.