You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2012/09/04 20:45:07 UTC

[jira] [Created] (PIG-2906) Commands in Pig scripts shouldn't trigger executeBatch()

Bill Graham created PIG-2906:
--------------------------------

             Summary: Commands in Pig scripts shouldn't trigger executeBatch()
                 Key: PIG-2906
                 URL: https://issues.apache.org/jira/browse/PIG-2906
             Project: Pig
          Issue Type: Bug
            Reporter: Bill Graham


There are a number of commands that can be used in Pig scripts that will trigger an {{executeBatch()}} from {{GruntParser}} during parsing. This can cause multiple runs of identical (non-shared) jobs. Consider this script:

{noformat}
A = ...
B = ...
C = ...

STORE C

rmf 'xyz'

D = JOIN C, A
STORE D
{noformat}

The DAG of jobs produced by A,B,C will run twice due to the {{executeBatch()}} call tied to the {{rmf}} statement. Instead we should insert a marker into the plan for these types of operations and execute them from the {{JobControlCompiler}} at the appropriate time.

A quick look of the code shows the following commands currently trigger {{executeBatch}}:

* cat
* move
* cp
* copyToLocal
* copyFromLocal
* rmf
* fs command
* sh command
* SQL command


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2906) Commands in Pig scripts shouldn't trigger executeBatch()

Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bill Graham updated PIG-2906:
-----------------------------

    Description: 
There are a number of commands that can be used in Pig scripts that will trigger an {{executeBatch()}} from {{GruntParser}} during parsing. This can cause multiple runs of identical (non-shared) jobs. Consider this script:

{noformat}
A = ...
B = ...
C = ...

STORE C

rmf 'xyz'

D = JOIN C, A
STORE D
{noformat}

The DAG of jobs produced by A,B,C will run twice due to the {{executeBatch()}} call tied to the {{rmf}} statement. Instead we should insert a marker into the plan for these types of operations and execute them from the {{JobControlCompiler}} at the appropriate time.

A quick look of the code shows the following commands currently trigger {{executeBatch}}:

* cat
* move
* cp
* copyToLocal
* copyFromLocal
* rmf
* exec
* fs command
* sh command
* SQL command


  was:
There are a number of commands that can be used in Pig scripts that will trigger an {{executeBatch()}} from {{GruntParser}} during parsing. This can cause multiple runs of identical (non-shared) jobs. Consider this script:

{noformat}
A = ...
B = ...
C = ...

STORE C

rmf 'xyz'

D = JOIN C, A
STORE D
{noformat}

The DAG of jobs produced by A,B,C will run twice due to the {{executeBatch()}} call tied to the {{rmf}} statement. Instead we should insert a marker into the plan for these types of operations and execute them from the {{JobControlCompiler}} at the appropriate time.

A quick look of the code shows the following commands currently trigger {{executeBatch}}:

* cat
* move
* cp
* copyToLocal
* copyFromLocal
* rmf
* fs command
* sh command
* SQL command


    
> Commands in Pig scripts shouldn't trigger executeBatch()
> --------------------------------------------------------
>
>                 Key: PIG-2906
>                 URL: https://issues.apache.org/jira/browse/PIG-2906
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Bill Graham
>
> There are a number of commands that can be used in Pig scripts that will trigger an {{executeBatch()}} from {{GruntParser}} during parsing. This can cause multiple runs of identical (non-shared) jobs. Consider this script:
> {noformat}
> A = ...
> B = ...
> C = ...
> STORE C
> rmf 'xyz'
> D = JOIN C, A
> STORE D
> {noformat}
> The DAG of jobs produced by A,B,C will run twice due to the {{executeBatch()}} call tied to the {{rmf}} statement. Instead we should insert a marker into the plan for these types of operations and execute them from the {{JobControlCompiler}} at the appropriate time.
> A quick look of the code shows the following commands currently trigger {{executeBatch}}:
> * cat
> * move
> * cp
> * copyToLocal
> * copyFromLocal
> * rmf
> * exec
> * fs command
> * sh command
> * SQL command

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira