You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "David Ciemiewicz (JIRA)" <ji...@apache.org> on 2008/12/22 19:28:44 UTC
[jira] Created: (PIG-574) run command for grunt
run command for grunt
---------------------
Key: PIG-574
URL: https://issues.apache.org/jira/browse/PIG-574
Project: Pig
Issue Type: New Feature
Components: grunt
Reporter: David Ciemiewicz
Priority: Minor
This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-574) run command for grunt
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672425#action_12672425 ]
Olga Natkovich commented on PIG-574:
------------------------------------
I reviewed the patch and it looks good. I also ran unit tests and they all passed.
In addition, I ran some manual tests and have a couple of comments:
(1) As implemented now, scripts that ran from within grunt would not be able to take advantage of the parameter substitution as it is not available in the interactive mode. I think this is ok for now and we can revisit it later if users ask for it.
(2) When using run, I could integrate commands in my script with the interactive commands in the shell which was really nice; however, the commands from the script did not show in the command history. If it is reasonably easy to integrate, it would be nice to have that functionality.
(3) Very minor thing: after I execute ran command, I see double prompt from grant:
grunt> grunt>
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-574) run command for grunt
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reassigned PIG-574:
------------------------------
Assignee: Olga Natkovich
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Assignee: Olga Natkovich
> Priority: Minor
> Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-574) run command for grunt
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-574:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed. Thanks Gunther for contributing!
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-574) run command for grunt
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12673074#action_12673074 ]
Olga Natkovich commented on PIG-574:
------------------------------------
I tested patch and all is good. I have made one small change - moving parameters in front of the pig script to be consistent with pig cmd syntax. New patch is attached
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-574) run command for grunt
Posted by "Gunther Hagleitner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gunther Hagleitner updated PIG-574:
-----------------------------------
Attachment: run_command.patch
Introduces run and exec command
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-574) run command for grunt
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-574:
-------------------------------
Attachment: PIG-574.patch
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: PIG-574.patch, run_command.patch, run_command_params.patch, run_command_params_021109.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-574) run command for grunt
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672365#action_12672365 ]
Olga Natkovich commented on PIG-574:
------------------------------------
I will be reviewing this patch
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-574) run command for grunt
Posted by "Gunther Hagleitner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gunther Hagleitner updated PIG-574:
-----------------------------------
Status: Patch Available (was: Open)
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-574) run command for grunt
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672699#action_12672699 ]
Olga Natkovich commented on PIG-574:
------------------------------------
Gunther, thanks for quick turnaround on the patch.
Is there a reason why we can't use exactly the same syntax on run/exec command for parameter substituion as we do on Pig command line. I think that might be easier for users to remember and provide a more consistent interface.
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch, run_command_params.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-574) run command for grunt
Posted by "Gunther Hagleitner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gunther Hagleitner updated PIG-574:
-----------------------------------
Attachment: run_command_params_021109.patch
Good point. I felt it was a little strange to specify "-param" on the grunt shell, but it is easier to remember if your using it outside the shell already.
So, this patch does the same as the last one, but the syntax is:
run myscript.pig -param LIMIT=5 -param FILE=/foo/bar.txt -param_file myparams.ppf
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch, run_command_params.patch, run_command_params_021109.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-574) run command for grunt
Posted by "Gunther Hagleitner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gunther Hagleitner updated PIG-574:
-----------------------------------
Attachment: run_command_params.patch
Thanks for reviewing the patch!
I tried to address the 3 issues you pointed out:
1) You can now specify parameters and param files in both the exec and run command
grunt> run myscript.pig using param_file myparams.ppf
or:
grunt> run myscript.pig using param LIMIT=5 param_file myparams.ppf
The syntax mimics what you can do on the command line when executing a script without the "-"s.
2) The script lines are now added to the command history in interactive mode
3) The double grunt... That's actually harder to fix than it thought, but I added a newline, so it won't say:
grunt> grunt>
but:
grunt>
grunt>
Let's just tell everyone that that's because they have extra newlines in their scripts. Maybe they won't find out. ;-)
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch, run_command_params.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-574) run command for grunt
Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672655#action_12672655 ]
David Ciemiewicz commented on PIG-574:
--------------------------------------
Thanks!
This will make so iterative development faster and less painful than preallocating a HOD subcluster and then forgetting to delete it.
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch, run_command_params.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-574) run command for grunt
Posted by "Gunther Hagleitner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672570#action_12672570 ]
Gunther Hagleitner commented on PIG-574:
----------------------------------------
Oh, I also ran the unit tests. They pass.
> run command for grunt
> ---------------------
>
> Key: PIG-574
> URL: https://issues.apache.org/jira/browse/PIG-574
> Project: Pig
> Issue Type: New Feature
> Components: grunt
> Reporter: David Ciemiewicz
> Priority: Minor
> Attachments: run_command.patch, run_command_params.patch
>
>
> This is a request for a "run file" command in grunt which will read a script from the local file system and execute the script interactively while in the grunt shell.
> One of the things that slows down iterative development of large, complicated Pig scripts that must operate on hadoop fs data is that the edit, run, debug cycle is slow because I must wait to allocate a Hadoop-on-Demand (hod) cluster for each iteration. I would prefer not to preallocate a cluster of nodes (though I could).
> Instead, I'd like to have one window open and edit my Pig script using vim or emacs, write it, and then type "run myscript.pig" at the grunt shell until I get things right.
> I'm used to doing similar things with Oracle, MySQL, and R.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.