You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2008/05/01 22:54:55 UTC
[jira] Created: (PIG-226) Problem with streaming optimization
Problem with streaming optimization
-----------------------------------
Key: PIG-226
URL: https://issues.apache.org/jira/browse/PIG-226
Project: Pig
Issue Type: Bug
Reporter: Olga Natkovich
Assignee: Arun C Murthy
Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
at org.apache.pig.Main.main(Main.java:296)
The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-226) Problem with streaming optimization
Posted by "Pi Song (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593715#action_12593715 ]
Pi Song commented on PIG-226:
-----------------------------
Actually we have an interface called "ReversibleLoadStoreFunc" that enforces the load/store operations to be reversible.
Only checking "if (streamStorer instanceof LoadFunc)" is not enough because the load/store operations might not be symmetric.
The ReversibleLoadStoreFunc interface guarantees the property that:-
{noformat}
LoadFunc (StoreFunc(x)) = x
{noformat}
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-226) Problem with streaming optimization
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-226:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
patch committed. thanks Arun
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-226) Problem with streaming optimization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-226:
------------------------------
Attachment: PIG-226_0_20080501.patch
I've fixed the {Load|Store}Optimizer to check if the StreamingCommand's {in|out}Specs implement {Load|Store}Func before casting them. The patch also contains a test case for the same. I have also checked that the test fails without the fix to the {Load|Store}Optimizer.
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-226) Problem with streaming optimization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-226:
------------------------------
Attachment: (was: PIG-227_0_20080501.patch)
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch, PIG-226_1_20080502.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-226) Problem with streaming optimization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-226:
------------------------------
Attachment: PIG-227_0_20080501.patch
Fixed the check to use ReversibleLoadStoreFunc as suggested by Pi...
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch, PIG-227_0_20080501.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-226) Problem with streaming optimization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-226:
------------------------------
Attachment: PIG-226_1_20080502.patch
The 'right' patch ...
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch, PIG-226_1_20080502.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-226) Problem with streaming optimization
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated PIG-226:
------------------------------
Status: Patch Available (was: Open)
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-226) Problem with streaming optimization
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12593808#action_12593808 ]
Olga Natkovich commented on PIG-226:
------------------------------------
Thanks, Pi. I did not realize that the change already ent in. Arun, could you make the change please.
> Problem with streaming optimization
> -----------------------------------
>
> Key: PIG-226
> URL: https://issues.apache.org/jira/browse/PIG-226
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Arun C Murthy
> Attachments: PIG-226_0_20080501.patch
>
>
> Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.
> 2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
> at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
> at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:296)
> The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.
> The logic that needs to be added is if the load/store interface is not implemented then disable optimization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.