You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Bill Graham (JIRA)" <ji...@apache.org> on 2009/12/28 03:16:33 UTC
[jira] Created: (PIG-1174) Creation of output path should be done
by storage function
Creation of output path should be done by storage function
----------------------------------------------------------
Key: PIG-1174
URL: https://issues.apache.org/jira/browse/PIG-1174
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
For example, when making a request like this:
STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
Two changes should be made:
1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (PIG-1174) Creation of output path should be done
by storage function
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reopened PIG-1174:
-----------------------------
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Fix For: 0.7.0
>
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-1174) Creation of output path should be done
by storage function
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates resolved PIG-1174.
-----------------------------
Resolution: Duplicate
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Fix For: 0.7.0
>
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1174) Creation of output path should be done
by storage function
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794758#action_12794758 ]
Bill Graham commented on PIG-1174:
----------------------------------
Actually, I believe that item #1 above is already possible via the UDF context.
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1174) Creation of output path should be done
by storage function
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-1174:
----------------------------
Fix Version/s: 0.7.0
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Fix For: 0.7.0
>
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1174) Creation of output path should be done
by storage function
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12806071#action_12806071 ]
Bill Graham commented on PIG-1174:
----------------------------------
Hi Olga,
You marked this bug as fixed, but from your comments it seems like instead
its a duplicate or child task of some other JIRA having to do with LSR. If
that's the case can you please mark it as such so we can have a JIRA to
track the progress of this work?
thanks,
Bill
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Fix For: 0.7.0
>
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1174) Creation of output path should be done by
storage function
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai closed PIG-1174.
---------------------------
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Fix For: 0.7.0
>
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1174) Creation of output path should be done
by storage function
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796449#action_12796449 ]
Alan Gates commented on PIG-1174:
---------------------------------
Delegating creation of the output path to the storage function is not trivial. The storage function is invoked on every reducer (or every mapper for map only jobs). So delaying creation until the storage function will create a race condition that the storage functions will handle. And if the solution is just to let the first one win and all the rest error out and ignore the error, for a large job this will still bombard the namenode with hundreds or thousands of bogus mkdir requests. It also has the problem that all the storage functions that get an error can't tell if it's really an error (there's old data there they are overwriting) versus they just lost the race and another function has already created it.
We are reworking the way load and store function interact with InputFormats and OutputFormats (see PIG-966 for full details). This will push the responsibility of file creation onto the OutputFormat. This may partially address your concerns.
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-1174) Creation of output path should be done
by storage function
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich resolved PIG-1174.
---------------------------------
Resolution: Fixed
With LSR, directory creation will be handled by the output format and will be store function specific
> Creation of output path should be done by storage function
> ----------------------------------------------------------
>
> Key: PIG-1174
> URL: https://issues.apache.org/jira/browse/PIG-1174
> Project: Pig
> Issue Type: Bug
> Reporter: Bill Graham
> Fix For: 0.7.0
>
>
> When executing a STORE command, Pig creates the output location before the storage function gets called. This causes problems with storage functions that have logic to determine the output location. See this thread:
> http://www.mail-archive.com/pig-user%40hadoop.apache.org/msg01538.html
> For example, when making a request like this:
> STORE A INTO '/my/home/output' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Pig creates a file '/my/home/output' and then an exception is thrown when MultiStorage tries to make a directory under '/my/home/output'. The workaround is to instead specify a dummy location as the first path like so:
> STORE A INTO '/my/home/output/temp' USING MultiStorage('/my/home/output','0', 'none', '\t');
> Two changes should be made:
> 1. The path specified in the INTO clause should be available to the storage function so it doesn't need to be duplicated.
> 2. The creation of the output paths should be delegated to the storage function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.