You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Thomas Weise (Created) (JIRA)" <ji...@apache.org> on 2012/04/10 06:41:25 UTC
[jira] [Created] (PIG-2642) StoreMetadata.storeSchema can't access
files in the output directory (Hadoop 0.23)
StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
----------------------------------------------------------------------------------
Key: PIG-2642
URL: https://issues.apache.org/jira/browse/PIG-2642
Project: Pig
Issue Type: Bug
Reporter: Thomas Weise
Assignee: Thomas Weise
Fix For: 0.10.0, 0.9.3, 0.11
With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2642) StoreMetadata.storeSchema can't
access files in the output directory (Hadoop 0.23)
Posted by "Daniel Dai (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252848#comment-13252848 ]
Daniel Dai commented on PIG-2642:
---------------------------------
+1
> StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
> ----------------------------------------------------------------------------------
>
> Key: PIG-2642
> URL: https://issues.apache.org/jira/browse/PIG-2642
> Project: Pig
> Issue Type: Bug
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.10.0, 0.9.3, 0.11
>
> Attachments: PIG-2642.patch
>
>
> With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
> In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
> We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2642) StoreMetadata.storeSchema can't access
files in the output directory (Hadoop 0.23)
Posted by "Thomas Weise (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated PIG-2642:
------------------------------
Attachment: (was: PIG-2642.patch)
> StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
> ----------------------------------------------------------------------------------
>
> Key: PIG-2642
> URL: https://issues.apache.org/jira/browse/PIG-2642
> Project: Pig
> Issue Type: Bug
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.10.0, 0.9.3, 0.11
>
> Attachments: PIG-2642.patch
>
>
> With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
> In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
> We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2642) StoreMetadata.storeSchema can't access
files in the output directory (Hadoop 0.23)
Posted by "Thomas Weise (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated PIG-2642:
------------------------------
Attachment: PIG-2642.patch
Updated comment.
> StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
> ----------------------------------------------------------------------------------
>
> Key: PIG-2642
> URL: https://issues.apache.org/jira/browse/PIG-2642
> Project: Pig
> Issue Type: Bug
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.10.0, 0.9.3, 0.11
>
> Attachments: PIG-2642.patch
>
>
> With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
> In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
> We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2642) StoreMetadata.storeSchema can't access
files in the output directory (Hadoop 0.23)
Posted by "Daniel Dai (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2642:
----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.9/0.10/trunk.
Thanks Thomas!
> StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
> ----------------------------------------------------------------------------------
>
> Key: PIG-2642
> URL: https://issues.apache.org/jira/browse/PIG-2642
> Project: Pig
> Issue Type: Bug
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.10.0, 0.9.3, 0.11
>
> Attachments: PIG-2642.patch
>
>
> With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
> In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
> We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2642) StoreMetadata.storeSchema can't access
files in the output directory (Hadoop 0.23)
Posted by "Thomas Weise (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated PIG-2642:
------------------------------
Attachment: PIG-2642.patch
Reversing order of commitJob and storeCleanup calls in PigOutputCommitter to maintain 0.20 behavior under 0.23.
> StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
> ----------------------------------------------------------------------------------
>
> Key: PIG-2642
> URL: https://issues.apache.org/jira/browse/PIG-2642
> Project: Pig
> Issue Type: Bug
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.10.0, 0.9.3, 0.11
>
> Attachments: PIG-2642.patch
>
>
> With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
> In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
> We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2642) StoreMetadata.storeSchema can't access
files in the output directory (Hadoop 0.23)
Posted by "Thomas Weise (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Weise updated PIG-2642:
------------------------------
Status: Patch Available (was: Open)
> StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
> ----------------------------------------------------------------------------------
>
> Key: PIG-2642
> URL: https://issues.apache.org/jira/browse/PIG-2642
> Project: Pig
> Issue Type: Bug
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.10.0, 0.9.3, 0.11
>
> Attachments: PIG-2642.patch
>
>
> With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
> In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
> We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2642) StoreMetadata.storeSchema can't
access files in the output directory (Hadoop 0.23)
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250852#comment-13250852 ]
Jonathan Eagles commented on PIG-2642:
--------------------------------------
Thomas, I have tested the patch and it is working for me. Great! One thing I would add to update the comment for commitJob reflecting this code is intended for 0.23 instead of 0.20.203.
> StoreMetadata.storeSchema can't access files in the output directory (Hadoop 0.23)
> ----------------------------------------------------------------------------------
>
> Key: PIG-2642
> URL: https://issues.apache.org/jira/browse/PIG-2642
> Project: Pig
> Issue Type: Bug
> Reporter: Thomas Weise
> Assignee: Thomas Weise
> Fix For: 0.10.0, 0.9.3, 0.11
>
> Attachments: PIG-2642.patch
>
>
> With Hadoop 0.20/1.0, storeSchema will be called after the temporary task files have been promoted to the output directory.
> In 0.23, the timing of temporary task file promotion has changed by design, now when storeSchema is called the files have not been promoted to the output directory.
> We have existing applications that depend on the temporary task files being promoted when storeSchema is called. This is a regression from 20.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira