You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Paul Yang (JIRA)" <ji...@apache.org> on 2010/04/23 20:26:54 UTC
[jira] Created: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Cached FileSystem can lead to persistant IOExceptions
-----------------------------------------------------
Key: HIVE-1322
URL: https://issues.apache.org/jira/browse/HIVE-1322
Project: Hadoop Hive
Issue Type: Bug
Components: Metastore
Reporter: Paul Yang
Assignee: Paul Yang
Priority: Minor
Fix For: 0.6.0
In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866712#action_12866712 ]
Namit Jain commented on HIVE-1322:
----------------------------------
+1
> Cached FileSystem can lead to persistant IOExceptions
> -----------------------------------------------------
>
> Key: HIVE-1322
> URL: https://issues.apache.org/jira/browse/HIVE-1322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: HIVE-1322.1.patch, HIVE-1322.2.patch
>
>
> In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
> One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866408#action_12866408 ]
Paul Yang commented on HIVE-1322:
---------------------------------
A unit test is just about impossible, so I was only able to run a manual test:
1. With a running DFS, I issued a create table command that would create a FileSystem object.
2. The DFS was shut down, and a create table command was issued again. This threw an IOException which would have cleared the cache.
3. The DFS was restarted, and I verified that a new create table command succeeded.
Given that this is a straightforward change, the above should be sufficient to demonstrate that this doesn't break anything.
> Cached FileSystem can lead to persistant IOExceptions
> -----------------------------------------------------
>
> Key: HIVE-1322
> URL: https://issues.apache.org/jira/browse/HIVE-1322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: HIVE-1322.1.patch, HIVE-1322.2.patch
>
>
> In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
> One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-1322:
----------------------------
Attachment: HIVE-1322.1.patch
Tricky to have a test for this one
> Cached FileSystem can lead to persistant IOExceptions
> -----------------------------------------------------
>
> Key: HIVE-1322
> URL: https://issues.apache.org/jira/browse/HIVE-1322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: HIVE-1322.1.patch
>
>
> In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
> One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-1322:
----------------------------
Attachment: HIVE-1322.2.patch
> Cached FileSystem can lead to persistant IOExceptions
> -----------------------------------------------------
>
> Key: HIVE-1322
> URL: https://issues.apache.org/jira/browse/HIVE-1322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: HIVE-1322.1.patch, HIVE-1322.2.patch
>
>
> In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
> One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-1322 started by Paul Yang.
> Cached FileSystem can lead to persistant IOExceptions
> -----------------------------------------------------
>
> Key: HIVE-1322
> URL: https://issues.apache.org/jira/browse/HIVE-1322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: HIVE-1322.1.patch, HIVE-1322.2.patch
>
>
> In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
> One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-1322:
----------------------------
Status: Patch Available (was: In Progress)
> Cached FileSystem can lead to persistant IOExceptions
> -----------------------------------------------------
>
> Key: HIVE-1322
> URL: https://issues.apache.org/jira/browse/HIVE-1322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: HIVE-1322.1.patch, HIVE-1322.2.patch
>
>
> In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
> One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1322) Cached FileSystem can lead to
persistant IOExceptions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1322:
-----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Committed. Thanks Paul
> Cached FileSystem can lead to persistant IOExceptions
> -----------------------------------------------------
>
> Key: HIVE-1322
> URL: https://issues.apache.org/jira/browse/HIVE-1322
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Paul Yang
> Assignee: Paul Yang
> Priority: Minor
> Fix For: 0.6.0
>
> Attachments: HIVE-1322.1.patch, HIVE-1322.2.patch
>
>
> In the metastore, the FileSystem object is created through Path.getFileSytem(), which caches the created instance for performance. For DFS, the cached FileSystem seems to retain the original IP (resolved from the supplied hostname). If the hostname->IP mapping changes, then FS operations will throw IOExceptions. Because the FileSystem is cached, re-creating the object has no effect and will continue to result in IOExceptions.
> One solution is to call close on an IOException. That will remove the entry in the cache.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.