You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Prasad Chakka (JIRA)" <ji...@apache.org> on 2009/10/13 19:33:31 UTC
[jira] Created: (HIVE-874) add partitions found during metastore
check
add partitions found during metastore check
-------------------------------------------
Key: HIVE-874
URL: https://issues.apache.org/jira/browse/HIVE-874
Project: Hadoop Hive
Issue Type: Improvement
Affects Versions: 0.3.0, 0.2.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.0
Reporter: Prasad Chakka
Assignee: Cyrus Katrak
'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-874) add partitions found during metastore
check
Posted by "Cyrus Katrak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cyrus Katrak updated HIVE-874:
------------------------------
Status: Patch Available (was: Open)
Tests added
> add partitions found during metastore check
> -------------------------------------------
>
> Key: HIVE-874
> URL: https://issues.apache.org/jira/browse/HIVE-874
> Project: Hadoop Hive
> Issue Type: Improvement
> Affects Versions: 0.3.0, 0.2.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.0
> Reporter: Prasad Chakka
> Assignee: Cyrus Katrak
> Attachments: HIVE-874.patch
>
>
> 'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-874) add partitions found during metastore
check
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767012#action_12767012 ]
Prasad Chakka commented on HIVE-874:
------------------------------------
looks good, will run tests and commit to trunk.
> add partitions found during metastore check
> -------------------------------------------
>
> Key: HIVE-874
> URL: https://issues.apache.org/jira/browse/HIVE-874
> Project: Hadoop Hive
> Issue Type: Improvement
> Affects Versions: 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.0
> Reporter: Prasad Chakka
> Assignee: Cyrus Katrak
> Attachments: HIVE-874.patch
>
>
> 'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-874) add partitions found during metastore
check
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767690#action_12767690 ]
Prasad Chakka commented on HIVE-874:
------------------------------------
committed to trunk. Tahnks Cyrus.
> add partitions found during metastore check
> -------------------------------------------
>
> Key: HIVE-874
> URL: https://issues.apache.org/jira/browse/HIVE-874
> Project: Hadoop Hive
> Issue Type: Improvement
> Affects Versions: 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.0
> Reporter: Prasad Chakka
> Assignee: Cyrus Katrak
> Attachments: HIVE-874.patch
>
>
> 'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-874) add partitions found during metastore
check
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-874:
----------------------------
Resolution: Fixed
Fix Version/s: 0.5.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
@Prasad, feel free to open it back if you think the patch needs to be backported to 0.4 also
> add partitions found during metastore check
> -------------------------------------------
>
> Key: HIVE-874
> URL: https://issues.apache.org/jira/browse/HIVE-874
> Project: Hadoop Hive
> Issue Type: Improvement
> Affects Versions: 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.0
> Reporter: Prasad Chakka
> Assignee: Cyrus Katrak
> Fix For: 0.5.0
>
> Attachments: HIVE-874.patch
>
>
> 'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-874) add partitions found during metastore
check
Posted by "Cyrus Katrak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cyrus Katrak updated HIVE-874:
------------------------------
Attachment: HIVE-874.patch
> add partitions found during metastore check
> -------------------------------------------
>
> Key: HIVE-874
> URL: https://issues.apache.org/jira/browse/HIVE-874
> Project: Hadoop Hive
> Issue Type: Improvement
> Affects Versions: 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.0
> Reporter: Prasad Chakka
> Assignee: Cyrus Katrak
> Attachments: HIVE-874.patch
>
>
> 'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-874) add partitions found during metastore
check
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765162#action_12765162 ]
Prasad Chakka commented on HIVE-874:
------------------------------------
Cyrus,
I created a separate JIRA since HIVE-493 is about automatically inferring partitions during query time and not updating metadata at all. I would like to keep that open since this may not be sufficient for some usecases. Could you upload your patch here?
As for unit tests, check hive/ql/src/test/queries/clientpositive directory that contains the CLI tests. One of the tests could be to create hdfs directories for a table like 'srcpart' and run 'msck repair' on it and do a 'show partitions' on the table again to see if the partition is listed. We need to have some unit tests so that this functionality will not be broken accidentally by some other check-in.
> add partitions found during metastore check
> -------------------------------------------
>
> Key: HIVE-874
> URL: https://issues.apache.org/jira/browse/HIVE-874
> Project: Hadoop Hive
> Issue Type: Improvement
> Affects Versions: 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.0
> Reporter: Prasad Chakka
> Assignee: Cyrus Katrak
>
> 'msck' just reports the list of partition directories that exist but do not have corresponding metadata. This can happen if a process outside of hive is populating the directories. Hive should support an option to 'msck' that would also add default metadata for these partitions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.