You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2010/06/08 20:31:17 UTC
[jira] Created: (HIVE-1394) do not update transient_lastDdlTime if
the partition is modified by a housekeeping operation
do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
--------------------------------------------------------------------------------------------
Key: HIVE-1394
URL: https://issues.apache.org/jira/browse/HIVE-1394
Project: Hadoop Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Namit Jain
Assignee: Paul Yang
Currently. purging looks at the hdfs time to see the last time the files got modified.
It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1394) do not update transient_lastDdlTime if
the partition is modified by a housekeeping operation
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1394:
-----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.7.0
Resolution: Fixed
Committed. Thanks Ning
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1394.2.patch, HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1394) do not update transient_lastDdlTime if
the partition is modified by a housekeeping operation
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1394:
-----------------------------
Attachment: HIVE-1394.2.patch
Added a check in SemanticAnalyzer to throw an exception when HOLD_DDLTIME is specified in dynamic partition insert or non-existence static partitions.
A negative test case is also added.
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-1394.2.patch, HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1394) do not update transient_lastDdlTime
if the partition is modified by a housekeeping operation
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881358#action_12881358 ]
Namit Jain commented on HIVE-1394:
----------------------------------
Some minor comments:
1. If the hint is set, can you throw an error if the partition is not present at compile time
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1394) do not update transient_lastDdlTime
if the partition is modified by a housekeeping operation
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885680#action_12885680 ]
John Sichi commented on HIVE-1394:
----------------------------------
Note HIVE-1442, which prevents this feature from being used. Also, someone needs to document it.
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1394.2.patch, HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1394) do not update transient_lastDdlTime if
the partition is modified by a housekeeping operation
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang reassigned HIVE-1394:
--------------------------------
Assignee: Ning Zhang (was: Paul Yang)
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1394) do not update transient_lastDdlTime if
the partition is modified by a housekeeping operation
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1394:
-----------------------------
Status: Patch Available (was: Open)
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1394) do not update transient_lastDdlTime
if the partition is modified by a housekeeping operation
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881414#action_12881414 ]
Namit Jain commented on HIVE-1394:
----------------------------------
+1
will commit if the tests pass
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-1394.2.patch, HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1394) do not update transient_lastDdlTime if
the partition is modified by a housekeeping operation
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1394:
-----------------------------
Attachment: HIVE-1394.patch
Adding a new hint to avoid updating transient_lastDdlTime for both table and partitons in metastore.
> do not update transient_lastDdlTime if the partition is modified by a housekeeping operation
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1394
> URL: https://issues.apache.org/jira/browse/HIVE-1394
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Paul Yang
> Attachments: HIVE-1394.patch
>
>
> Currently. purging looks at the hdfs time to see the last time the files got modified.
> It should look at the metastore property instead - these are facebook specific utilities, which do not require any changes to hive.
> However, in some cases, the operation might be performed by some housekeeping job, which should not modify the timestamp.
> Since, hive has no way of knowing the origin of the query, it might be a good idea to add a new hint which specifies that the
> operation is a cleanup operation, and the timestamp in the metastore need not be touched for that scenario.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.