You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by debugger87 <gi...@git.apache.org> on 2017/08/10 04:20:06 UTC

[GitHub] spark pull request #18900: [SPARK-21687][SQL] Spark SQL should set createTim...

GitHub user debugger87 opened a pull request:

    https://github.com/apache/spark/pull/18900

    [SPARK-21687][SQL] Spark SQL should set createTime for Hive partition

    ## What changes were proposed in this pull request?
    
    Set createTime for every hive partition created in Spark SQL, which could be used to manage data lifecycle in Hive warehouse.
    
    ## How was this patch tested?
    
    No tests
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/debugger87/spark fix/set-create-time-for-hive-partition

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18900.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18900
    
----
commit 71a660ac8dad869d9ba3b4e206b74f5c44660ee6
Author: debugger87 <ya...@gmail.com>
Date:   2017-08-10T04:17:00Z

    [SPARK-21687][SQL] Spark SQL should set createTime for Hive partition

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

Posted by debugger87 <gi...@git.apache.org>.

Github user debugger87 commented on the issue:

    https://github.com/apache/spark/pull/18900
  
    `createTime` is set by HiveMetaStore#initializeAddedPartition
    
    ```
    private void initializeAddedPartition(Table tbl, PartitionIterator part, boolean madeDir) throws MetaException {
          if(HiveConf.getBoolVar(this.hiveConf, ConfVars.HIVESTATSAUTOGATHER) && !MetaStoreUtils.isView(tbl)) {
            MetaStoreUtils.updatePartitionStatsFast(part, this.wh, madeDir, false);
          }
    
          long time = System.currentTimeMillis() / 1000L;
          part.setCreateTime((long)((int)time));
          if(part.getParameters() == null || part.getParameters().get("transient_lastDdlTime") == null) {
            part.putToParameters("transient_lastDdlTime", Long.toString(time));
          }
          // ignore code lines
    }
    ```
    
    This PR should be closed and we will check the reason why createTime is zero for partitions created by spark sql again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18900
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92376/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

Posted by maropu <gi...@git.apache.org>.

Github user maropu commented on the issue:

    https://github.com/apache/spark/pull/18900
  
    Any update?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18900
  
    **[Test build #80558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80558/testReport)** for PR 18900 at commit [`bf2a105`](https://github.com/apache/spark/commit/bf2a1052f807a7ae36004c819e66fff5c4b45820).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18900
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18900
  
    **[Test build #92222 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92222/testReport)** for PR 18900 at commit [`c843ef1`](https://github.com/apache/spark/commit/c843ef18c4073281c8e8e59abf17a7bcbc6c5560).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18900: [SPARK-21687][SQL] Spark SQL should set createTime for H...

Posted by cxzl25 <gi...@git.apache.org>.

Github user cxzl25 commented on the issue:

    https://github.com/apache/spark/pull/18900
  
    **Modify the partition will lose createTime.**
    Reading the hive partitions ignores createTime when converting the CatalogTablePartition, it will also be lost when modifying partitions.
    Calling this method SessionCatalog#alterPartitions will be lost createTime.
    So can you reopen this pr? @debugger87 
    
    ```sql
    CREATE  TABLE `tmp_test_partition`(
      `c1` string
    )
    PARTITIONED BY (
      `d` string
    );
    ALTER TABLE `tmp_test_partition_1` ADD PARTITION (d='1');
    ALTER TABLE `tmp_test_partition_1` PARTITION (d='1') SET LOCATION 'xxx';
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org