You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2020/05/21 16:18:47 UTC

[GitHub] [kylin] zzcclp opened a new pull request #1226: KYLIN-4495 Support custom date format for partition date column

zzcclp opened a new pull request #1226:
URL: https://github.com/apache/kylin/pull/1226


   Support custom date format for partition date column.
   
   ## Proposed changes
   
   Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
   
   ## Types of changes
   
   What types of changes does your code introduce to Kylin?
   _Put an `x` in the boxes that apply_
   
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
   - [ ] Documentation Update (if none of the other choices apply)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
   
   - [x] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail
   - [x] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
   - [x] Compiling and unit tests pass locally with my changes
   - [ ] I have added tests that prove my fix is effective or that my feature works
   - [ ] If this change need a document change, I will prepare another pr against the `document` branch
   - [ ] Any dependent changes have been merged
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643607845


   ## Verify test case
   
   #### Case-1
   The result is correct 
   ![image](https://user-images.githubusercontent.com/9884693/84567245-f46bb880-ada9-11ea-8eea-69716bd77330.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] nichunen commented on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
nichunen commented on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-632453394


   @zzcclp Hi, have you tested the pr with your data source with a specified date format?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang commented on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang commented on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643605325


   ### Check list
   - [x] Step 1. [**OPTIONAL**]Understand background/root cause/design basically(one hour EST) . Its issue type is 
       - [ ] bug fix
       - [ ] new feature
       - [x] enhancement
   - [x] Step 2. Test cases are designed and documented (30 minutes EST).
   - [x] Step 3. Prepare specific env, for example:
       - mock data (maybe some specific data type)
       - test env (maybe install a RDBMS instance).
   - [x] Step 4. Verify and make sure test cases passed.
   - [x] Step 5. Paste manual important steps and screenshots here (20 minutes EST). 
       - If you find difficulty in pick most important evidence, please attach diagnosis package. 
   - [x] Step 6. Do more check in user perspective (20 minutes EST)
       - [x] Doc need be updated? And if it is updated? Ask help for release manager if so.
       - [x] If it is a breaking change so we should notify Kylin community? Ask help for release manager if so.
   - [x] Step 7.  Summarize this test (20 minutes EST).
   
   ### Total estimate
   - 4 hours for small issue from optimistic estimate. 
   
   ### Note
   If you find some unexpected and **unrelated** error/mistake, please DO report it if it is truly a mistake, I think we may research and fix it in the future.
   
   If you find background information/root cause analysis is not complete or ambiguous, please try to contact to author or do a quick research and record what you find. It is a good chance to learn something interesting.
   
   If you find it is not easy to design testcase, please notify release manager.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643605367


   ## Design test cases
   
   ### Prepare test env
   #### Hadoop Env
   HDP 2.4
   #### source table
   Create hive table using local csv file
   1. create local csv file
   id,name,dt
   1,wang,2019/01/01
   2,zhang,2019/01/02
   3,Lee,2019/01/03
   4,Liu,2019/01/04
   
   2. create table in hive
   ```sql
   CREATE TABLE wangrupeng.self_define_date(id int, name string, dt date) row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ("separatorChar"=",") STORED AS TEXTFILE;
   ```
   
   3. local csv into hive table
   ```sql
   load data local inpath '/root/wangrupeng/kylin-3.1.0/test.csv' overwrite into table self_define_date;
   ```
   
   #### Create a new model and cube
   1. model
   ```json
   {
     "uuid": "ce9eaaeb-974d-1ae2-fb51-7333a6d7aab4",
     "last_modified": 1592045287556,
     "version": "3.0.0.20500",
     "name": "test_selfdefine_date",
     "owner": "ADMIN",
     "is_draft": false,
     "description": "",
     "fact_table": "WANGRUPENG.SELF_DEFINE_DATE",
     "lookups": [],
     "dimensions": [
       {
         "table": "SELF_DEFINE_DATE",
         "columns": [
           "ID"
         ]
       }
     ],
     "metrics": [
       "SELF_DEFINE_DATE.NAME"
     ],
     "filter_condition": "",
     "partition_desc": {
       "partition_date_column": "SELF_DEFINE_DATE.DT",
       "partition_time_column": null,
       "partition_date_start": 0,
       "partition_date_format": "yyyy/MM/dd",
       "partition_time_format": "HH:mm:ss",
       "partition_type": "APPEND",
       "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
     },
     "capacity": "MEDIUM",
     "projectName": "learn_kylin"
   }
   ```
   
   2. cube
   ```json
   {
     "uuid": "96fc0227-ad30-49ca-316b-b23df48dff5e",
     "last_modified": 1592045402854,
     "version": "3.0.0.20500",
     "name": "test_selfdefine_cube",
     "is_draft": false,
     "model_name": "test_selfdefine_date",
     "description": "",
     "null_string": null,
     "dimensions": [
       {
         "name": "ID",
         "table": "SELF_DEFINE_DATE",
         "column": "ID",
         "derived": null
       }
     ],
     "measures": [
       {
         "name": "_COUNT_",
         "function": {
           "expression": "COUNT",
           "parameter": {
             "type": "constant",
             "value": "1"
           },
           "returntype": "bigint"
         }
       }
     ],
     "dictionaries": [],
     "rowkey": {
       "rowkey_columns": [
         {
           "column": "SELF_DEFINE_DATE.ID",
           "encoding": "dict",
           "encoding_version": 1,
           "isShardBy": false
         }
       ]
     },
     "hbase_mapping": {
       "column_family": [
         {
           "name": "F1",
           "columns": [
             {
               "qualifier": "M",
               "measure_refs": [
                 "_COUNT_"
               ]
             }
           ]
         }
       ]
     },
     "aggregation_groups": [
       {
         "includes": [
           "SELF_DEFINE_DATE.ID"
         ],
         "select_rule": {
           "hierarchy_dims": [],
           "mandatory_dims": [],
           "joint_dims": []
         }
       }
     ],
     "signature": "ZIYGtYJLdlaTYyih6VMUyQ==",
     "notify_list": [],
     "status_need_notify": [
       "ERROR",
       "DISCARDED",
       "SUCCEED"
     ],
     "partition_date_start": 0,
     "partition_date_end": 3153600000000,
     "auto_merge_time_ranges": [],
     "volatile_range": 0,
     "retention_range": 0,
     "engine_type": 2,
     "storage_type": 2,
     "override_kylin_properties": {},
     "cuboid_black_list": [],
     "parent_forward": 3,
     "mandatory_dimension_set_list": [],
     "snapshot_table_desc_list": []
   }
   ```
   
   ### Case-1
   
   ```sql
   select count(*) from SELF_DEFINE_DATE where dt > '2019/01/02';
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] coveralls edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-632233973


   ## Pull Request Test Coverage Report for [Build 5934](https://coveralls.io/builds/30965423)
   
   * **0** of **0**   changed or added relevant lines in **0** files are covered.
   * **9** unchanged lines in **4** files lost coverage.
   * Overall coverage decreased (**-0.004%**) to **27.289%**
   
   ---
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-----|--------------|--: |
   | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/30965423/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% |
   | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/30965423/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% |
   | [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/30965423/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
   | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/30965423/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 4 | 78.12% |
   <!-- | **Total:** | **9** |  | -->
   
   |  Totals | [![Coverage Status](https://coveralls.io/builds/30965423/badge)](https://coveralls.io/builds/30965423) |
   | :-- | --: |
   | Change from base [Build 5911](https://coveralls.io/builds/30861301): |  -0.004% |
   | Covered Lines: | 24349 |
   | Relevant Lines: | 89225 |
   
   ---
   ##### 💛  - [Coveralls](https://coveralls.io)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] shaofengshi merged pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
shaofengshi merged pull request #1226:
URL: https://github.com/apache/kylin/pull/1226


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang commented on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang commented on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643607845


   ## Verify test case
   
   #### Case-1
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] codecov-commenter edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-632231689


   # [Codecov](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=h1) Report
   > Merging [#1226](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/7886a2498178ded085497d6204b4b367f56ce0c8&el=desc) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/kylin/pull/1226/graphs/tree.svg?width=650&height=150&src=pr&token=JawVgbgsVo)](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1226      +/-   ##
   ============================================
   - Coverage     24.85%   24.84%   -0.02%     
   + Complexity     6266     6263       -3     
   ============================================
     Files          1453     1453              
     Lines         89225    89225              
     Branches      12456    12456              
   ============================================
   - Hits          22181    22166      -15     
   - Misses        64869    64879      +10     
   - Partials       2175     2180       +5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0.00%> (-5.41%)` | `6.00% <0.00%> (-1.00%)` | |
   | [...org/apache/kylin/rest/util/QueryRequestLimits.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvdXRpbC9RdWVyeVJlcXVlc3RMaW1pdHMuamF2YQ==) | `35.71% <0.00%> (-4.77%)` | `5.00% <0.00%> (-1.00%)` | |
   | [...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh) | `74.41% <0.00%> (-2.33%)` | `12.00% <0.00%> (ø%)` | |
   | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0.00%> (-2.31%)` | `0.00% <0.00%> (ø%)` | |
   | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.60% <0.00%> (-1.52%)` | `7.00% <0.00%> (ø%)` | |
   | [...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh) | `81.74% <0.00%> (-0.80%)` | `17.00% <0.00%> (-1.00%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=footer). Last update [7886a24...a11cdb5](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643605367


   ## Design test cases
   
   ### Prepare test env
   #### source table
   Create hive table using local csv file
   1. create local csv file
   id,name,dt
   1,wang,2019/01/01
   2,zhang,2019/01/02
   3,Lee,2019/01/03
   4,Liu,2019/01/04
   
   2. create table in hive
   ```sql
   CREATE TABLE wangrupeng.self_define_date(id int, name string, dt date) row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ("separatorChar"=",") STORED AS TEXTFILE;
   ```
   
   3. local csv into hive table
   ```sql
   load data local inpath '/root/wangrupeng/kylin-3.1.0/test.csv' overwrite into table self_define_date;
   ```
   
   #### Create a new model and cube
   1. model
   ```json
   {
     "uuid": "ce9eaaeb-974d-1ae2-fb51-7333a6d7aab4",
     "last_modified": 1592045287556,
     "version": "3.0.0.20500",
     "name": "test_selfdefine_date",
     "owner": "ADMIN",
     "is_draft": false,
     "description": "",
     "fact_table": "WANGRUPENG.SELF_DEFINE_DATE",
     "lookups": [],
     "dimensions": [
       {
         "table": "SELF_DEFINE_DATE",
         "columns": [
           "ID"
         ]
       }
     ],
     "metrics": [
       "SELF_DEFINE_DATE.NAME"
     ],
     "filter_condition": "",
     "partition_desc": {
       "partition_date_column": "SELF_DEFINE_DATE.DT",
       "partition_time_column": null,
       "partition_date_start": 0,
       "partition_date_format": "yyyy/MM/dd",
       "partition_time_format": "HH:mm:ss",
       "partition_type": "APPEND",
       "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
     },
     "capacity": "MEDIUM",
     "projectName": "learn_kylin"
   }
   ```
   
   2. cube
   ```json
   {
     "uuid": "96fc0227-ad30-49ca-316b-b23df48dff5e",
     "last_modified": 1592045402854,
     "version": "3.0.0.20500",
     "name": "test_selfdefine_cube",
     "is_draft": false,
     "model_name": "test_selfdefine_date",
     "description": "",
     "null_string": null,
     "dimensions": [
       {
         "name": "ID",
         "table": "SELF_DEFINE_DATE",
         "column": "ID",
         "derived": null
       }
     ],
     "measures": [
       {
         "name": "_COUNT_",
         "function": {
           "expression": "COUNT",
           "parameter": {
             "type": "constant",
             "value": "1"
           },
           "returntype": "bigint"
         }
       }
     ],
     "dictionaries": [],
     "rowkey": {
       "rowkey_columns": [
         {
           "column": "SELF_DEFINE_DATE.ID",
           "encoding": "dict",
           "encoding_version": 1,
           "isShardBy": false
         }
       ]
     },
     "hbase_mapping": {
       "column_family": [
         {
           "name": "F1",
           "columns": [
             {
               "qualifier": "M",
               "measure_refs": [
                 "_COUNT_"
               ]
             }
           ]
         }
       ]
     },
     "aggregation_groups": [
       {
         "includes": [
           "SELF_DEFINE_DATE.ID"
         ],
         "select_rule": {
           "hierarchy_dims": [],
           "mandatory_dims": [],
           "joint_dims": []
         }
       }
     ],
     "signature": "ZIYGtYJLdlaTYyih6VMUyQ==",
     "notify_list": [],
     "status_need_notify": [
       "ERROR",
       "DISCARDED",
       "SUCCEED"
     ],
     "partition_date_start": 0,
     "partition_date_end": 3153600000000,
     "auto_merge_time_ranges": [],
     "volatile_range": 0,
     "retention_range": 0,
     "engine_type": 2,
     "storage_type": 2,
     "override_kylin_properties": {},
     "cuboid_black_list": [],
     "parent_forward": 3,
     "mandatory_dimension_set_list": [],
     "snapshot_table_desc_list": []
   }
   ```
   
   ### Case-1
   
   ```sql
   select count(*) from SELF_DEFINE_DATE where dt > '2019/01/02';
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] zzcclp commented on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
zzcclp commented on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-632456654


   > @zzcclp Hi, have you tested the pr with your data source with a specified date format?
   
   Yes, I have tested this pr locally, it builded cube successfully with custom data format: yyyy/MM/dd. 
   And will add ut in PartitionDescTest.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang commented on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang commented on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643605367


   ## Design test cases
   
   ### Prepare test env
   Kylin sample cube
   
   ### Case-1
   No special query scenario
   ```sql
   SELECT lstg_format_name, SUM(price) AS gmv, COUNT(DISTINCT seller_id) AS dist_seller FROM kylin_sales WHERE lstg_format_name='FP-GTC' GROUP BY lstg_format_name HAVING COUNT(DISTINCT seller_id) > 50;
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] coveralls commented on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-632233973


   ## Pull Request Test Coverage Report for [Build 5930](https://coveralls.io/builds/30949870)
   
   * **0** of **0**   changed or added relevant lines in **0** files are covered.
   * **9** unchanged lines in **4** files lost coverage.
   * Overall coverage decreased (**-0.004%**) to **27.289%**
   
   ---
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-----|--------------|--: |
   | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/30949870/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% |
   | [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/30949870/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
   | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/CheckPointStore.java](https://coveralls.io/builds/30949870/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2FCheckPointStore.java#L98) | 2 | 73.74% |
   | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/30949870/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% |
   <!-- | **Total:** | **9** |  | -->
   
   |  Totals | [![Coverage Status](https://coveralls.io/builds/30949870/badge)](https://coveralls.io/builds/30949870) |
   | :-- | --: |
   | Change from base [Build 5911](https://coveralls.io/builds/30861301): |  -0.004% |
   | Covered Lines: | 24349 |
   | Relevant Lines: | 89225 |
   
   ---
   ##### 💛  - [Coveralls](https://coveralls.io)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643605367


   ## Design test cases
   
   ### Prepare test env
   #### source table
   Create hive table using local csv file
   1. create local csv file
   id,name,dt
   1,wang,2019/01/01
   2,zhang,2019/01/02
   3,Lee,2019/01/03
   4,Liu,2019/01/04
   
   2. create table in hive
   ```sql
   CREATE TABLE wangrupeng.self_define_date(id int, name string, dt date) row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ("separatorChar"=",") STORED AS TEXTFILE;
   ```
   
   3. local csv into hive table
   ```sql
   load data local inpath '/root/wangrupeng/kylin-3.1.0/test.csv' overwrite into table self_define_date;
   ```
   
   #### Create a new model and cube
   1. model
   
   
   2. cube
   
   
   ### Case-1
   
   ```sql
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643607845


   ## Verify test case
   
   #### Case-1
   The result is correct with the csv source 
   ![image](https://user-images.githubusercontent.com/9884693/84567245-f46bb880-ada9-11ea-8eea-69716bd77330.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] codecov-commenter commented on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-632231689


   # [Codecov](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=h1) Report
   > Merging [#1226](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/7886a2498178ded085497d6204b4b367f56ce0c8&el=desc) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/kylin/pull/1226/graphs/tree.svg?width=650&height=150&src=pr&token=JawVgbgsVo)](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1226      +/-   ##
   ============================================
   - Coverage     24.85%   24.84%   -0.02%     
   + Complexity     6266     6264       -2     
   ============================================
     Files          1453     1453              
     Lines         89225    89225              
     Branches      12456    12456              
   ============================================
   - Hits          22181    22168      -13     
   - Misses        64869    64879      +10     
   - Partials       2175     2178       +3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...org/apache/kylin/rest/util/QueryRequestLimits.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvdXRpbC9RdWVyeVJlcXVlc3RMaW1pdHMuamF2YQ==) | `35.71% <0.00%> (-4.77%)` | `5.00% <0.00%> (-1.00%)` | |
   | [...che/kylin/stream/core/storage/CheckPointStore.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-c3RyZWFtLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3N0cmVhbS9jb3JlL3N0b3JhZ2UvQ2hlY2tQb2ludFN0b3JlLmphdmE=) | `63.63% <0.00%> (-3.04%)` | `21.00% <0.00%> (-1.00%)` | |
   | [...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh) | `74.41% <0.00%> (-2.33%)` | `12.00% <0.00%> (ø%)` | |
   | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0.00%> (-2.31%)` | `0.00% <0.00%> (ø%)` | |
   | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/1226/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0.00%> (-0.92%)` | `7.00% <0.00%> (ø%)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=footer). Last update [7886a24...b10eeab](https://codecov.io/gh/apache/kylin/pull/1226?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kylin] RupengWang edited a comment on pull request #1226: KYLIN-4495 Support custom date format for partition date column

Posted by GitBox <gi...@apache.org>.
RupengWang edited a comment on pull request #1226:
URL: https://github.com/apache/kylin/pull/1226#issuecomment-643605367


   ## Design test cases
   
   ### Prepare test env
   #### source table
   Create hive table using local csv file
   1. create local csv file
   id,name,dt
   1,wang,2019/01/01
   2,zhang,2019/01/02
   3,Lee,2019/01/03
   4,Liu,2019/01/04
   
   2. create table in hive
   ```sql
   CREATE TABLE wangrupeng.self_define_date(id int, name string, dt date) row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ("separatorChar"=",") STORED AS TEXTFILE;
   ```
   
   3. local csv into hive table
   ```sql
   load data local inpath '/root/wangrupeng/kylin-3.1.0/test.csv' overwrite into table self_define_date;
   ```
   
   #### Create a new model and cube
   1. model
   ```json
   {
     "uuid": "ce9eaaeb-974d-1ae2-fb51-7333a6d7aab4",
     "last_modified": 1592045287556,
     "version": "3.0.0.20500",
     "name": "test_selfdefine_date",
     "owner": "ADMIN",
     "is_draft": false,
     "description": "",
     "fact_table": "WANGRUPENG.SELF_DEFINE_DATE",
     "lookups": [],
     "dimensions": [
       {
         "table": "SELF_DEFINE_DATE",
         "columns": [
           "ID"
         ]
       }
     ],
     "metrics": [
       "SELF_DEFINE_DATE.NAME"
     ],
     "filter_condition": "",
     "partition_desc": {
       "partition_date_column": "SELF_DEFINE_DATE.DT",
       "partition_time_column": null,
       "partition_date_start": 0,
       "partition_date_format": "yyyy/MM/dd",
       "partition_time_format": "HH:mm:ss",
       "partition_type": "APPEND",
       "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
     },
     "capacity": "MEDIUM",
     "projectName": "learn_kylin"
   }
   ```
   
   2. cube
   ```json
   {
     "uuid": "96fc0227-ad30-49ca-316b-b23df48dff5e",
     "last_modified": 1592045402854,
     "version": "3.0.0.20500",
     "name": "test_selfdefine_cube",
     "is_draft": false,
     "model_name": "test_selfdefine_date",
     "description": "",
     "null_string": null,
     "dimensions": [
       {
         "name": "ID",
         "table": "SELF_DEFINE_DATE",
         "column": "ID",
         "derived": null
       }
     ],
     "measures": [
       {
         "name": "_COUNT_",
         "function": {
           "expression": "COUNT",
           "parameter": {
             "type": "constant",
             "value": "1"
           },
           "returntype": "bigint"
         }
       }
     ],
     "dictionaries": [],
     "rowkey": {
       "rowkey_columns": [
         {
           "column": "SELF_DEFINE_DATE.ID",
           "encoding": "dict",
           "encoding_version": 1,
           "isShardBy": false
         }
       ]
     },
     "hbase_mapping": {
       "column_family": [
         {
           "name": "F1",
           "columns": [
             {
               "qualifier": "M",
               "measure_refs": [
                 "_COUNT_"
               ]
             }
           ]
         }
       ]
     },
     "aggregation_groups": [
       {
         "includes": [
           "SELF_DEFINE_DATE.ID"
         ],
         "select_rule": {
           "hierarchy_dims": [],
           "mandatory_dims": [],
           "joint_dims": []
         }
       }
     ],
     "signature": "ZIYGtYJLdlaTYyih6VMUyQ==",
     "notify_list": [],
     "status_need_notify": [
       "ERROR",
       "DISCARDED",
       "SUCCEED"
     ],
     "partition_date_start": 0,
     "partition_date_end": 3153600000000,
     "auto_merge_time_ranges": [],
     "volatile_range": 0,
     "retention_range": 0,
     "engine_type": 2,
     "storage_type": 2,
     "override_kylin_properties": {},
     "cuboid_black_list": [],
     "parent_forward": 3,
     "mandatory_dimension_set_list": [],
     "snapshot_table_desc_list": []
   }
   ```
   
   ### Case-1
   
   ```sql
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org