You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2020/03/10 06:01:45 UTC

[GitHub] [kylin] harveyyue opened a new pull request #1154: KYLIN-3847 Flink cubing step : fact distinct

harveyyue opened a new pull request #1154: KYLIN-3847 Flink cubing step : fact distinct
URL: https://github.com/apache/kylin/pull/1154
 
 
   Refer jira: https://issues.apache.org/jira/browse/KYLIN-3847
   Add flink fact distinct columns step, to avoid a mass of duplicated code with Spark and Flink implement self fact distinct step, add base class FactDistinctColumnsBase, extract core methods and variables from mapper/reducer class.  
   About different engine will implement the abstract method FactDistinctColumnsBase.Visitor.collect(...) to output the result from Spark/Flink engine framework.
   Please help to review this code changes 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [kylin] codecov-io commented on issue #1154: KYLIN-3847 Flink cubing step : fact distinct

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1154: KYLIN-3847 Flink cubing step : fact distinct
URL: https://github.com/apache/kylin/pull/1154#issuecomment-596982997
 
 
   # [Codecov](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=h1) Report
   > Merging [#1154](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/ebc4c267b56833475881fb482b1467fbe8367df1?src=pr&el=desc) will **decrease** coverage by `0.14%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/kylin/pull/1154/graphs/tree.svg?width=650&token=JawVgbgsVo&height=150&src=pr)](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1154      +/-   ##
   ============================================
   - Coverage     25.05%   24.91%   -0.15%     
   + Complexity     6248     6245       -3     
   ============================================
     Files          1446     1449       +3     
     Lines         88305    88803     +498     
     Branches      12357    12413      +56     
   ============================================
   - Hits          22128    22124       -4     
   - Misses        64007    64506     +499     
   - Partials       2170     2173       +3
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...kylin/engine/mr/steps/FactDistinctColumnsBase.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLW1yL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvbXIvc3RlcHMvRmFjdERpc3RpbmN0Q29sdW1uc0Jhc2UuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...ylin/engine/flink/FlinkBatchCubingJobBuilder2.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvRmxpbmtCYXRjaEN1YmluZ0pvYkJ1aWxkZXIyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [.../java/org/apache/kylin/common/KylinConfigBase.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZ0Jhc2UuamF2YQ==) | `12.24% <0%> (-0.02%)` | `44 <0> (ø)` | |
   | [.../java/org/apache/kylin/common/util/StringUtil.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL1N0cmluZ1V0aWwuamF2YQ==) | `40.81% <0%> (-0.43%)` | `22 <0> (ø)` | |
   | [...kylin/engine/flink/HadoopMultipleOutputFormat.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvSGFkb29wTXVsdGlwbGVPdXRwdXRGb3JtYXQuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...e/kylin/engine/flink/FlinkFactDistinctColumns.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvRmxpbmtGYWN0RGlzdGluY3RDb2x1bW5zLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | |
   | [...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh) | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | |
   | [...ain/java/org/apache/kylin/dict/TrieDictionary.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L1RyaWVEaWN0aW9uYXJ5LmphdmE=) | `72.55% <0%> (-0.47%)` | `56% <0%> (-1%)` | |
   | ... and [2 more](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=footer). Last update [ebc4c26...bc5a7b8](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [kylin] codecov-io edited a comment on issue #1154: KYLIN-3847 Flink cubing step : fact distinct

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #1154: KYLIN-3847 Flink cubing step : fact distinct
URL: https://github.com/apache/kylin/pull/1154#issuecomment-596982997
 
 
   # [Codecov](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=h1) Report
   > Merging [#1154](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/ebc4c267b56833475881fb482b1467fbe8367df1?src=pr&el=desc) will **decrease** coverage by `0.14%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/kylin/pull/1154/graphs/tree.svg?width=650&token=JawVgbgsVo&height=150&src=pr)](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1154      +/-   ##
   ============================================
   - Coverage     25.05%   24.91%   -0.15%     
   + Complexity     6248     6245       -3     
   ============================================
     Files          1446     1449       +3     
     Lines         88305    88803     +498     
     Branches      12357    12413      +56     
   ============================================
   - Hits          22128    22124       -4     
   - Misses        64007    64506     +499     
   - Partials       2170     2173       +3
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...kylin/engine/mr/steps/FactDistinctColumnsBase.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLW1yL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvbXIvc3RlcHMvRmFjdERpc3RpbmN0Q29sdW1uc0Jhc2UuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...ylin/engine/flink/FlinkBatchCubingJobBuilder2.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvRmxpbmtCYXRjaEN1YmluZ0pvYkJ1aWxkZXIyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | [.../java/org/apache/kylin/common/KylinConfigBase.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZ0Jhc2UuamF2YQ==) | `12.24% <0%> (-0.02%)` | `44 <0> (ø)` | |
   | [.../java/org/apache/kylin/common/util/StringUtil.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL1N0cmluZ1V0aWwuamF2YQ==) | `40.81% <0%> (-0.43%)` | `22 <0> (ø)` | |
   | [...kylin/engine/flink/HadoopMultipleOutputFormat.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvSGFkb29wTXVsdGlwbGVPdXRwdXRGb3JtYXQuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...e/kylin/engine/flink/FlinkFactDistinctColumns.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvRmxpbmtGYWN0RGlzdGluY3RDb2x1bW5zLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | |
   | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | |
   | [...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh) | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | |
   | [...ain/java/org/apache/kylin/dict/TrieDictionary.java](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L1RyaWVEaWN0aW9uYXJ5LmphdmE=) | `72.55% <0%> (-0.47%)` | `56% <0%> (-1%)` | |
   | ... and [2 more](https://codecov.io/gh/apache/kylin/pull/1154/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=footer). Last update [ebc4c26...bc5a7b8](https://codecov.io/gh/apache/kylin/pull/1154?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [kylin] coveralls commented on issue #1154: KYLIN-3847 Flink cubing step : fact distinct

Posted by GitBox <gi...@apache.org>.
coveralls commented on issue #1154: KYLIN-3847 Flink cubing step : fact distinct
URL: https://github.com/apache/kylin/pull/1154#issuecomment-596985885
 
 
   ## Pull Request Test Coverage Report for [Build 5731](https://coveralls.io/builds/29244240)
   
   * **0** of **504**   **(0.0%)**  changed or added relevant lines in **6** files are covered.
   * **1** unchanged line in **1** file lost coverage.
   * Overall coverage decreased (**-0.2%**) to **27.364%**
   
   ---
   
   |  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
   | :-----|--------------|--------|---: |
   | [core-common/src/main/java/org/apache/kylin/common/util/StringUtil.java](https://coveralls.io/builds/29244240/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FStringUtil.java#L224) | 0 | 1 | 0.0%
   | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/29244240/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L1563) | 0 | 6 | 0.0%
   | [engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkBatchCubingJobBuilder2.java](https://coveralls.io/builds/29244240/source?filename=engine-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fflink%2FFlinkBatchCubingJobBuilder2.java#L64) | 0 | 22 | 0.0%
   | [engine-flink/src/main/java/org/apache/kylin/engine/flink/HadoopMultipleOutputFormat.java](https://coveralls.io/builds/29244240/source?filename=engine-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fflink%2FHadoopMultipleOutputFormat.java#L42) | 0 | 33 | 0.0%
   | [engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkFactDistinctColumns.java](https://coveralls.io/builds/29244240/source?filename=engine-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fflink%2FFlinkFactDistinctColumns.java#L73) | 0 | 157 | 0.0%
   | [engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/FactDistinctColumnsBase.java](https://coveralls.io/builds/29244240/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2Fsteps%2FFactDistinctColumnsBase.java#L73) | 0 | 285 | 0.0%
   <!-- | **Total:** | **0** | **504** | **0.0%** | -->
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-----|--------------|--: |
   | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/29244240/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% |
   <!-- | **Total:** | **1** |  | -->
   
   |  Totals | [![Coverage Status](https://coveralls.io/builds/29244240/badge)](https://coveralls.io/builds/29244240) |
   | :-- | --: |
   | Change from base [Build 5720](https://coveralls.io/builds/29194855): |  -0.2% |
   | Covered Lines: | 24300 |
   | Relevant Lines: | 88803 |
   
   ---
   ##### 💛  - [Coveralls](https://coveralls.io)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [kylin] lgtm-com[bot] commented on issue #1154: KYLIN-3847 Flink cubing step : fact distinct

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #1154: KYLIN-3847 Flink cubing step : fact distinct
URL: https://github.com/apache/kylin/pull/1154#issuecomment-596931656
 
 
   This pull request **introduces 1 alert** when merging bc5a7b812cbfd91bed3313bb52da5c1f2d3f8a51 into ebc4c267b56833475881fb482b1467fbe8367df1 - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-811336ac8b1587166486320e944b794a10217efe)
   
   **new alerts:**
   
   * 1 for Boxed variable is never null

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services