You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2020/01/14 01:51:06 UTC
[GitHub] [kylin] zhangayqian opened a new pull request #1059: KYLIN-4291
Parallel segment building may causes WriteConflictException
zhangayqian opened a new pull request #1059: KYLIN-4291 Parallel segment building may causes WriteConflictException
URL: https://github.com/apache/kylin/pull/1059
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] nichunen merged pull request #1059: KYLIN-4291 Parallel
segment building may causes WriteConflictException
Posted by GitBox <gi...@apache.org>.
nichunen merged pull request #1059: KYLIN-4291 Parallel segment building may causes WriteConflictException
URL: https://github.com/apache/kylin/pull/1059
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #1059: KYLIN-4291 Parallel
segment building may causes WriteConflictException
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1059: KYLIN-4291 Parallel segment building may causes WriteConflictException
URL: https://github.com/apache/kylin/pull/1059#issuecomment-573970351
# [Codecov](https://codecov.io/gh/apache/kylin/pull/1059?src=pr&el=h1) Report
> Merging [#1059](https://codecov.io/gh/apache/kylin/pull/1059?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/37770217b2a9d7a62a40acae74fe77ea1d170b2a?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `46.42%`.
[](https://codecov.io/gh/apache/kylin/pull/1059?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1059 +/- ##
============================================
+ Coverage 25% 25.05% +0.04%
- Complexity 6180 6221 +41
============================================
Files 1438 1438
Lines 87669 87692 +23
Branches 12256 12260 +4
============================================
+ Hits 21926 21972 +46
+ Misses 63593 63572 -21
+ Partials 2150 2148 -2
```
| [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1059?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../org/apache/kylin/dict/lookup/SnapshotManager.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9TbmFwc2hvdE1hbmFnZXIuamF2YQ==) | `67.76% <41.66%> (-4.56%)` | `27 <0> (ø)` | |
| [.../java/org/apache/kylin/dict/DictionaryManager.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L0RpY3Rpb25hcnlNYW5hZ2VyLmphdmE=) | `43.37% <50%> (-1.12%)` | `28 <1> (+1)` | |
| [...lin/stream/core/storage/columnar/FragmentData.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-c3RyZWFtLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3N0cmVhbS9jb3JlL3N0b3JhZ2UvY29sdW1uYXIvRnJhZ21lbnREYXRhLmphdmE=) | `75.67% <0%> (-13.52%)` | `11% <0%> (-1%)` | |
| [...ream/core/storage/columnar/ColumnarStoreCache.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-c3RyZWFtLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3N0cmVhbS9jb3JlL3N0b3JhZ2UvY29sdW1uYXIvQ29sdW1uYXJTdG9yZUNhY2hlLmphdmE=) | `48.86% <0%> (-6.82%)` | `8% <0%> (ø)` | |
| [...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh) | `74.41% <0%> (-2.33%)` | `12% <0%> (ø)` | |
| [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.3% <0%> (-0.92%)` | `7% <0%> (ø)` | |
| [...apache/kylin/engine/spark/SparkExecutableLivy.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-ZW5naW5lLXNwYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvc3BhcmsvU3BhcmtFeGVjdXRhYmxlTGl2eS5qYXZh) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
| [...org/apache/kylin/engine/spark/SparkExecutable.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-ZW5naW5lLXNwYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvc3BhcmsvU3BhcmtFeGVjdXRhYmxlLmphdmE=) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
| [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `78.37% <0%> (+5.4%)` | `7% <0%> (+1%)` | :arrow_up: |
| [...va/org/apache/kylin/dict/AppendTrieDictionary.java](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L0FwcGVuZFRyaWVEaWN0aW9uYXJ5LmphdmE=) | `78.09% <0%> (+10.47%)` | `28% <0%> (+9%)` | :arrow_up: |
| ... and [3 more](https://codecov.io/gh/apache/kylin/pull/1059/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1059?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1059?src=pr&el=footer). Last update [3777021...057fd82](https://codecov.io/gh/apache/kylin/pull/1059?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] zhoukangcn commented on a change in pull request #1059:
KYLIN-4291 Parallel segment building may causes WriteConflictException
Posted by GitBox <gi...@apache.org>.
zhoukangcn commented on a change in pull request #1059: KYLIN-4291 Parallel segment building may causes WriteConflictException
URL: https://github.com/apache/kylin/pull/1059#discussion_r366121753
##########
File path: core-dictionary/src/main/java/org/apache/kylin/dict/DictionaryManager.java
##########
@@ -193,12 +194,24 @@ private DictionaryInfo updateExistingDictLastModifiedTime(String dictPath) throw
ResourceStore store = getStore();
if (StringUtils.isBlank(dictPath))
return NONE_INDICATOR;
- long now = System.currentTimeMillis();
- store.updateTimestamp(dictPath, now);
- logger.info("Update dictionary {} lastModifiedTime to {}", dictPath, now);
- DictionaryInfo dictInfo = load(dictPath, true);
- updateDictCache(dictInfo);
- return dictInfo;
+
+ int retry = 7;
Review comment:
I think change constant to KylinConfig is a better way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] zhangayqian commented on a change in pull request #1059:
KYLIN-4291 Parallel segment building may causes WriteConflictException
Posted by GitBox <gi...@apache.org>.
zhangayqian commented on a change in pull request #1059: KYLIN-4291 Parallel segment building may causes WriteConflictException
URL: https://github.com/apache/kylin/pull/1059#discussion_r366260366
##########
File path: core-dictionary/src/main/java/org/apache/kylin/dict/DictionaryManager.java
##########
@@ -193,12 +194,24 @@ private DictionaryInfo updateExistingDictLastModifiedTime(String dictPath) throw
ResourceStore store = getStore();
if (StringUtils.isBlank(dictPath))
return NONE_INDICATOR;
- long now = System.currentTimeMillis();
- store.updateTimestamp(dictPath, now);
- logger.info("Update dictionary {} lastModifiedTime to {}", dictPath, now);
- DictionaryInfo dictInfo = load(dictPath, true);
- updateDictCache(dictInfo);
- return dictInfo;
+
+ int retry = 7;
Review comment:
Great advice! Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] coveralls commented on issue #1059: KYLIN-4291 Parallel
segment building may causes WriteConflictException
Posted by GitBox <gi...@apache.org>.
coveralls commented on issue #1059: KYLIN-4291 Parallel segment building may causes WriteConflictException
URL: https://github.com/apache/kylin/pull/1059#issuecomment-573970445
## Pull Request Test Coverage Report for [Build 5518](https://coveralls.io/builds/28085746)
* **15** of **28** **(53.57%)** changed or added relevant lines in **2** files are covered.
* **20** unchanged lines in **5** files lost coverage.
* Overall coverage decreased (**-0.02%**) to **27.509%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/SnapshotManager.java](https://coveralls.io/builds/28085746/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2FSnapshotManager.java#L238) | 6 | 12 | 50.0%
| [core-dictionary/src/main/java/org/apache/kylin/dict/DictionaryManager.java](https://coveralls.io/builds/28085746/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2FDictionaryManager.java#L205) | 9 | 16 | 56.25%
<!-- | **Total:** | **15** | **28** | **53.57%** | -->
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/SnapshotManager.java](https://coveralls.io/builds/28085746/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2FSnapshotManager.java#L248) | 1 | 71.07% |
| [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/28085746/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
| [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/28085746/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% |
| [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/ColumnarStoreCache.java](https://coveralls.io/builds/28085746/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FColumnarStoreCache.java#L75) | 5 | 54.55% |
| [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/FragmentData.java](https://coveralls.io/builds/28085746/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FFragmentData.java#L99) | 7 | 75.68% |
<!-- | **Total:** | **20** | | -->
| Totals | [](https://coveralls.io/builds/28085746) |
| :-- | --: |
| Change from base [Build 5512](https://coveralls.io/builds/28061631): | -0.02% |
| Covered Lines: | 24123 |
| Relevant Lines: | 87692 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services