You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@distributedlog.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/12/21 17:26:58 UTC
[jira] [Commented] (DL-99) Enable thrift mux on DL smoketest reader
[ https://issues.apache.org/jira/browse/DL-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767616#comment-15767616 ]
ASF GitHub Bot commented on DL-99:
----------------------------------
GitHub user sijie opened a pull request:
https://github.com/apache/incubator-distributedlog/pull/82
DL-99 - DL-111
This is a umbrella merge for DL-99 to DL-111. Individual commit in this pull request is associated with one DL jira.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sijie/incubator-distributedlog squashed_merge/DL-111
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-distributedlog/pull/82.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #82
----
commit 9ff22f4239754d25fb3c7238cf25cf453f0ebbd6
Author: Dave Rusek <dr...@apache.org>
Date: 2016-07-18T19:23:49Z
DL-99: Enable thriftmux for distributedlog smoketest reader
commit 785ff1cdfb515882f934d4bd79fe34bb39cd95a9
Author: Sijie Guo <si...@twitter.com>
Date: 2016-07-29T04:10:33Z
DL-101: Improve session expire handling on fetching log segments for readers
This change focuses on improving the session expire handling on fetching log segments for readers.
The log segment management in DL is now done by 3 parts.
- a LogSegmentMetadataStore (one per namespace instance): it is used for fetching the log segments from log segment metadata store (ZooKeeper). it doesn't do any caching.
- a LogSegmentMetadataCache (one per namespace instance): it is a guava cache based metadata cache. it maintains a mapping between log segment metadata path and the log segment metadata. it manages the cache for the log segments that will be accessed in this namespace instance. it doesn't manage the sequence of the log segments for streams.
- a PerStreamLogSegmentCache for each BKLogHandler. the log segment cache is per stream. it maintains the sequence of the log segments.
BKLogWriteHandler doesn't watch the log segment changes. It fetches minimal number of log segments when it is created and fetches the full list of log segments for truncations. New log segments will be added to the per stream log segment cache with log segment rolling.
BKLogReadHandler watch the log segments changes and only notify when the list of log segments is changed. the session handling which is specific to the metadata store is hidden to the implementations of LogSegmentMetadataStore.
The change tries to cleanup bunch of unused methods in BKLog{Read,Write}Handler too.
commit 10ff44510437d4087df5dc4775d968e174df83b0
Author: Leigh Stewart <ls...@apache.org>
Date: 2016-07-29T04:36:49Z
DL-102: Add routing service to write proxy server side
this change is to add getOwner rpc in write proxy. so we can change the client side to get owner from write proxy first for routing service. in this way, we can start experiementing any resource placement algorithms.
commit 9f80812ab4293de2f7073642f7ef452b6a88f8f6
Author: Sijie Guo <si...@twitter.com>
Date: 2016-07-29T05:02:09Z
DL-103: DL client can use server-side routing service to retrieve ownerships for streams
If the dl client configures a routing service finagle name, the client will use this to retrieve ownerships for streams.
commit 75774256dbaa5fcccb2e0d319e5d7d272a976901
Author: Jordan Bull <jb...@twitter.com>
Date: 2016-08-09T18:36:53Z
DL-104: BKClientReadTimeout can be equal to getReadLACLongPollTimeout so current config validates
commit afe64639b871b4dbc564f904fc4f522c0de334d3
Author: Yiming Zang <yz...@twitter.com>
Date: 2016-08-10T23:18:53Z
DL-105: Make compression stats available per stream
commit 36a07f6b1d5d3fde395687f97ee10fb9376b5d7b
Author: Yiming Zang <yz...@twitter.com>
Date: 2016-08-12T17:30:39Z
DL-106: Use namespace after it is closed will throw AlreadyClosedException
commit d156c1d6223034fceba90750febc42d25e389d5c
Author: Phillip Su <ps...@twitter.com>
Date: 2016-08-23T00:47:28Z
DL-107: Added unregistering gauges for distributedlog-core and distributedlog-benchmark
commit e18cd48c2f9a9d03dfc7f46e96e74ebaa1471659
Author: Yiming Zang <yz...@twitter.com>
Date: 2016-09-17T00:35:51Z
DL-108: Log rate limiting more clearly
commit a743b44f4614154c67cc4d6a98a9003d80062d70
Author: Sijie Guo <si...@twitter.com>
Date: 2016-10-27T17:18:52Z
DL-109: Add a tool to find the stream containing the given ledger
commit 5f50a41b3883e2107befa95f774687deae9d62fe
Author: Sijie Guo <si...@twitter.com>
Date: 2016-11-22T00:17:51Z
DL-110: Write control record if necessary when roll a log segment
commit 0d2e20c7c9189dec7909ae67f4a62443a9aa40bc
Author: Sijie Guo <si...@twitter.com>
Date: 2016-11-22T01:30:13Z
DL-111: ReadAhead Cache should cache entries rather than records
Current readahead cache cache records. So it will be a lot of callbacks (function calls) when polling a record off the readahead cache. Most of the cpu cycles are unnecessarily spent
on function calls on polling records off the readahead cache. It is the throughput bottleneck for a DL reader.
This change is to change ReadAhead cache to cache entries rather than records. Defer the deserilization of records later on when the reader wants to access the records. It also make
the cache more efficient to reduce the memory footprint.
commit 63b62709e982a8cdabfa0b15ae88c956e71606d0
Author: Sijie Guo <si...@twitter.com>
Date: 2016-11-22T01:35:40Z
DL-112: Improve SyncReader to access readahead cache directly
This changes the implementation of SyncReader to let it access the read ahead cache directly, rather
than calling AsyncReader to fetch the entries. It would reduce the function calls that SyncReader spends
on reading individual records.
commit 27157ec59d30e3e519e5bdef9dfb157c5185ffea
Author: Sijie Guo <si...@twitter.com>
Date: 2016-11-22T01:41:26Z
DL-113: Improve the sync reader benchmark
Add more logging information for calcuating the stats for catching up reads
----
> Enable thrift mux on DL smoketest reader
> ----------------------------------------
>
> Key: DL-99
> URL: https://issues.apache.org/jira/browse/DL-99
> Project: DistributedLog
> Issue Type: Bug
> Components: distributedlog-benchmark
> Reporter: Sijie Guo
> Fix For: 0.4.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)