You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Enis Söztutar <en...@apache.org> on 2014/06/28 01:37:53 UTC
[NOTICE] No checkins to master for 1-2 hours please
Hi,
I am merging the hbase-10070 branch to master. No checkins please.
We were able to get unit tests working, and IT's working. There may be a
couple of edge cases after rebase, but we can resolve them in master.
I'll send a detailed update once done.
Enis
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Ted Yu <yu...@gmail.com>.
You're welcome.
Fix is attached to HBASE-11443
Cheers
On Mon, Jun 30, 2014 at 3:13 PM, Enis Söztutar <en...@gmail.com> wrote:
> I'm taking a look at the TestIOFencing. I can repro locally. Thanks Ted for
> reporting.
>
> Enis
>
>
> On Fri, Jun 27, 2014 at 8:16 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Enis:
> > I ran test suite on Linux and found that TestIOFencing hung.
> >
> > jstack is here:
> > http://people.apache.org/~tedyu/TestIOFencing.stack
> >
> > Unfortunately there was no test output.
> >
> > Will continue digging in the coming days.
> >
> > Cheers
> >
> >
> > On Fri, Jun 27, 2014 at 7:32 PM, Enis Söztutar <en...@apache.org> wrote:
> >
> > > Update:
> > >
> > > I've completed the merge for branch hbase-10070 to master now. As a
> short
> > > reminder the committed changes include phase 1 work for the new feature
> > > called "read high availability using timeline consistent reads with
> > region
> > > replicas". Some more context can be found in [1] and vote thread here
> > [2].
> > >
> > > The merge contains 44 commits in total as follows. I've uploaded the
> > > committed patches to relevant jiras, and set the fix versions for the
> > jiras
> > > as well (using git format-patch and the excellent jira-cli client).
> > >
> > > Applying: HBASE-10347 HRegionInfo changes for adding replicaId and
> > > MetaEditor/MetaReader changes for region replicas
> > > Applying: HBASE-10354 Add an API for defining consistency per request
> > > Applying: HBASE-10348. HTableDescriptor changes for region replicas
> > > Applying: HBASE-10517 NPE in MetaCache.clearCache()
> > > Applying: HBASE-10354. Addendum commit. 10354.add.patch committed
> > > Applying: HBASE-10356 Failover RPC's for multi-get
> > > Applying: HBASE-10350. Master/AM/RegionStates changes to create and
> > assign
> > > region replicas (ddas)
> > > Applying: HBASE-10359 Master/RS WebUI changes for region replicas
> > > Applying: HBASE-10361. Enable/AlterTable support for region replicas
> > (ddas)
> > > Applying: HBASE-10352 Region and RegionServer changes for opening
> region
> > > replicas, and refreshing store files
> > > Applying: HBASE-10362. HBCK changes for supporting region replicas
> > > Applying: HBASE-10351 LoadBalancer changes for supporting region
> replicas
> > > Applying: HBASE-10630 NullPointerException in
> > > ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to
> > > missing region info
> > > Applying: HBASE-10633. StoreFileRefresherChore throws
> > > ConcurrentModificationException sometimes
> > > Applying: HBASE-10672. Table snapshot should handle tables whose
> > > REGION_REPLICATION is greater than one
> > > Applying: HBASE-10620. LoadBalancer.needsBalance() should check for
> > > co-located region replicas as well
> > > Applying: HBASE-10355 Failover RPC's from client using region replicas
> > > Applying: HBASE-10704 BaseLoadBalancer#roundRobinAssignment() may add
> > same
> > > region to assignment plan multiple times
> > > Applying: HBASE-10572 Create an IntegrationTest for region replicas
> > > Applying: HBASE-10729. Enable table doesn't balance out replicas evenly
> > if
> > > the replicas were unassigned earlier
> > > Applying: HBASE-10620 LoadBalancer.needsBalance() should check for
> > > co-located region replicas as well (Addendum patch to fix return value
> > from
> > > balance() call)
> > > Applying: HBASE-10734. Fix RegionStates.getRegionAssignments to not add
> > > duplicate regions
> > > Applying: HBASE-10616. Integration test for multi-get calls
> > > Applying: HBASE-10743. Replica map update is problematic in
> RegionStates
> > > Applying: HBASE-10616. Addendum that fixes the case of
> multiget_batchsize
> > > == 1
> > > Applying: HBASE-10778 Unique keys accounting in MultiThreadedReader is
> > > incorrect
> > > Applying: HBASE-10858 TestRegionRebalancing is failing
> > > Applying: HBASE-10817 Add some tests on a real cluster for replica:
> multi
> > > master, replication
> > > Applying: HBASE-10701 Cache invalidation improvements from client side
> > > Applying: HBASE-10859 Use HFileLink in opening region files from
> > > secondaries
> > > Applying: HBASE-10791 Add integration test to demonstrate performance
> > > improvement
> > > Applying: HBASE-10634 Multiget doesn't fully work
> > > Applying: HBASE-10794 multi-get should handle replica location missing
> > from
> > > cache
> > > Applying: HBASE-10785 Metas own location should be cached
> > > Applying: HBASE-10957 HMaster can abort with NPE in #rebuildUserRegions
> > > (Nicolas Liochon)
> > > Applying: HBASE-10661
> > > TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks()
> > is
> > > flaky
> > > Applying: HBASE-10513 Provide user documentation for region replicas
> > > Applying: HBASE-10810 LoadTestTool should share the connection and
> > > connection pool
> > > Applying: HBASE-10357 Failover RPC's for scans (Devaraj Das)
> > > Applying: HBASE-11214. Fixes for scans on a replicated table
> > > Applying: HBASE-10818. Add integration test for bulkload with replicas
> > > (Nick Dimiduk and Devaraj Das)
> > > Applying: HBASE-10818. Addendum to reduce the number of times sleep
> > happens
> > > in the coprocessor. Also, puts back code to set consistency that got
> > > deleted by accident in the last commit.
> > > Applying: HBASE-11332 Fix for metas location cache from HBASE-10785
> > > Applying: Increase -XX:MaxPermSize for unit tests
> > >
> > >
> > > Unit tests should be working now, but in case it fails on the apache
> > infra,
> > > I'll take a closer look at the jenkins build at
> > > https://builds.apache.org/view/All/job/HBase-TRUNK/. We've been
> running
> > > the
> > > integration tests as well but ideally we want to spend some more time
> on
> > > perf regression analysis and long tail latency improvements. We can
> iron
> > > out the edges in time. Nicolas has already been doing work around this.
> > >
> > > Let me know if you see any issues related to this merge.
> > >
> > >
> > > [1]
> > >
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201405.mbox/browser
> > > [2]
> > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201406.mbox/%3CCAMUu0w8uvb29%2BQYpZ8VwSO4qEpKRyKGNPzyj1kw7tLRhLPoZkA%40mail.gmail.com%3E
> > >
> > >
> > >
> > >
> > > On Fri, Jun 27, 2014 at 7:03 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Good job.
> > > >
> > > >
> > > > On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <en...@apache.org>
> > wrote:
> > > >
> > > > > Pushed the rebased branch. You can go crazy on the master now.
> > > > >
> > > > > Enis
> > > > >
> > > > >
> > > > > On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am merging the hbase-10070 branch to master. No checkins
> please.
> > > > > >
> > > > > > We were able to get unit tests working, and IT's working. There
> may
> > > be
> > > > a
> > > > > > couple of edge cases after rebase, but we can resolve them in
> > master.
> > > > > >
> > > > > > I'll send a detailed update once done.
> > > > > >
> > > > > > Enis
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Enis Söztutar <en...@gmail.com>.
I'm taking a look at the TestIOFencing. I can repro locally. Thanks Ted for
reporting.
Enis
On Fri, Jun 27, 2014 at 8:16 PM, Ted Yu <yu...@gmail.com> wrote:
> Enis:
> I ran test suite on Linux and found that TestIOFencing hung.
>
> jstack is here:
> http://people.apache.org/~tedyu/TestIOFencing.stack
>
> Unfortunately there was no test output.
>
> Will continue digging in the coming days.
>
> Cheers
>
>
> On Fri, Jun 27, 2014 at 7:32 PM, Enis Söztutar <en...@apache.org> wrote:
>
> > Update:
> >
> > I've completed the merge for branch hbase-10070 to master now. As a short
> > reminder the committed changes include phase 1 work for the new feature
> > called "read high availability using timeline consistent reads with
> region
> > replicas". Some more context can be found in [1] and vote thread here
> [2].
> >
> > The merge contains 44 commits in total as follows. I've uploaded the
> > committed patches to relevant jiras, and set the fix versions for the
> jiras
> > as well (using git format-patch and the excellent jira-cli client).
> >
> > Applying: HBASE-10347 HRegionInfo changes for adding replicaId and
> > MetaEditor/MetaReader changes for region replicas
> > Applying: HBASE-10354 Add an API for defining consistency per request
> > Applying: HBASE-10348. HTableDescriptor changes for region replicas
> > Applying: HBASE-10517 NPE in MetaCache.clearCache()
> > Applying: HBASE-10354. Addendum commit. 10354.add.patch committed
> > Applying: HBASE-10356 Failover RPC's for multi-get
> > Applying: HBASE-10350. Master/AM/RegionStates changes to create and
> assign
> > region replicas (ddas)
> > Applying: HBASE-10359 Master/RS WebUI changes for region replicas
> > Applying: HBASE-10361. Enable/AlterTable support for region replicas
> (ddas)
> > Applying: HBASE-10352 Region and RegionServer changes for opening region
> > replicas, and refreshing store files
> > Applying: HBASE-10362. HBCK changes for supporting region replicas
> > Applying: HBASE-10351 LoadBalancer changes for supporting region replicas
> > Applying: HBASE-10630 NullPointerException in
> > ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to
> > missing region info
> > Applying: HBASE-10633. StoreFileRefresherChore throws
> > ConcurrentModificationException sometimes
> > Applying: HBASE-10672. Table snapshot should handle tables whose
> > REGION_REPLICATION is greater than one
> > Applying: HBASE-10620. LoadBalancer.needsBalance() should check for
> > co-located region replicas as well
> > Applying: HBASE-10355 Failover RPC's from client using region replicas
> > Applying: HBASE-10704 BaseLoadBalancer#roundRobinAssignment() may add
> same
> > region to assignment plan multiple times
> > Applying: HBASE-10572 Create an IntegrationTest for region replicas
> > Applying: HBASE-10729. Enable table doesn't balance out replicas evenly
> if
> > the replicas were unassigned earlier
> > Applying: HBASE-10620 LoadBalancer.needsBalance() should check for
> > co-located region replicas as well (Addendum patch to fix return value
> from
> > balance() call)
> > Applying: HBASE-10734. Fix RegionStates.getRegionAssignments to not add
> > duplicate regions
> > Applying: HBASE-10616. Integration test for multi-get calls
> > Applying: HBASE-10743. Replica map update is problematic in RegionStates
> > Applying: HBASE-10616. Addendum that fixes the case of multiget_batchsize
> > == 1
> > Applying: HBASE-10778 Unique keys accounting in MultiThreadedReader is
> > incorrect
> > Applying: HBASE-10858 TestRegionRebalancing is failing
> > Applying: HBASE-10817 Add some tests on a real cluster for replica: multi
> > master, replication
> > Applying: HBASE-10701 Cache invalidation improvements from client side
> > Applying: HBASE-10859 Use HFileLink in opening region files from
> > secondaries
> > Applying: HBASE-10791 Add integration test to demonstrate performance
> > improvement
> > Applying: HBASE-10634 Multiget doesn't fully work
> > Applying: HBASE-10794 multi-get should handle replica location missing
> from
> > cache
> > Applying: HBASE-10785 Metas own location should be cached
> > Applying: HBASE-10957 HMaster can abort with NPE in #rebuildUserRegions
> > (Nicolas Liochon)
> > Applying: HBASE-10661
> > TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks()
> is
> > flaky
> > Applying: HBASE-10513 Provide user documentation for region replicas
> > Applying: HBASE-10810 LoadTestTool should share the connection and
> > connection pool
> > Applying: HBASE-10357 Failover RPC's for scans (Devaraj Das)
> > Applying: HBASE-11214. Fixes for scans on a replicated table
> > Applying: HBASE-10818. Add integration test for bulkload with replicas
> > (Nick Dimiduk and Devaraj Das)
> > Applying: HBASE-10818. Addendum to reduce the number of times sleep
> happens
> > in the coprocessor. Also, puts back code to set consistency that got
> > deleted by accident in the last commit.
> > Applying: HBASE-11332 Fix for metas location cache from HBASE-10785
> > Applying: Increase -XX:MaxPermSize for unit tests
> >
> >
> > Unit tests should be working now, but in case it fails on the apache
> infra,
> > I'll take a closer look at the jenkins build at
> > https://builds.apache.org/view/All/job/HBase-TRUNK/. We've been running
> > the
> > integration tests as well but ideally we want to spend some more time on
> > perf regression analysis and long tail latency improvements. We can iron
> > out the edges in time. Nicolas has already been doing work around this.
> >
> > Let me know if you see any issues related to this merge.
> >
> >
> > [1]
> > https://mail-archives.apache.org/mod_mbox/hbase-dev/201405.mbox/browser
> > [2]
> >
> >
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201406.mbox/%3CCAMUu0w8uvb29%2BQYpZ8VwSO4qEpKRyKGNPzyj1kw7tLRhLPoZkA%40mail.gmail.com%3E
> >
> >
> >
> >
> > On Fri, Jun 27, 2014 at 7:03 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Good job.
> > >
> > >
> > > On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <en...@apache.org>
> wrote:
> > >
> > > > Pushed the rebased branch. You can go crazy on the master now.
> > > >
> > > > Enis
> > > >
> > > >
> > > > On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am merging the hbase-10070 branch to master. No checkins please.
> > > > >
> > > > > We were able to get unit tests working, and IT's working. There may
> > be
> > > a
> > > > > couple of edge cases after rebase, but we can resolve them in
> master.
> > > > >
> > > > > I'll send a detailed update once done.
> > > > >
> > > > > Enis
> > > > >
> > > >
> > >
> >
>
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Ted Yu <yu...@gmail.com>.
Enis:
I ran test suite on Linux and found that TestIOFencing hung.
jstack is here:
http://people.apache.org/~tedyu/TestIOFencing.stack
Unfortunately there was no test output.
Will continue digging in the coming days.
Cheers
On Fri, Jun 27, 2014 at 7:32 PM, Enis Söztutar <en...@apache.org> wrote:
> Update:
>
> I've completed the merge for branch hbase-10070 to master now. As a short
> reminder the committed changes include phase 1 work for the new feature
> called "read high availability using timeline consistent reads with region
> replicas". Some more context can be found in [1] and vote thread here [2].
>
> The merge contains 44 commits in total as follows. I've uploaded the
> committed patches to relevant jiras, and set the fix versions for the jiras
> as well (using git format-patch and the excellent jira-cli client).
>
> Applying: HBASE-10347 HRegionInfo changes for adding replicaId and
> MetaEditor/MetaReader changes for region replicas
> Applying: HBASE-10354 Add an API for defining consistency per request
> Applying: HBASE-10348. HTableDescriptor changes for region replicas
> Applying: HBASE-10517 NPE in MetaCache.clearCache()
> Applying: HBASE-10354. Addendum commit. 10354.add.patch committed
> Applying: HBASE-10356 Failover RPC's for multi-get
> Applying: HBASE-10350. Master/AM/RegionStates changes to create and assign
> region replicas (ddas)
> Applying: HBASE-10359 Master/RS WebUI changes for region replicas
> Applying: HBASE-10361. Enable/AlterTable support for region replicas (ddas)
> Applying: HBASE-10352 Region and RegionServer changes for opening region
> replicas, and refreshing store files
> Applying: HBASE-10362. HBCK changes for supporting region replicas
> Applying: HBASE-10351 LoadBalancer changes for supporting region replicas
> Applying: HBASE-10630 NullPointerException in
> ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to
> missing region info
> Applying: HBASE-10633. StoreFileRefresherChore throws
> ConcurrentModificationException sometimes
> Applying: HBASE-10672. Table snapshot should handle tables whose
> REGION_REPLICATION is greater than one
> Applying: HBASE-10620. LoadBalancer.needsBalance() should check for
> co-located region replicas as well
> Applying: HBASE-10355 Failover RPC's from client using region replicas
> Applying: HBASE-10704 BaseLoadBalancer#roundRobinAssignment() may add same
> region to assignment plan multiple times
> Applying: HBASE-10572 Create an IntegrationTest for region replicas
> Applying: HBASE-10729. Enable table doesn't balance out replicas evenly if
> the replicas were unassigned earlier
> Applying: HBASE-10620 LoadBalancer.needsBalance() should check for
> co-located region replicas as well (Addendum patch to fix return value from
> balance() call)
> Applying: HBASE-10734. Fix RegionStates.getRegionAssignments to not add
> duplicate regions
> Applying: HBASE-10616. Integration test for multi-get calls
> Applying: HBASE-10743. Replica map update is problematic in RegionStates
> Applying: HBASE-10616. Addendum that fixes the case of multiget_batchsize
> == 1
> Applying: HBASE-10778 Unique keys accounting in MultiThreadedReader is
> incorrect
> Applying: HBASE-10858 TestRegionRebalancing is failing
> Applying: HBASE-10817 Add some tests on a real cluster for replica: multi
> master, replication
> Applying: HBASE-10701 Cache invalidation improvements from client side
> Applying: HBASE-10859 Use HFileLink in opening region files from
> secondaries
> Applying: HBASE-10791 Add integration test to demonstrate performance
> improvement
> Applying: HBASE-10634 Multiget doesn't fully work
> Applying: HBASE-10794 multi-get should handle replica location missing from
> cache
> Applying: HBASE-10785 Metas own location should be cached
> Applying: HBASE-10957 HMaster can abort with NPE in #rebuildUserRegions
> (Nicolas Liochon)
> Applying: HBASE-10661
> TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is
> flaky
> Applying: HBASE-10513 Provide user documentation for region replicas
> Applying: HBASE-10810 LoadTestTool should share the connection and
> connection pool
> Applying: HBASE-10357 Failover RPC's for scans (Devaraj Das)
> Applying: HBASE-11214. Fixes for scans on a replicated table
> Applying: HBASE-10818. Add integration test for bulkload with replicas
> (Nick Dimiduk and Devaraj Das)
> Applying: HBASE-10818. Addendum to reduce the number of times sleep happens
> in the coprocessor. Also, puts back code to set consistency that got
> deleted by accident in the last commit.
> Applying: HBASE-11332 Fix for metas location cache from HBASE-10785
> Applying: Increase -XX:MaxPermSize for unit tests
>
>
> Unit tests should be working now, but in case it fails on the apache infra,
> I'll take a closer look at the jenkins build at
> https://builds.apache.org/view/All/job/HBase-TRUNK/. We've been running
> the
> integration tests as well but ideally we want to spend some more time on
> perf regression analysis and long tail latency improvements. We can iron
> out the edges in time. Nicolas has already been doing work around this.
>
> Let me know if you see any issues related to this merge.
>
>
> [1]
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201405.mbox/browser
> [2]
>
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201406.mbox/%3CCAMUu0w8uvb29%2BQYpZ8VwSO4qEpKRyKGNPzyj1kw7tLRhLPoZkA%40mail.gmail.com%3E
>
>
>
>
> On Fri, Jun 27, 2014 at 7:03 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Good job.
> >
> >
> > On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <en...@apache.org> wrote:
> >
> > > Pushed the rebased branch. You can go crazy on the master now.
> > >
> > > Enis
> > >
> > >
> > > On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am merging the hbase-10070 branch to master. No checkins please.
> > > >
> > > > We were able to get unit tests working, and IT's working. There may
> be
> > a
> > > > couple of edge cases after rebase, but we can resolve them in master.
> > > >
> > > > I'll send a detailed update once done.
> > > >
> > > > Enis
> > > >
> > >
> >
>
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Stack <st...@duboce.net>.
Nice work Enis.
St.Ack
On Fri, Jun 27, 2014 at 7:32 PM, Enis Söztutar <en...@apache.org> wrote:
> Update:
>
> I've completed the merge for branch hbase-10070 to master now. As a short
> reminder the committed changes include phase 1 work for the new feature
> called "read high availability using timeline consistent reads with region
> replicas". Some more context can be found in [1] and vote thread here [2].
>
> The merge contains 44 commits in total as follows. I've uploaded the
> committed patches to relevant jiras, and set the fix versions for the jiras
> as well (using git format-patch and the excellent jira-cli client).
>
> Applying: HBASE-10347 HRegionInfo changes for adding replicaId and
> MetaEditor/MetaReader changes for region replicas
> Applying: HBASE-10354 Add an API for defining consistency per request
> Applying: HBASE-10348. HTableDescriptor changes for region replicas
> Applying: HBASE-10517 NPE in MetaCache.clearCache()
> Applying: HBASE-10354. Addendum commit. 10354.add.patch committed
> Applying: HBASE-10356 Failover RPC's for multi-get
> Applying: HBASE-10350. Master/AM/RegionStates changes to create and assign
> region replicas (ddas)
> Applying: HBASE-10359 Master/RS WebUI changes for region replicas
> Applying: HBASE-10361. Enable/AlterTable support for region replicas (ddas)
> Applying: HBASE-10352 Region and RegionServer changes for opening region
> replicas, and refreshing store files
> Applying: HBASE-10362. HBCK changes for supporting region replicas
> Applying: HBASE-10351 LoadBalancer changes for supporting region replicas
> Applying: HBASE-10630 NullPointerException in
> ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to
> missing region info
> Applying: HBASE-10633. StoreFileRefresherChore throws
> ConcurrentModificationException sometimes
> Applying: HBASE-10672. Table snapshot should handle tables whose
> REGION_REPLICATION is greater than one
> Applying: HBASE-10620. LoadBalancer.needsBalance() should check for
> co-located region replicas as well
> Applying: HBASE-10355 Failover RPC's from client using region replicas
> Applying: HBASE-10704 BaseLoadBalancer#roundRobinAssignment() may add same
> region to assignment plan multiple times
> Applying: HBASE-10572 Create an IntegrationTest for region replicas
> Applying: HBASE-10729. Enable table doesn't balance out replicas evenly if
> the replicas were unassigned earlier
> Applying: HBASE-10620 LoadBalancer.needsBalance() should check for
> co-located region replicas as well (Addendum patch to fix return value from
> balance() call)
> Applying: HBASE-10734. Fix RegionStates.getRegionAssignments to not add
> duplicate regions
> Applying: HBASE-10616. Integration test for multi-get calls
> Applying: HBASE-10743. Replica map update is problematic in RegionStates
> Applying: HBASE-10616. Addendum that fixes the case of multiget_batchsize
> == 1
> Applying: HBASE-10778 Unique keys accounting in MultiThreadedReader is
> incorrect
> Applying: HBASE-10858 TestRegionRebalancing is failing
> Applying: HBASE-10817 Add some tests on a real cluster for replica: multi
> master, replication
> Applying: HBASE-10701 Cache invalidation improvements from client side
> Applying: HBASE-10859 Use HFileLink in opening region files from
> secondaries
> Applying: HBASE-10791 Add integration test to demonstrate performance
> improvement
> Applying: HBASE-10634 Multiget doesn't fully work
> Applying: HBASE-10794 multi-get should handle replica location missing from
> cache
> Applying: HBASE-10785 Metas own location should be cached
> Applying: HBASE-10957 HMaster can abort with NPE in #rebuildUserRegions
> (Nicolas Liochon)
> Applying: HBASE-10661
> TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is
> flaky
> Applying: HBASE-10513 Provide user documentation for region replicas
> Applying: HBASE-10810 LoadTestTool should share the connection and
> connection pool
> Applying: HBASE-10357 Failover RPC's for scans (Devaraj Das)
> Applying: HBASE-11214. Fixes for scans on a replicated table
> Applying: HBASE-10818. Add integration test for bulkload with replicas
> (Nick Dimiduk and Devaraj Das)
> Applying: HBASE-10818. Addendum to reduce the number of times sleep happens
> in the coprocessor. Also, puts back code to set consistency that got
> deleted by accident in the last commit.
> Applying: HBASE-11332 Fix for metas location cache from HBASE-10785
> Applying: Increase -XX:MaxPermSize for unit tests
>
>
> Unit tests should be working now, but in case it fails on the apache infra,
> I'll take a closer look at the jenkins build at
> https://builds.apache.org/view/All/job/HBase-TRUNK/. We've been running
> the
> integration tests as well but ideally we want to spend some more time on
> perf regression analysis and long tail latency improvements. We can iron
> out the edges in time. Nicolas has already been doing work around this.
>
> Let me know if you see any issues related to this merge.
>
>
> [1]
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201405.mbox/browser
> [2]
>
> https://mail-archives.apache.org/mod_mbox/hbase-dev/201406.mbox/%3CCAMUu0w8uvb29%2BQYpZ8VwSO4qEpKRyKGNPzyj1kw7tLRhLPoZkA%40mail.gmail.com%3E
>
>
>
>
> On Fri, Jun 27, 2014 at 7:03 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Good job.
> >
> >
> > On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <en...@apache.org> wrote:
> >
> > > Pushed the rebased branch. You can go crazy on the master now.
> > >
> > > Enis
> > >
> > >
> > > On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am merging the hbase-10070 branch to master. No checkins please.
> > > >
> > > > We were able to get unit tests working, and IT's working. There may
> be
> > a
> > > > couple of edge cases after rebase, but we can resolve them in master.
> > > >
> > > > I'll send a detailed update once done.
> > > >
> > > > Enis
> > > >
> > >
> >
>
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Enis Söztutar <en...@apache.org>.
Update:
I've completed the merge for branch hbase-10070 to master now. As a short
reminder the committed changes include phase 1 work for the new feature
called "read high availability using timeline consistent reads with region
replicas". Some more context can be found in [1] and vote thread here [2].
The merge contains 44 commits in total as follows. I've uploaded the
committed patches to relevant jiras, and set the fix versions for the jiras
as well (using git format-patch and the excellent jira-cli client).
Applying: HBASE-10347 HRegionInfo changes for adding replicaId and
MetaEditor/MetaReader changes for region replicas
Applying: HBASE-10354 Add an API for defining consistency per request
Applying: HBASE-10348. HTableDescriptor changes for region replicas
Applying: HBASE-10517 NPE in MetaCache.clearCache()
Applying: HBASE-10354. Addendum commit. 10354.add.patch committed
Applying: HBASE-10356 Failover RPC's for multi-get
Applying: HBASE-10350. Master/AM/RegionStates changes to create and assign
region replicas (ddas)
Applying: HBASE-10359 Master/RS WebUI changes for region replicas
Applying: HBASE-10361. Enable/AlterTable support for region replicas (ddas)
Applying: HBASE-10352 Region and RegionServer changes for opening region
replicas, and refreshing store files
Applying: HBASE-10362. HBCK changes for supporting region replicas
Applying: HBASE-10351 LoadBalancer changes for supporting region replicas
Applying: HBASE-10630 NullPointerException in
ConnectionManager$HConnectionImplementation.locateRegionInMeta() due to
missing region info
Applying: HBASE-10633. StoreFileRefresherChore throws
ConcurrentModificationException sometimes
Applying: HBASE-10672. Table snapshot should handle tables whose
REGION_REPLICATION is greater than one
Applying: HBASE-10620. LoadBalancer.needsBalance() should check for
co-located region replicas as well
Applying: HBASE-10355 Failover RPC's from client using region replicas
Applying: HBASE-10704 BaseLoadBalancer#roundRobinAssignment() may add same
region to assignment plan multiple times
Applying: HBASE-10572 Create an IntegrationTest for region replicas
Applying: HBASE-10729. Enable table doesn't balance out replicas evenly if
the replicas were unassigned earlier
Applying: HBASE-10620 LoadBalancer.needsBalance() should check for
co-located region replicas as well (Addendum patch to fix return value from
balance() call)
Applying: HBASE-10734. Fix RegionStates.getRegionAssignments to not add
duplicate regions
Applying: HBASE-10616. Integration test for multi-get calls
Applying: HBASE-10743. Replica map update is problematic in RegionStates
Applying: HBASE-10616. Addendum that fixes the case of multiget_batchsize
== 1
Applying: HBASE-10778 Unique keys accounting in MultiThreadedReader is
incorrect
Applying: HBASE-10858 TestRegionRebalancing is failing
Applying: HBASE-10817 Add some tests on a real cluster for replica: multi
master, replication
Applying: HBASE-10701 Cache invalidation improvements from client side
Applying: HBASE-10859 Use HFileLink in opening region files from secondaries
Applying: HBASE-10791 Add integration test to demonstrate performance
improvement
Applying: HBASE-10634 Multiget doesn't fully work
Applying: HBASE-10794 multi-get should handle replica location missing from
cache
Applying: HBASE-10785 Metas own location should be cached
Applying: HBASE-10957 HMaster can abort with NPE in #rebuildUserRegions
(Nicolas Liochon)
Applying: HBASE-10661
TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is
flaky
Applying: HBASE-10513 Provide user documentation for region replicas
Applying: HBASE-10810 LoadTestTool should share the connection and
connection pool
Applying: HBASE-10357 Failover RPC's for scans (Devaraj Das)
Applying: HBASE-11214. Fixes for scans on a replicated table
Applying: HBASE-10818. Add integration test for bulkload with replicas
(Nick Dimiduk and Devaraj Das)
Applying: HBASE-10818. Addendum to reduce the number of times sleep happens
in the coprocessor. Also, puts back code to set consistency that got
deleted by accident in the last commit.
Applying: HBASE-11332 Fix for metas location cache from HBASE-10785
Applying: Increase -XX:MaxPermSize for unit tests
Unit tests should be working now, but in case it fails on the apache infra,
I'll take a closer look at the jenkins build at
https://builds.apache.org/view/All/job/HBase-TRUNK/. We've been running the
integration tests as well but ideally we want to spend some more time on
perf regression analysis and long tail latency improvements. We can iron
out the edges in time. Nicolas has already been doing work around this.
Let me know if you see any issues related to this merge.
[1] https://mail-archives.apache.org/mod_mbox/hbase-dev/201405.mbox/browser
[2]
https://mail-archives.apache.org/mod_mbox/hbase-dev/201406.mbox/%3CCAMUu0w8uvb29%2BQYpZ8VwSO4qEpKRyKGNPzyj1kw7tLRhLPoZkA%40mail.gmail.com%3E
On Fri, Jun 27, 2014 at 7:03 PM, Ted Yu <yu...@gmail.com> wrote:
> Good job.
>
>
> On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <en...@apache.org> wrote:
>
> > Pushed the rebased branch. You can go crazy on the master now.
> >
> > Enis
> >
> >
> > On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > I am merging the hbase-10070 branch to master. No checkins please.
> > >
> > > We were able to get unit tests working, and IT's working. There may be
> a
> > > couple of edge cases after rebase, but we can resolve them in master.
> > >
> > > I'll send a detailed update once done.
> > >
> > > Enis
> > >
> >
>
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Ted Yu <yu...@gmail.com>.
Good job.
On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <en...@apache.org> wrote:
> Pushed the rebased branch. You can go crazy on the master now.
>
> Enis
>
>
> On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org> wrote:
>
> > Hi,
> >
> > I am merging the hbase-10070 branch to master. No checkins please.
> >
> > We were able to get unit tests working, and IT's working. There may be a
> > couple of edge cases after rebase, but we can resolve them in master.
> >
> > I'll send a detailed update once done.
> >
> > Enis
> >
>
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Andrew Purtell <ap...@apache.org>.
Sweet!
On Fri, Jun 27, 2014 at 5:45 PM, Enis Söztutar <en...@apache.org> wrote:
> Pushed the rebased branch. You can go crazy on the master now.
>
> Enis
>
>
> On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org> wrote:
>
> > Hi,
> >
> > I am merging the hbase-10070 branch to master. No checkins please.
> >
> > We were able to get unit tests working, and IT's working. There may be a
> > couple of edge cases after rebase, but we can resolve them in master.
> >
> > I'll send a detailed update once done.
> >
> > Enis
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [NOTICE] No checkins to master for 1-2 hours please
Posted by Enis Söztutar <en...@apache.org>.
Pushed the rebased branch. You can go crazy on the master now.
Enis
On Fri, Jun 27, 2014 at 4:37 PM, Enis Söztutar <en...@apache.org> wrote:
> Hi,
>
> I am merging the hbase-10070 branch to master. No checkins please.
>
> We were able to get unit tests working, and IT's working. There may be a
> couple of edge cases after rebase, but we can resolve them in master.
>
> I'll send a detailed update once done.
>
> Enis
>