You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/06/12 23:26:00 UTC
[jira] [Commented] (GEODE-292) Optimize speed of backup/ restore
cache snapshots
[ https://issues.apache.org/jira/browse/GEODE-292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047214#comment-16047214 ]
ASF GitHub Bot commented on GEODE-292:
--------------------------------------
GitHub user PurelyApplied opened a pull request:
https://github.com/apache/geode/pull/576
Geode 2920, 2921, 2922, 2924
This PR is intended for initial review, not to be actively considered for acceptance. Sufficient testing is still required and `develop` shifted enough today that I need to rebase again.
This currently addresses most of the finer-grained security as listed [here](https://cwiki.apache.org/confluence/display/GEODE/Finer+grained+security). As yet unaddressed remains:
* alter disk-store
* disconnect
* echo
* encrypt password
* execute function
* import cluster-configuration
* DistributedSystemMXBean.backupAllMembers
Additional points of concern:
* compact offline-disk-store is not actually updated? [None -> None]
* Should destroy region also require CLUSTER:WRITE:DISK if persistent?
* These do not exist:
* GatewayReceiverMXBean.pause DATA:MANAGE CLUSTER:MANAGE:GATEWAY
* GatewayReceiverMXBean.rebalance DATA:MANAGE CLUSTER:MANAGE:GATEWAY
* GatewayReceiverMXBean.resume DATA:MANAGE CLUSTER:MANAGE:GATEWAY
* `execute function` is listed twice, with different "original" permissions. Do these refer to different executions?
For your convenience, the functional diff is the second commit. The first commit only resolves `imports`, and the third commit is a general cleanup of touched files.
======
Current commit log:
GEODE-292*: Migration of security from DATA:MANAGE
* DATA:MANAGE -> CLUSTER:MANAGE
*
* configure pdx
* LockServiceMXBean.becomeLockGrantor
*
* DATA:MANAGE -> CLUSTER:MANAGE:DISK
*
* compact disk-store
* create disk-store
* destroy disk-store
* revoke missing-disk-store
* DiskStoreMXBean.forceRoll
* DiskStoreMXBean.forceCompaction
* DiskStoreMXBean.flush
* DiskStoreMXBean.setDiskUsageWarningPercentage
* DiskStoreMXBean.setDiskUsageCriticalPercentage
* DistributedSystemMXBean.revokeMissingDiskStores
* MemberMXBean.compactAllDistStores
*
* DATA:MANAGE -> CLUSTER:MANAGE:GATEWAY
*
* create gateway-receiver
* create gateway-sender
* destroy gateway-sender
* load-balance gateway-sender
* pause gateway-sender
* resume gateway-sender
* start gateway-receiver
* start gateway-sender
* stop gateway-receiver
* stop gateway-sender
* GatewayReceiverMXBean.start
* GatewayReceiverMXBean.stop
* GatewaySenderMXBean.start
* GatewaySenderMXBean.stop
* GatewaySenderMXBean.pause
* GatewaySenderMXBean.resume
* GatewaySenderMXBean.rebalance
*
* DATA:MANAGE -> CLUSTER:MANAGE:JAR
*
* create async-event-queue (Requires CLUSTER:WRITE:DISK if persistent)
* destroy function
* undeploy
*
* DATA:MANAGE -> CLUSTER:MANAGE:QUERY
*
* clear defined indexes
* close durable-client
* close durable-cq
* create defined indexes
* stop continuous-query
* CacheServerMXBean.closeAllContinuousQuery
* CacheServerMXBean.closeContinuousQuery
* DistributedSystemMXBean.setQueryResultSetLimit
* DistributedSystemMXBean.setQueryCollectionsDepth
*
* DATA:READ -> CLUSTER:READ
*
* list region
*
* DATA:MANAGE -> [None]
*
* pdx rename
*
* DATA:READ -> DATA:READ and CLUSTER:WRITE:DISK
*
* backup disk-store
*
* DATA:MANAGE:RegionName -> CLUSTER:MANAGE:QUERY
*
* create index
* create lucene index (also requires CLUSTER:WRITE:DISK)
* define index
* destroy lucene index
*
* DATA:MANAGE, DATA:WRITE, CLUSTER:MANAGE, and CLUSTER:WRITE -> CLUSTER:MANAGE:JAR
*
* deploy
*
* DATA:MANAGE or DATA:MANAGE:RegionName -> CLUSTER:MANAGE:QUERY
*
* destroy index
*
* CLUSTER:READ -> CLUSTER:READ:QUERY
*
* describe lucene index
* list lucene indexes
*
* DATA:WRITE -> DATA:READ:Region
*
* search lucene index
*
* DATA:MANAGE -> DATA:MANAGE and CLUSTER:WRITE:DISK if persistent
* create region
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/PurelyApplied/geode geode-2924
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/geode/pull/576.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #576
----
commit d8155ae44e767890790566b66cb7cc5ed7ea0cbc
Author: Patrick Rhomberg <pr...@pivotal.io>
Date: 2017-06-12T19:14:08Z
Most of the imports.
commit 49ad4f4144c22ab334f061964b745640cc4664d6
Author: Patrick Rhomberg <pr...@pivotal.io>
Date: 2017-06-12T19:49:52Z
GEODE-2924: Migration of security from DATA:MANAGE
* DATA:MANAGE -> CLUSTER:MANAGE
*
* configure pdx
* LockServiceMXBean.becomeLockGrantor
*
* DATA:MANAGE -> CLUSTER:MANAGE:DISK
*
* compact disk-store
* create disk-store
* destroy disk-store
* revoke missing-disk-store
* DiskStoreMXBean.forceRoll
* DiskStoreMXBean.forceCompaction
* DiskStoreMXBean.flush
* DiskStoreMXBean.setDiskUsageWarningPercentage
* DiskStoreMXBean.setDiskUsageCriticalPercentage
* DistributedSystemMXBean.revokeMissingDiskStores
* MemberMXBean.compactAllDistStores
*
* DATA:MANAGE -> CLUSTER:MANAGE:GATEWAY
*
* create gateway-receiver
* create gateway-sender
* destroy gateway-sender
* load-balance gateway-sender
* pause gateway-sender
* resume gateway-sender
* start gateway-receiver
* start gateway-sender
* stop gateway-receiver
* stop gateway-sender
* GatewayReceiverMXBean.start
* GatewayReceiverMXBean.stop
* GatewaySenderMXBean.start
* GatewaySenderMXBean.stop
* GatewaySenderMXBean.pause
* GatewaySenderMXBean.resume
* GatewaySenderMXBean.rebalance
*
* DATA:MANAGE -> CLUSTER:MANAGE:JAR
*
* create async-event-queue (Requires CLUSTER:WRITE:DISK if persistent)
* destroy function
* undeploy
*
* DATA:MANAGE -> CLUSTER:MANAGE:QUERY
*
* clear defined indexes
* close durable-client
* close durable-cq
* create defined indexes
* stop continuous-query
* CacheServerMXBean.closeAllContinuousQuery
* CacheServerMXBean.closeContinuousQuery
* DistributedSystemMXBean.setQueryResultSetLimit
* DistributedSystemMXBean.setQueryCollectionsDepth
*
* DATA:READ -> CLUSTER:READ
*
* list region
*
* DATA:MANAGE -> [None]
*
* pdx rename
*
* DATA:READ -> DATA:READ and CLUSTER:WRITE:DISK
*
* backup disk-store
*
* DATA:MANAGE:RegionName -> CLUSTER:MANAGE:QUERY
*
* create index
* create lucene index (also requires CLUSTER:WRITE:DISK)
* define index
* destroy lucene index
*
* DATA:MANAGE, DATA:WRITE, CLUSTER:MANAGE, and CLUSTER:WRITE -> CLUSTER:MANAGE:JAR
*
* deploy
*
* DATA:MANAGE or DATA:MANAGE:RegionName -> CLUSTER:MANAGE:QUERY
*
* destroy index
*
* CLUSTER:READ -> CLUSTER:READ:QUERY
*
* describe lucene index
* list index
* list lucene indexes
*
* DATA:WRITE -> DATA:READ:Region
*
* search lucene index
*
* DATA:MANAGE -> DATA:MANAGE and CLUSTER:WRITE:DISK if persistent
* create region
commit b3977c1c15f9dfd7476b1d584d63b78875333504
Author: Patrick Rhomberg <pr...@pivotal.io>
Date: 2017-06-12T19:42:29Z
General cleanup of all touched files.
----
> Optimize speed of backup/ restore cache snapshots
> -------------------------------------------------
>
> Key: GEODE-292
> URL: https://issues.apache.org/jira/browse/GEODE-292
> Project: Geode
> Issue Type: Improvement
> Components: persistence
> Affects Versions: 1.0.0-incubating
> Reporter: Wes Williams
> Assignee: Wes Williams
> Labels: backups, performance
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> Backup/ restore using snapshots takes a very, very long time when:
> 1) There are a lot of PDXTypes, and
> 2) There are a lot of regions.
> Specifically, it takes 35 minutes to restore only 50MB of unstructured JSON from snapshots. In contrast, it takes only 3 minutes to reload all the data from scratch.
> PROBLEM
> CacheSnapshot loops all regions and saves all cache PDXTypes in every region.gfd. On restore, it reloads all cache PDXTypes again for every region where they only need to be loaded once.
> SOLUTION
> This JIRA issue will create an option to save PDXTypes only once and reload them once and store only data in the region snapshots. Existing functionality will remain for those who want it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)