[jira] [Commented] (METRON-2239) Metron Automated backup and restore

["Dale Richardson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953353#comment-16953353 ] 

Dale Richardson commented on METRON-2239:
-----------------------------------------

Heh [~otto],

So I've been thinking about upgrade scenarios over the past week, so I though I might share some thoughts.

From what I can see, Metron-specific configurations / customisations are in:
 #  Zookeeper
 # Stored in Ambari
 # Stored in /usr/metron/$METRON_VER/* (including parser_contrib)
 # Anywhere else I've missed?

Metron processed data is stored:
 # HDFS
 # HBase
 # Solr or ES

A major upgrade of Metron would need all configurations stored.   A DR backup would require all configurations plus a copy of all Metron processed data.

The current upgrade process is slightly complicated by the fact that the new version of Metron will not work on HDP 2.6, and the old version of Metron will not work on HDP 3.1.

So the process as I see it
 # Cut-off all inputs to Metron
 # Monitor processing until all in-flight data is indexed and stored
 # Stop Metron service
 # Export file-based metron configs
 # Export zookeeper based metron configs
 # Export ambari-based metron configs
 # Remove the metron service.
 # Remove Metron mpack
 # Update Ambari to latest version
 # Update to HDP 3.1 using existing Ambari/HDP update documentation (including Solr/ES etc).
 # Reinstall Metron mpack
 # Reinstall Metron service
 # Turn off Metron service
 # Tweak any configs required to support new Metron version
 # Re-import file-based metron configs
 # Re-import zookeeper based metron configs
 # Re-import ambari-based zookeeper configs
 # Turn on Metron service
 # Profit from real-time big data security.

I think that we can do up a script that could do steps 4-6 and 14-17.  Nothing about step 10 is a "metron thing", even though metron data is stored in HDP.  A pointer to the existing HDP upgrade documentation to guide the user in upgrading the cluster could be added.


What do you think?

 

 

 

 

 

 

> Metron Automated backup and restore
> -----------------------------------
>
>                 Key: METRON-2239
>                 URL: https://issues.apache.org/jira/browse/METRON-2239
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Otto Fowler
>            Priority: Blocker
>
> Metron, for upgrading to HDP 3.1 should have the ability to backup and restore metron specific configurations.
> For many, this upgrade will involve OS upgrade and Hadoop/HDP upgrade/Amabari etc.
> Such a tool would:
> Backup to file metron configurations from different locations, along with enough meta data on the files to restore them.  The directory/backup location would be structured.
> * Disk on nodes
> * HDFS
> * Zookeeper
> * ??
> This backup will then be archived by the tool, and could then be saved by the user.
> The tool ( or a companion tool ) would be able to take these archives and restore them back to the cluster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)