[jira] [Updated] (FLINK-959) Automated bare-metal deployment of FLINK on Amazon EC2 and OpenStack instances

["Ufuk Celebi (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ufuk Celebi updated FLINK-959:
------------------------------

    Component/s: Build System

> Automated bare-metal deployment of FLINK on Amazon EC2 and OpenStack instances
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-959
>                 URL: https://issues.apache.org/jira/browse/FLINK-959
>             Project: Flink
>          Issue Type: New Feature
>          Components: Build System
>    Affects Versions: pre-apache-0.5
>            Reporter: Tobias
>            Assignee: Tobias
>             Fix For: pre-apache-0.5
>
>
> This python script does start Amazon ec2|OpenStack instances to install java+hadoop and configure hdfs/yarn via puppet. In order to run FLINK on top of hadoop YARN.
> In order to install java and hadoop binaries are downloaded by the script and handed over to puppet for automated provisioning.
> User-data scripts are used to install puppet (only debian) on the master and slave instances. Accordingly security groups are created and configured. 
> The master instance then starts a self configuration process, so that the puppet modules are set up according to the cluster structure. 
> The master  detects if hadoop YARN web interface is accessible and waits for all expected nodes to be up and running. Then a stratosphere yarn session is started. Taskmanager and Jobmanager memory allocations are set up in the instances.cfg.
> Notes:
> - Configuration reserves 600mb for the operating system and allocates the rest for the YARN node.
> - The Flink web interface is not accessible because the yarn.web.proxy throws a NullpointerException
> - Only runs on Debian derivatives because it uses apt-get 
> - Tested with ubuntu-13.08
> - FLINK is still named Stratosphere
> Code at: https://github.com/tobwiens/StratopshereBareMetalProvPuppet



--
This message was sent by Atlassian JIRA
(v6.2#6252)