[jira] [Updated] (HAWQ-706) Final steps to bring HAWQ in compliance with Bigtop reqs

["Goden Yao (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HAWQ-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Goden Yao updated HAWQ-706:
---------------------------
    Fix Version/s: backlog

> Final steps to bring HAWQ in compliance with Bigtop reqs
> --------------------------------------------------------
>
>                 Key: HAWQ-706
>                 URL: https://issues.apache.org/jira/browse/HAWQ-706
>             Project: Apache HAWQ
>          Issue Type: Task
>          Components: Build
>    Affects Versions: 2.0.0.0-incubating
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: backlog
>
>
> This ticket is to list and track the required steps to finally enable the integration of HAWQ into Bigtop.
> All relevant resources are linked below, and here's the overview of the remaining steps and the overall status of the integration work.
> *External dependencies*
> - the biggest issue was and remains the use of libthrift, which isn't packaged, provided nor supported by anyone. Right now, Bigtop-HAWQ integration branch [uses|https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob_plain;f=bigtop_toolchain/manifests/libhdfs.pp;hb=refs/heads/BIGTOP-2320] my own pre-built version of the library, hosted [here|https://bintray.com/artifact/download/wangzw/deb/dists/trusty/contrib/binary-amd64]. However, this is clearly an insecure and has to be either solved by HAWQ adding this dependency as the source; or by convincing Bigtop community that hosting libthrift library is beneficial for the community at large
> *Packaging*
> - overall, the packaging code is complete and is pushed to the Bigtop branch (see link below). Considering that the work has been completed about 5 weeks ago and was aimed at the state of trunk back in the March, there might be some minor changes, which would require additional tweaks
> - libhdfs library code (if already included into HAWQ project) might require additional changes to the packaging code, so the library can be produces and properly set in the installation phase
> - Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from the BIGTOP-2320 below)
> *Tests*
> - smoke tests need to be created (as per BIGTOP-2322), but that seems to be a minor undertaking once the rest of the work is finished
> - packaging tests are required to be integrated into Bigtop stack BIGTOP-2324
> *Deployment*
> - deployment code is completed. However, it needs to be extended to property support cluster roles and to be linked to the main {{site.pp}} recipe
> - because real-life deployment can not rely on in-house python wrappers using passwordless-ssh, the lifecycle management and initial bootstrap are done directly by calling into HAWQ scripts, providing such functionality. It is possible that some of these interfaces were updated in the last 6 weeks, so additional testing would be needed.
> - it should be responsibility of the HAWQ to provide a concise way of initializing a master, segment, and so on without a need for password-less ssh, which is suboptimal and won't be accepted by Bigtop community as it is breaks the deployment model
> *Toolchain*
> - toolchain code is completed in the bigtop branch. This will allow to build HAWQ in the standard Bigtop container available for the CI and 3rd party users
> - toolchain code needs to be rebased on top of current Bigtop master. and possible conflicts would have to be resolved
> - once the integration is finished, Bigtop slave images will have to be updated to enable automatic CI runs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)