You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Richard Pelavin (JIRA)" <ji...@apache.org> on 2015/04/01 09:04:52 UTC
[jira] [Comment Edited] (BIGTOP-1772) Split init-hdfs.sh script and
let the individual modules take care of folder creation
[ https://issues.apache.org/jira/browse/BIGTOP-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14390107#comment-14390107 ]
Richard Pelavin edited comment on BIGTOP-1772 at 4/1/15 7:04 AM:
-----------------------------------------------------------------
Sorry late to this conversation. I think it is useful to conceptually distinguish between logic that creates hdfs directories and services that have the needs for these directories. As discussed, the logic that actually creates the directories can be done by services themselves or by logic as it is today tied to hdfs
Now having services that create the directories seems fine as long as you know there will be no name clashes where two services want same directory and that the services have access to a user that has access rights to create the directory
The complication with having hdfs mechanism create directories is that need to make sure that it has Init-hcfs.json or equivalent with all the needed directories. Simplest to create directories for all possible services, but clearly better to create only needed directories
Any of these schemes is compatible with puppet being mechanism to create hdfs directories. With puppet makes sense to have a class that takes an array where each element is directory to create. If this is done by one central place can check array for name clashes and raise error
was (Author: rnp):
Sorry late to this conversation. I think it is useful to conceptually distinguish between logic that creates hdfs directories and services that have the needs for these directories. As discussed, the logic that actually creates the directories can be done by services themselves or by logic as it is today tied to hdfs
Now having services that create the directories seems fine as long as you know there will be no name clashes where two services want same directory and that the services have access to a user that has access rights to create the directory
The complication with having hdfs mechanism create directories is that need to make sure that it has Init-hcfs.json or equivalent with all the needed directories. Simplest to create directories for all possible services, but clearly better to create only needed directories
Any of these schemes is compatible with puppet being mechanism to create hdfs directories. With puppet makes sense to have a define that takes an array where each element is directory to create. If this is done by one central place can check array for name clashes and raise error
> Split init-hdfs.sh script and let the individual modules take care of folder creation
> -------------------------------------------------------------------------------------
>
> Key: BIGTOP-1772
> URL: https://issues.apache.org/jira/browse/BIGTOP-1772
> Project: Bigtop
> Issue Type: Improvement
> Components: Init scripts
> Affects Versions: 0.8.0
> Reporter: vishnu gajendran
>
> Currently, there is one init-hdfs.sh script which is responsible for creating folders in hdfs for other services like hbase, oozie etc.. This adds depedency between modules (for example, before starting hbase master, we need to ensure that this script is run) which gets further complicated in a scenario where different components run on different node. In order to avoid this, we can let the individual modules take care of creating respective folders in hdfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)