You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sai Sai <sa...@yahoo.in> on 2013/05/22 08:11:24 UTC
Re: Hadoop Development on cloud in a secure and economical way.
Is it possible to do Hadoop development on cloud in a secure and economical way without worrying about our source being taken away. We would like to have Hadoop and eclipse installed on a vm in cloud and our developers will log into the cloud on a daily basis and work on the cloud. Like this we r hoping if we develop any product we will minimize our source being taken away by the devs or others and is secure. Please let me know, any suggestions u may have.
Thanks,
Sai
Re: diff between these 2 dirs
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering if someone can explain what is the diff between these 2 dirs:
Contents of directory /home/satish/work/mapred/staging/satish/.staging
and this dir:
/hadoop/mapred/system
Thanks
Sai
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Ellis Miller <ou...@gmail.com>.
Configure private cloud: install VMWare / VirtualBox / KVM on internal
server / cluster and levearge either Cloudera Hadoop (free version) or
Hortonworks (Yahoo introduced Hortonworks and where Cloudera is exceptional
but proprietary Hortonworks requires some configuration and tuning of
Hadoop in their promise to keep their Hadoop offering entirely open
source).
Both have Hadoop appliances which can be imported into VMWare / VirtualBox.
Couple this with, perhaps, using Amazon Web Services for production hosting
and all development can be done in-house with Eclipse and appropriate
Source Control Management.
With Amazon don't have to worry so much about your proprietary code being
stolen as the biggest concern (depending on your HA requirements) have been
several high-profile outages where their servers in VA (when you subscribe
to Amazon Web Services and create a Hadoop cluster, for example, it asks
you to specify the geographical region).
Nice thing about using Virtualization in-house for development, test,
integration environments your developers can code and perform initial
testing in Dev using VMWare / VirtualBox to then take a snapshot and export
the Dev environment then import into production. VWare and VirtualBox both
provide proprietary means to accelerate this process streamlining an Agile
/ Extreme Programming software development framework while simplifying the
release cycle.
Personally, if I were going to do it on the cheap and keep even production
in-house would leverage x86 Linux servers (running Ubuntu, Fedora, Centos,
or even OpenSuSE) which natively support KVM (linux based Virtualization
which is free) to host Hortonworks (Hadoop) leveraging the Unix
Administrators and Network Engineering monkey's to securing the Production
server which would have to be configured to accept public / external
connections (assuming you are hosting an application which is Software as a
Service for your customers) while not compromising security.
Have worked at 2 firms in the past several years where we did the very same
and it worked exceptionally well just takes time to get your IT staff up to
speed on Virtualization, Cluster administration and synchronizing Virtual
Machine backups with the promotion of releases from Dev to Test to
Production.
On Tue, May 21, 2013 at 11:11 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
--
Ellis R. Miller
937.829.2380
<http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
Mundo Nulla Fides
<http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Amazon elastic cloud computer.
Pay per use
Thanks,
Rahul
On Wed, May 22, 2013 at 11:41 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Amazon elastic cloud computer.
Pay per use
Thanks,
Rahul
On Wed, May 22, 2013 at 11:41 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Amazon elastic cloud computer.
Pay per use
Thanks,
Rahul
On Wed, May 22, 2013 at 11:41 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Amazon elastic cloud computer.
Pay per use
Thanks,
Rahul
On Wed, May 22, 2013 at 11:41 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Ellis Miller <ou...@gmail.com>.
Configure private cloud: install VMWare / VirtualBox / KVM on internal
server / cluster and levearge either Cloudera Hadoop (free version) or
Hortonworks (Yahoo introduced Hortonworks and where Cloudera is exceptional
but proprietary Hortonworks requires some configuration and tuning of
Hadoop in their promise to keep their Hadoop offering entirely open
source).
Both have Hadoop appliances which can be imported into VMWare / VirtualBox.
Couple this with, perhaps, using Amazon Web Services for production hosting
and all development can be done in-house with Eclipse and appropriate
Source Control Management.
With Amazon don't have to worry so much about your proprietary code being
stolen as the biggest concern (depending on your HA requirements) have been
several high-profile outages where their servers in VA (when you subscribe
to Amazon Web Services and create a Hadoop cluster, for example, it asks
you to specify the geographical region).
Nice thing about using Virtualization in-house for development, test,
integration environments your developers can code and perform initial
testing in Dev using VMWare / VirtualBox to then take a snapshot and export
the Dev environment then import into production. VWare and VirtualBox both
provide proprietary means to accelerate this process streamlining an Agile
/ Extreme Programming software development framework while simplifying the
release cycle.
Personally, if I were going to do it on the cheap and keep even production
in-house would leverage x86 Linux servers (running Ubuntu, Fedora, Centos,
or even OpenSuSE) which natively support KVM (linux based Virtualization
which is free) to host Hortonworks (Hadoop) leveraging the Unix
Administrators and Network Engineering monkey's to securing the Production
server which would have to be configured to accept public / external
connections (assuming you are hosting an application which is Software as a
Service for your customers) while not compromising security.
Have worked at 2 firms in the past several years where we did the very same
and it worked exceptionally well just takes time to get your IT staff up to
speed on Virtualization, Cluster administration and synchronizing Virtual
Machine backups with the promotion of releases from Dev to Test to
Production.
On Tue, May 21, 2013 at 11:11 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
--
Ellis R. Miller
937.829.2380
<http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
Mundo Nulla Fides
<http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Ellis Miller <ou...@gmail.com>.
Configure private cloud: install VMWare / VirtualBox / KVM on internal
server / cluster and levearge either Cloudera Hadoop (free version) or
Hortonworks (Yahoo introduced Hortonworks and where Cloudera is exceptional
but proprietary Hortonworks requires some configuration and tuning of
Hadoop in their promise to keep their Hadoop offering entirely open
source).
Both have Hadoop appliances which can be imported into VMWare / VirtualBox.
Couple this with, perhaps, using Amazon Web Services for production hosting
and all development can be done in-house with Eclipse and appropriate
Source Control Management.
With Amazon don't have to worry so much about your proprietary code being
stolen as the biggest concern (depending on your HA requirements) have been
several high-profile outages where their servers in VA (when you subscribe
to Amazon Web Services and create a Hadoop cluster, for example, it asks
you to specify the geographical region).
Nice thing about using Virtualization in-house for development, test,
integration environments your developers can code and perform initial
testing in Dev using VMWare / VirtualBox to then take a snapshot and export
the Dev environment then import into production. VWare and VirtualBox both
provide proprietary means to accelerate this process streamlining an Agile
/ Extreme Programming software development framework while simplifying the
release cycle.
Personally, if I were going to do it on the cheap and keep even production
in-house would leverage x86 Linux servers (running Ubuntu, Fedora, Centos,
or even OpenSuSE) which natively support KVM (linux based Virtualization
which is free) to host Hortonworks (Hadoop) leveraging the Unix
Administrators and Network Engineering monkey's to securing the Production
server which would have to be configured to accept public / external
connections (assuming you are hosting an application which is Software as a
Service for your customers) while not compromising security.
Have worked at 2 firms in the past several years where we did the very same
and it worked exceptionally well just takes time to get your IT staff up to
speed on Virtualization, Cluster administration and synchronizing Virtual
Machine backups with the promotion of releases from Dev to Test to
Production.
On Tue, May 21, 2013 at 11:11 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
--
Ellis R. Miller
937.829.2380
<http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
Mundo Nulla Fides
<http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>
Re: Hadoop Development on cloud in a secure and economical way.
Posted by Ellis Miller <ou...@gmail.com>.
Configure private cloud: install VMWare / VirtualBox / KVM on internal
server / cluster and levearge either Cloudera Hadoop (free version) or
Hortonworks (Yahoo introduced Hortonworks and where Cloudera is exceptional
but proprietary Hortonworks requires some configuration and tuning of
Hadoop in their promise to keep their Hadoop offering entirely open
source).
Both have Hadoop appliances which can be imported into VMWare / VirtualBox.
Couple this with, perhaps, using Amazon Web Services for production hosting
and all development can be done in-house with Eclipse and appropriate
Source Control Management.
With Amazon don't have to worry so much about your proprietary code being
stolen as the biggest concern (depending on your HA requirements) have been
several high-profile outages where their servers in VA (when you subscribe
to Amazon Web Services and create a Hadoop cluster, for example, it asks
you to specify the geographical region).
Nice thing about using Virtualization in-house for development, test,
integration environments your developers can code and perform initial
testing in Dev using VMWare / VirtualBox to then take a snapshot and export
the Dev environment then import into production. VWare and VirtualBox both
provide proprietary means to accelerate this process streamlining an Agile
/ Extreme Programming software development framework while simplifying the
release cycle.
Personally, if I were going to do it on the cheap and keep even production
in-house would leverage x86 Linux servers (running Ubuntu, Fedora, Centos,
or even OpenSuSE) which natively support KVM (linux based Virtualization
which is free) to host Hortonworks (Hadoop) leveraging the Unix
Administrators and Network Engineering monkey's to securing the Production
server which would have to be configured to accept public / external
connections (assuming you are hosting an application which is Software as a
Service for your customers) while not compromising security.
Have worked at 2 firms in the past several years where we did the very same
and it worked exceptionally well just takes time to get your IT staff up to
speed on Virtualization, Cluster administration and synchronizing Virtual
Machine backups with the promotion of releases from Dev to Test to
Production.
On Tue, May 21, 2013 at 11:11 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any suggestions u may have.
> Thanks,
> Sai
>
--
Ellis R. Miller
937.829.2380
<http://my.wisestamp.com/link?u=2hxhdfd4p76bkhcm&site=www.wisestamp.com/email-install>
Mundo Nulla Fides
<http://my.wisestamp.com/link?u=gfbmwhzrwxzcrjqx&site=www.wisestamp.com/email-install>
Re: diff between these 2 dirs
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering if someone can explain what is the diff between these 2 dirs:
Contents of directory /home/satish/work/mapred/staging/satish/.staging
and this dir:
/hadoop/mapred/system
Thanks
Sai
Re: diff between these 2 dirs
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering if someone can explain what is the diff between these 2 dirs:
Contents of directory /home/satish/work/mapred/staging/satish/.staging
and this dir:
/hadoop/mapred/system
Thanks
Sai
Re: diff between these 2 dirs
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering if someone can explain what is the diff between these 2 dirs:
Contents of directory /home/satish/work/mapred/staging/satish/.staging
and this dir:
/hadoop/mapred/system
Thanks
Sai