Re: Kubernetes integration for HDFS

[Alexander Batyrshin <0x...@gmail.com>]

ENV JAVA_HOME /usr/lib/jvm/jre-11-openjdk Is Hadoop OK with this version?
As far i can see - this version is not compatible
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
--
Alexander Batyrshin aka bash
Biomechanical Artificial Sabotage Humanoid


On Fri, 20 Sep 2019 at 08:22, Anu Engineer <ae...@cloudera.com.invalid>
wrote:

> That is very interesting. A sub-project of Hadoop called Ozone tests
> regularly with Kubernetes. If you would like to contribute your work to
> Hadoop, you can send us a pull request and we can maintain this code as a
> sample under Ozone K8s samples directory. An advantage with that approach
> is that this will get regularly tested. Please note that K8s support is
> very early / experimental under Ozone at this point of time.
>
> -- Anu
>
>
> On Thu, Sep 19, 2019 at 7:30 PM Allan Espinosa <ye...@gmail.com> wrote:
>
>> Hello folks,
>>
>> I just wanted to share a PoC of running HDFS on Kubernetes that I hacked
>> on over the summer.
>>
>> https://github.com/aespinosa/cloud-hadoop
>>
>> The above repo introduces a "hdfs kube_namenode" command where the
>> namenode formats itself or runs "-bootstrapStandby" depending on the state
>> of provisioning an HDFS cluster.  The state information is stored inside
>> Kubernetes as metadata annotations to the manifest .yml of running HDFS.
>>
>> With this, it is simpler to bring up HA sandbox HDFS clusters without the
>> need for externally orchestrating namenode and zkfc formatting.
>>
>> Hope others find this useful.
>>
>> Regards,
>> Allan
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>

Re: Kubernetes integration for HDFS

[Allan Espinosa <ye...@gmail.com>]

Hi Alexander,

I didn’t do extensive HDFS testing that I didn’t encounter those JIRAs in the Confluence page.  From a smoketest standpoint it did seem to work.

 

Allan

 

 

From: Alexander Batyrshin <0x...@gmail.com>
Date: Saturday, October 5, 2019 at 6:39 PM
To: Anu Engineer <ae...@cloudera.com.invalid>
Cc: Allan Espinosa <ye...@gmail.com>, <us...@hadoop.apache.org>
Subject: Re: Kubernetes integration for HDFS

 

ENV JAVA_HOME /usr/lib/jvm/jre-11-openjdk

Is Hadoop OK with this version?
As far i can see - this version is not compatible https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
--
Alexander Batyrshin aka bash
Biomechanical Artificial Sabotage Humanoid

 

 

On Fri, 20 Sep 2019 at 08:22, Anu Engineer <ae...@cloudera.com.invalid> wrote:

That is very interesting. A sub-project of Hadoop called Ozone tests regularly with Kubernetes. If you would like to contribute your work to  Hadoop, you can send us a pull request and we can maintain this code as a sample under Ozone K8s samples directory. An advantage with that approach is that this will get regularly tested. Please note that K8s support is very early / experimental under Ozone at this point of time.

 

-- Anu

 

 

On Thu, Sep 19, 2019 at 7:30 PM Allan Espinosa <ye...@gmail.com> wrote:

Hello folks,

I just wanted to share a PoC of running HDFS on Kubernetes that I hacked on over the summer.

https://github.com/aespinosa/cloud-hadoop

The above repo introduces a "hdfs kube_namenode" command where the namenode formats itself or runs "-bootstrapStandby" depending on the state of provisioning an HDFS cluster.  The state information is stored inside Kubernetes as metadata annotations to the manifest .yml of running HDFS.

With this, it is simpler to bring up HA sandbox HDFS clusters without the need for externally orchestrating namenode and zkfc formatting.

Hope others find this useful.

Regards,
Allan



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org