You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kamil Szewczyk <sz...@gmail.com> on 2018/05/23 11:59:10 UTC
Kubernetes cluster of apache-beam-testing project
Dear Beam Devs,
we are using kubernetes to on demand create/tear down resources for
performance testing. It is done automatically by Jenkins jobs using
PerfKit. On 20th of May, there was a blog post about security issues in
kubernetes
https://cloud.google.com/kubernetes-engine/docs/security-bulletins and we
should upgrade it to the latest version of the main branch. As far as I
know in apache-beam-testing, we are now using kubernetes cluster with
engine version 1.8, can we update it to version 1.9(1.9.7-gke.1)?
We are missing some new cool features introduced in kubernetes 1.9 like
StatefulSets which I would like to use to setup HDFS large cluster for
performance testing, I have already submitted PR for that. My findings are
described in this two JIRA issues:
- https://issues.apache.org/jira/browse/BEAM-4362
- https://issues.apache.org/jira/browse/BEAM-4390
Who is in charge of Google Cloud Platform (have admin access) and can help
with that?
Re: Kubernetes cluster of apache-beam-testing project
Posted by Chamikara Jayalath <ch...@google.com>.
We had to file a JIRA against Apache infra for getting kubectl installed:
https://issues.apache.org/jira/browse/INFRA-14819
Not sure if we need to get infra involved for updating the version as well.
- Cham
On Thu, May 24, 2018 at 10:42 AM Alan Myrvold <am...@google.com> wrote:
> Updated the io-datastores kubernetes cluster in apache-beam-testing to
> version 1.9.7-gke.1 for the master and version 1.9.7-gke.1 for the node
> pool.
>
> On Thu, May 24, 2018 at 10:31 AM Yifan Zou <yi...@google.com> wrote:
>
>> Seems like kubectl v1.9.0 was installed on those nodes. Here is the
>> puppet script I found that used to download and install kubectl on
>> beam-jenkins.
>>
>> https://github.com/apache/infrastructure-puppet/blob/95aed4f4f7cdce79a1f0fc8b1e581f13fedf63ff/modules/build_slaves/manifests/kube.pp
>>
>> On Thu, May 24, 2018 at 10:18 AM Alan Myrvold <am...@google.com>
>> wrote:
>>
>>> I can update the version of the io-datastores cluster master and node
>>> pools.
>>> I don't have access to the kubectl version on the executors ... I can
>>> ask around for that.
>>>
>>> On Wed, May 23, 2018 at 4:59 AM Kamil Szewczyk <sz...@gmail.com>
>>> wrote:
>>>
>>>> Dear Beam Devs,
>>>>
>>>> we are using kubernetes to on demand create/tear down resources for
>>>> performance testing. It is done automatically by Jenkins jobs using
>>>> PerfKit. On 20th of May, there was a blog post about security issues in
>>>> kubernetes
>>>> https://cloud.google.com/kubernetes-engine/docs/security-bulletins and
>>>> we should upgrade it to the latest version of the main branch. As far as I
>>>> know in apache-beam-testing, we are now using kubernetes cluster with
>>>> engine version 1.8, can we update it to version 1.9(1.9.7-gke.1)?
>>>> We are missing some new cool features introduced in kubernetes 1.9
>>>> like StatefulSets which I would like to use to setup HDFS large cluster for
>>>> performance testing, I have already submitted PR for that. My findings are
>>>> described in this two JIRA issues:
>>>> - https://issues.apache.org/jira/browse/BEAM-4362
>>>> - https://issues.apache.org/jira/browse/BEAM-4390
>>>>
>>>> Who is in charge of Google Cloud Platform (have admin access) and can
>>>> help with that?
>>>>
>>>>
>>>>
Re: Kubernetes cluster of apache-beam-testing project
Posted by Alan Myrvold <am...@google.com>.
Updated the io-datastores kubernetes cluster in apache-beam-testing to
version 1.9.7-gke.1 for the master and version 1.9.7-gke.1 for the node
pool.
On Thu, May 24, 2018 at 10:31 AM Yifan Zou <yi...@google.com> wrote:
> Seems like kubectl v1.9.0 was installed on those nodes. Here is the puppet
> script I found that used to download and install kubectl on beam-jenkins.
>
> https://github.com/apache/infrastructure-puppet/blob/95aed4f4f7cdce79a1f0fc8b1e581f13fedf63ff/modules/build_slaves/manifests/kube.pp
>
> On Thu, May 24, 2018 at 10:18 AM Alan Myrvold <am...@google.com> wrote:
>
>> I can update the version of the io-datastores cluster master and node
>> pools.
>> I don't have access to the kubectl version on the executors ... I can ask
>> around for that.
>>
>> On Wed, May 23, 2018 at 4:59 AM Kamil Szewczyk <sz...@gmail.com>
>> wrote:
>>
>>> Dear Beam Devs,
>>>
>>> we are using kubernetes to on demand create/tear down resources for
>>> performance testing. It is done automatically by Jenkins jobs using
>>> PerfKit. On 20th of May, there was a blog post about security issues in
>>> kubernetes
>>> https://cloud.google.com/kubernetes-engine/docs/security-bulletins and
>>> we should upgrade it to the latest version of the main branch. As far as I
>>> know in apache-beam-testing, we are now using kubernetes cluster with
>>> engine version 1.8, can we update it to version 1.9(1.9.7-gke.1)?
>>> We are missing some new cool features introduced in kubernetes 1.9 like
>>> StatefulSets which I would like to use to setup HDFS large cluster for
>>> performance testing, I have already submitted PR for that. My findings are
>>> described in this two JIRA issues:
>>> - https://issues.apache.org/jira/browse/BEAM-4362
>>> - https://issues.apache.org/jira/browse/BEAM-4390
>>>
>>> Who is in charge of Google Cloud Platform (have admin access) and can
>>> help with that?
>>>
>>>
>>>
Re: Kubernetes cluster of apache-beam-testing project
Posted by Yifan Zou <yi...@google.com>.
Seems like kubectl v1.9.0 was installed on those nodes. Here is the puppet
script I found that used to download and install kubectl on beam-jenkins.
https://github.com/apache/infrastructure-puppet/blob/95aed4f4f7cdce79a1f0fc8b1e581f13fedf63ff/modules/build_slaves/manifests/kube.pp
On Thu, May 24, 2018 at 10:18 AM Alan Myrvold <am...@google.com> wrote:
> I can update the version of the io-datastores cluster master and node
> pools.
> I don't have access to the kubectl version on the executors ... I can ask
> around for that.
>
> On Wed, May 23, 2018 at 4:59 AM Kamil Szewczyk <sz...@gmail.com> wrote:
>
>> Dear Beam Devs,
>>
>> we are using kubernetes to on demand create/tear down resources for
>> performance testing. It is done automatically by Jenkins jobs using
>> PerfKit. On 20th of May, there was a blog post about security issues in
>> kubernetes
>> https://cloud.google.com/kubernetes-engine/docs/security-bulletins and
>> we should upgrade it to the latest version of the main branch. As far as I
>> know in apache-beam-testing, we are now using kubernetes cluster with
>> engine version 1.8, can we update it to version 1.9(1.9.7-gke.1)?
>> We are missing some new cool features introduced in kubernetes 1.9 like
>> StatefulSets which I would like to use to setup HDFS large cluster for
>> performance testing, I have already submitted PR for that. My findings are
>> described in this two JIRA issues:
>> - https://issues.apache.org/jira/browse/BEAM-4362
>> - https://issues.apache.org/jira/browse/BEAM-4390
>>
>> Who is in charge of Google Cloud Platform (have admin access) and can
>> help with that?
>>
>>
>>
Re: Kubernetes cluster of apache-beam-testing project
Posted by Alan Myrvold <am...@google.com>.
I can update the version of the io-datastores cluster master and node
pools.
I don't have access to the kubectl version on the executors ... I can ask
around for that.
On Wed, May 23, 2018 at 4:59 AM Kamil Szewczyk <sz...@gmail.com> wrote:
> Dear Beam Devs,
>
> we are using kubernetes to on demand create/tear down resources for
> performance testing. It is done automatically by Jenkins jobs using
> PerfKit. On 20th of May, there was a blog post about security issues in
> kubernetes
> https://cloud.google.com/kubernetes-engine/docs/security-bulletins and we
> should upgrade it to the latest version of the main branch. As far as I
> know in apache-beam-testing, we are now using kubernetes cluster with
> engine version 1.8, can we update it to version 1.9(1.9.7-gke.1)?
> We are missing some new cool features introduced in kubernetes 1.9 like
> StatefulSets which I would like to use to setup HDFS large cluster for
> performance testing, I have already submitted PR for that. My findings are
> described in this two JIRA issues:
> - https://issues.apache.org/jira/browse/BEAM-4362
> - https://issues.apache.org/jira/browse/BEAM-4390
>
> Who is in charge of Google Cloud Platform (have admin access) and can help
> with that?
>
>
>