You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Gayan Gunarathne <ga...@wso2.com> on 2015/09/18 16:50:33 UTC
Multi-region Kubernetes service un-deployment
When we try to un-deploy the application in a multi region deployment, It
will throw the following exception.
javax.ws.rs.WebApplicationException: HTTP 404 pods "tomcatr1-pod-1" not
found
Issue seems like,in a topology service clusterIdClusterMap has only one
value even there is two clusters in two regions. So when un-deploying it is
map to the same region cluster. So when un-deploying the application, first
region will remove successfully but in the second region it will throw the
exception of "HTTP 404 pod not found"
I think we may need to recheck the logic of adding the clusterIdClusterMap
in the case of multi region deployment.
Thanks,
Gayan
--
Gayan Gunarathne
Technical Lead, WSO2 Inc. (http://wso2.com)
Committer & PMC Member, Apache Stratos
email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
Re: Multi-region Kubernetes service un-deployment
Posted by Gayan Gunarathne <ga...@wso2.com>.
@Udara
Great. Once you finish the tests, let us know the results.
@Akila
I know you are facing some merge conflict issues when you try to push your
changes. Please let us know once you are done.I think we can start the
4.1.3 release work after that.
Thanks,
Gayan
On Sun, Sep 20, 2015 at 6:14 AM, Udara Liyanage <ud...@wso2.com> wrote:
> Hi Gayan,
>
> I performed a quick smoke test with two clusters and it was successful.
> I will do some more tests and let you know the results.
>
> On Sat, Sep 19, 2015 at 10:47 AM, Gayan Gunarathne <ga...@wso2.com>
> wrote:
>
>> Additionally we have removed the kubernetesClusterId from the
>> ClusterContext and added the kubernetesClusterId to the
>> kuberentesServices.Idea behind that is we can't have kubernetesClusterId in
>> ClusterContext as there can be one or more Kubernetes clusters in the case
>> of multi-region deployment.But ideally Kubernetes service needs to bound
>> with a Kubernetes cluster.
>>
>> I have push the changes to the stratos-4.1.x branch . Shall we have a
>> quick round of smoke testing and do the 4.1.3 release?
>>
>> Thanks,
>> Gayan
>>
>>
>> On Sat, Sep 19, 2015 at 9:30 PM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> I had an offline discussion on this with Gayan. It looks like the root
>>> cause of this problem is that we have added Kubernetes Cluster ID to the
>>> Cluster Context. This seems to be incorrect. A cluster can have multiple
>>> Kubernetes Clusters if its a multi-region deployment.
>>>
>>> We need to remove the property ClusterContext.kubernetesClsuterId() and
>>> do following to find the Kubernetes Cluster ID of a member:
>>>
>>> Partition partition = memberContext.getPartition();
>>>
>>> String kubernetesClusterId = partition.getKubernetesClusterId();
>>>
>>> KubernetesCluster kubernetesCluster =
>>> CloudControllerContext.getInstance().getKubernetesCluster(kubernetesClusterId);
>>>
>>>
>>> In addition we also identified few improvements that we can do in
>>> KubernetesIaas class. Gayan is now working on this fix.
>>>
>>> Thanks
>>>
>>> On Sat, Sep 19, 2015 at 1:06 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> We might need to use cluster id and network partition id as the key of
>>>> this map.
>>>>
>>>> Once we are done with this fix shall we do 4.1.3 release build?
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Friday, September 18, 2015, Gayan Gunarathne <ga...@wso2.com>
>>>> wrote:
>>>>
>>>>> When we try to un-deploy the application in a multi region deployment,
>>>>> It will throw the following exception.
>>>>>
>>>>> javax.ws.rs.WebApplicationException: HTTP 404 pods "tomcatr1-pod-1"
>>>>> not found
>>>>>
>>>>> Issue seems like,in a topology service clusterIdClusterMap has only
>>>>> one value even there is two clusters in two regions. So when un-deploying
>>>>> it is map to the same region cluster. So when un-deploying the application,
>>>>> first region will remove successfully but in the second region it will
>>>>> throw the exception of "HTTP 404 pod not found"
>>>>>
>>>>> I think we may need to recheck the logic of adding the
>>>>> clusterIdClusterMap in the case of multi region deployment.
>>>>>
>>>>> Thanks,
>>>>> Gayan
>>>>> --
>>>>>
>>>>> Gayan Gunarathne
>>>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>>>> Committer & PMC Member, Apache Stratos
>>>>> email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Senior Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Senior Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead, WSO2 Inc. (http://wso2.com)
>> Committer & PMC Member, Apache Stratos
>> email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>
>>
>>
>
>
>
> --
>
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
>
--
Gayan Gunarathne
Technical Lead, WSO2 Inc. (http://wso2.com)
Committer & PMC Member, Apache Stratos
email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
Re: Multi-region Kubernetes service un-deployment
Posted by Udara Liyanage <ud...@wso2.com>.
Hi Gayan,
I performed a quick smoke test with two clusters and it was successful.
I will do some more tests and let you know the results.
On Sat, Sep 19, 2015 at 10:47 AM, Gayan Gunarathne <ga...@wso2.com> wrote:
> Additionally we have removed the kubernetesClusterId from the
> ClusterContext and added the kubernetesClusterId to the
> kuberentesServices.Idea behind that is we can't have kubernetesClusterId in
> ClusterContext as there can be one or more Kubernetes clusters in the case
> of multi-region deployment.But ideally Kubernetes service needs to bound
> with a Kubernetes cluster.
>
> I have push the changes to the stratos-4.1.x branch . Shall we have a
> quick round of smoke testing and do the 4.1.3 release?
>
> Thanks,
> Gayan
>
>
> On Sat, Sep 19, 2015 at 9:30 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> I had an offline discussion on this with Gayan. It looks like the root
>> cause of this problem is that we have added Kubernetes Cluster ID to the
>> Cluster Context. This seems to be incorrect. A cluster can have multiple
>> Kubernetes Clusters if its a multi-region deployment.
>>
>> We need to remove the property ClusterContext.kubernetesClsuterId() and
>> do following to find the Kubernetes Cluster ID of a member:
>>
>> Partition partition = memberContext.getPartition();
>>
>> String kubernetesClusterId = partition.getKubernetesClusterId();
>>
>> KubernetesCluster kubernetesCluster =
>> CloudControllerContext.getInstance().getKubernetesCluster(kubernetesClusterId);
>>
>>
>> In addition we also identified few improvements that we can do in
>> KubernetesIaas class. Gayan is now working on this fix.
>>
>> Thanks
>>
>> On Sat, Sep 19, 2015 at 1:06 PM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> We might need to use cluster id and network partition id as the key of
>>> this map.
>>>
>>> Once we are done with this fix shall we do 4.1.3 release build?
>>>
>>> Thanks
>>>
>>>
>>> On Friday, September 18, 2015, Gayan Gunarathne <ga...@wso2.com> wrote:
>>>
>>>> When we try to un-deploy the application in a multi region deployment,
>>>> It will throw the following exception.
>>>>
>>>> javax.ws.rs.WebApplicationException: HTTP 404 pods "tomcatr1-pod-1" not
>>>> found
>>>>
>>>> Issue seems like,in a topology service clusterIdClusterMap has only one
>>>> value even there is two clusters in two regions. So when un-deploying it is
>>>> map to the same region cluster. So when un-deploying the application, first
>>>> region will remove successfully but in the second region it will throw the
>>>> exception of "HTTP 404 pod not found"
>>>>
>>>> I think we may need to recheck the logic of adding the
>>>> clusterIdClusterMap in the case of multi region deployment.
>>>>
>>>> Thanks,
>>>> Gayan
>>>> --
>>>>
>>>> Gayan Gunarathne
>>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>>> Committer & PMC Member, Apache Stratos
>>>> email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Senior Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Senior Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead, WSO2 Inc. (http://wso2.com)
> Committer & PMC Member, Apache Stratos
> email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>
>
>
--
Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897
Re: Multi-region Kubernetes service un-deployment
Posted by Gayan Gunarathne <ga...@wso2.com>.
Additionally we have removed the kubernetesClusterId from the
ClusterContext and added the kubernetesClusterId to the
kuberentesServices.Idea behind that is we can't have kubernetesClusterId in
ClusterContext as there can be one or more Kubernetes clusters in the case
of multi-region deployment.But ideally Kubernetes service needs to bound
with a Kubernetes cluster.
I have push the changes to the stratos-4.1.x branch . Shall we have a quick
round of smoke testing and do the 4.1.3 release?
Thanks,
Gayan
On Sat, Sep 19, 2015 at 9:30 PM, Imesh Gunaratne <im...@apache.org> wrote:
> I had an offline discussion on this with Gayan. It looks like the root
> cause of this problem is that we have added Kubernetes Cluster ID to the
> Cluster Context. This seems to be incorrect. A cluster can have multiple
> Kubernetes Clusters if its a multi-region deployment.
>
> We need to remove the property ClusterContext.kubernetesClsuterId() and do
> following to find the Kubernetes Cluster ID of a member:
>
> Partition partition = memberContext.getPartition();
>
> String kubernetesClusterId = partition.getKubernetesClusterId();
>
> KubernetesCluster kubernetesCluster =
> CloudControllerContext.getInstance().getKubernetesCluster(kubernetesClusterId);
>
>
> In addition we also identified few improvements that we can do in
> KubernetesIaas class. Gayan is now working on this fix.
>
> Thanks
>
> On Sat, Sep 19, 2015 at 1:06 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> We might need to use cluster id and network partition id as the key of
>> this map.
>>
>> Once we are done with this fix shall we do 4.1.3 release build?
>>
>> Thanks
>>
>>
>> On Friday, September 18, 2015, Gayan Gunarathne <ga...@wso2.com> wrote:
>>
>>> When we try to un-deploy the application in a multi region deployment,
>>> It will throw the following exception.
>>>
>>> javax.ws.rs.WebApplicationException: HTTP 404 pods "tomcatr1-pod-1" not
>>> found
>>>
>>> Issue seems like,in a topology service clusterIdClusterMap has only one
>>> value even there is two clusters in two regions. So when un-deploying it is
>>> map to the same region cluster. So when un-deploying the application, first
>>> region will remove successfully but in the second region it will throw the
>>> exception of "HTTP 404 pod not found"
>>>
>>> I think we may need to recheck the logic of adding the
>>> clusterIdClusterMap in the case of multi region deployment.
>>>
>>> Thanks,
>>> Gayan
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead, WSO2 Inc. (http://wso2.com)
>>> Committer & PMC Member, Apache Stratos
>>> email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>>
>>>
>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Senior Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>>
>
>
> --
> Imesh Gunaratne
>
> Senior Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Gayan Gunarathne
Technical Lead, WSO2 Inc. (http://wso2.com)
Committer & PMC Member, Apache Stratos
email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
Re: Multi-region Kubernetes service un-deployment
Posted by Imesh Gunaratne <im...@apache.org>.
I had an offline discussion on this with Gayan. It looks like the root
cause of this problem is that we have added Kubernetes Cluster ID to the
Cluster Context. This seems to be incorrect. A cluster can have multiple
Kubernetes Clusters if its a multi-region deployment.
We need to remove the property ClusterContext.kubernetesClsuterId() and do
following to find the Kubernetes Cluster ID of a member:
Partition partition = memberContext.getPartition();
String kubernetesClusterId = partition.getKubernetesClusterId();
KubernetesCluster kubernetesCluster =
CloudControllerContext.getInstance().getKubernetesCluster(kubernetesClusterId);
In addition we also identified few improvements that we can do in
KubernetesIaas class. Gayan is now working on this fix.
Thanks
On Sat, Sep 19, 2015 at 1:06 PM, Imesh Gunaratne <im...@apache.org> wrote:
> We might need to use cluster id and network partition id as the key of
> this map.
>
> Once we are done with this fix shall we do 4.1.3 release build?
>
> Thanks
>
>
> On Friday, September 18, 2015, Gayan Gunarathne <ga...@wso2.com> wrote:
>
>> When we try to un-deploy the application in a multi region deployment, It
>> will throw the following exception.
>>
>> javax.ws.rs.WebApplicationException: HTTP 404 pods "tomcatr1-pod-1" not
>> found
>>
>> Issue seems like,in a topology service clusterIdClusterMap has only one
>> value even there is two clusters in two regions. So when un-deploying it is
>> map to the same region cluster. So when un-deploying the application, first
>> region will remove successfully but in the second region it will throw the
>> exception of "HTTP 404 pod not found"
>>
>> I think we may need to recheck the logic of adding the
>> clusterIdClusterMap in the case of multi region deployment.
>>
>> Thanks,
>> Gayan
>> --
>>
>> Gayan Gunarathne
>> Technical Lead, WSO2 Inc. (http://wso2.com)
>> Committer & PMC Member, Apache Stratos
>> email : gayang@wso2.com | mobile : +94 775030545 <%2B94%20766819985>
>>
>>
>>
>
>
> --
> Imesh Gunaratne
>
> Senior Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
>
--
Imesh Gunaratne
Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: Multi-region Kubernetes service un-deployment
Posted by Imesh Gunaratne <im...@apache.org>.
We might need to use cluster id and network partition id as the key of this
map.
Once we are done with this fix shall we do 4.1.3 release build?
Thanks
On Friday, September 18, 2015, Gayan Gunarathne <ga...@wso2.com> wrote:
> When we try to un-deploy the application in a multi region deployment, It
> will throw the following exception.
>
> javax.ws.rs.WebApplicationException: HTTP 404 pods "tomcatr1-pod-1" not
> found
>
> Issue seems like,in a topology service clusterIdClusterMap has only one
> value even there is two clusters in two regions. So when un-deploying it is
> map to the same region cluster. So when un-deploying the application, first
> region will remove successfully but in the second region it will throw the
> exception of "HTTP 404 pod not found"
>
> I think we may need to recheck the logic of adding the clusterIdClusterMap
> in the case of multi region deployment.
>
> Thanks,
> Gayan
> --
>
> Gayan Gunarathne
> Technical Lead, WSO2 Inc. (http://wso2.com)
> Committer & PMC Member, Apache Stratos
> email : gayang@wso2.com <javascript:_e(%7B%7D,'cvml','gayang@wso2.com');>
> | mobile : +94 775030545 <%2B94%20766819985>
>
>
>
--
Imesh Gunaratne
Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos