You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Min Chen <mi...@citrix.com> on 2013/06/11 07:35:47 UTC
Object_Store storage refactor GoToMeeting Tomorrow
Hi there,
To reach consensus on some remaining NFS cache issues on object_store storage refactor work in a more effective manner, John, Edison and I have scheduled a GoToMeeting tomorrow to discuss them over the phone, any interested parties are welcome to join and brainstorm. Here are detailed GTM information:
Meeting Time: 10:30 AM – 12:30 PM PST
Meeting Details:
1. Please join my meeting.
https://www1.gotomeeting.com/join/188620897
2. Use your microphone and speakers (VoIP) - a headset is recommended. Or, call in using your telephone.
United States: +1 (626) 521-0017
United States (toll-free): 1 877 309 2070
Access Code: 188-620-897
Audio PIN: Shown after joining the meeting
Meeting ID: 188-620-897
GoToMeeting®
Online Meetings Made Easy®
Not at your computer? Click the link to join this meeting from your iPhone®, iPad® or Android® device via the GoToMeeting app.
Thanks
-min
Re: Object_Store storage refactor Meeting Notes
Posted by John Burwell <jb...@basho.com>.
David and Chip,
There was no intention to side step to the list. I will be more careful in my phrasing in the future.
I apologize for the lack of clarity,
-John
On Jun 13, 2013, at 1:58 PM, David Nalley <da...@gnsa.us> wrote:
> On Thu, Jun 13, 2013 at 1:55 PM, Chip Childers
> <ch...@sungard.com> wrote:
>> On Thu, Jun 13, 2013 at 05:52:01PM +0000, Animesh Chaturvedi wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Chip Childers [mailto:chip.childers@sungard.com]
>>>> Sent: Thursday, June 13, 2013 10:43 AM
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: Object_Store storage refactor Meeting Notes
>>>>
>>>> First, thanks for bringing this back to the list. I'm +1 on the
>>>> technical approach.
>>>>
>>>> A couple of thoughts though, just so that we make sure that we keep
>>>> operating in the right manner as an Apache project:
>>>>
>>>> Let's be careful about declaring something a "decision" or that
>>>> something was "determined" when it happened off-list. I think that, in
>>>> the case below, you were really saying "we agreed to make this the
>>>> proposal to the list", so no harm there.
>>>>
>>>> The last bit, about a meeting in person, is a little concerning to me
>>>> though... It really needs to be open to anyone that might want to
>>>> participate if at all possible (and this means folks that aren't
>>>> physically there). Also, please be sure to bring back a summary of the
>>>> discussions to the list, so that those not there have an opportunity to
>>>> see what was discussed (and comment if they have comments). Think of
>>>> the outcome of the discussions as "proposals" that will be brought back
>>>> to the list.
>>>>
>>>> Sorry for sounding preachy, but it's important to keep remembering that
>>>> the list is where decisions are made... and discussions shouldn't be
>>>> closed to community members that may have geographic or temporal
>>>> constraints.
>>>>
>>>> -chip
>>> [Animesh>] Chip I think by now everyone knows that all decisions need to happen on the list. John B's proposal to have a face-face meeting I think is good to bring in a good proposal which can be discussed and decided on the mailing list.
>>
>> It needed to be said, only because of some of the wording. I'm sure you
>> guys are going to do the right thing.
>
> Agreed.
> There are more people reading this thread than those participating,
> and some of those folks may not yet be indoctrinated in the Apache
> Way, and could give the impression, despite the best of intentions,
> that there is some cabal that is making decisions in secret.
> Perception is reality far too often.
>
> --David
Re: Object_Store storage refactor Meeting Notes
Posted by David Nalley <da...@gnsa.us>.
On Thu, Jun 13, 2013 at 1:55 PM, Chip Childers
<ch...@sungard.com> wrote:
> On Thu, Jun 13, 2013 at 05:52:01PM +0000, Animesh Chaturvedi wrote:
>>
>>
>> > -----Original Message-----
>> > From: Chip Childers [mailto:chip.childers@sungard.com]
>> > Sent: Thursday, June 13, 2013 10:43 AM
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: Object_Store storage refactor Meeting Notes
>> >
>> > First, thanks for bringing this back to the list. I'm +1 on the
>> > technical approach.
>> >
>> > A couple of thoughts though, just so that we make sure that we keep
>> > operating in the right manner as an Apache project:
>> >
>> > Let's be careful about declaring something a "decision" or that
>> > something was "determined" when it happened off-list. I think that, in
>> > the case below, you were really saying "we agreed to make this the
>> > proposal to the list", so no harm there.
>> >
>> > The last bit, about a meeting in person, is a little concerning to me
>> > though... It really needs to be open to anyone that might want to
>> > participate if at all possible (and this means folks that aren't
>> > physically there). Also, please be sure to bring back a summary of the
>> > discussions to the list, so that those not there have an opportunity to
>> > see what was discussed (and comment if they have comments). Think of
>> > the outcome of the discussions as "proposals" that will be brought back
>> > to the list.
>> >
>> > Sorry for sounding preachy, but it's important to keep remembering that
>> > the list is where decisions are made... and discussions shouldn't be
>> > closed to community members that may have geographic or temporal
>> > constraints.
>> >
>> > -chip
>> [Animesh>] Chip I think by now everyone knows that all decisions need to happen on the list. John B's proposal to have a face-face meeting I think is good to bring in a good proposal which can be discussed and decided on the mailing list.
>
> It needed to be said, only because of some of the wording. I'm sure you
> guys are going to do the right thing.
Agreed.
There are more people reading this thread than those participating,
and some of those folks may not yet be indoctrinated in the Apache
Way, and could give the impression, despite the best of intentions,
that there is some cabal that is making decisions in secret.
Perception is reality far too often.
--David
Re: Object_Store storage refactor Meeting Notes
Posted by Chip Childers <ch...@sungard.com>.
On Fri, Jun 14, 2013 at 01:13:37AM +0000, Animesh Chaturvedi wrote:
> [Animesh>] I was not sure if we have an open slot in Collab for design session on storage discussed above but now that Joe mentioned the HackDay being an open collaborative day in other email it's probably best to have that discussion as one of the session on HACK day.
>
Great idea!
RE: Object_Store storage refactor Meeting Notes
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: Chip Childers [mailto:chip.childers@sungard.com]
> Sent: Thursday, June 13, 2013 10:55 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Object_Store storage refactor Meeting Notes
>
> On Thu, Jun 13, 2013 at 05:52:01PM +0000, Animesh Chaturvedi wrote:
> >
> >
> > > -----Original Message-----
> > > From: Chip Childers [mailto:chip.childers@sungard.com]
> > > Sent: Thursday, June 13, 2013 10:43 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: Object_Store storage refactor Meeting Notes
> > >
> > > First, thanks for bringing this back to the list. I'm +1 on the
> > > technical approach.
> > >
> > > A couple of thoughts though, just so that we make sure that we keep
> > > operating in the right manner as an Apache project:
> > >
> > > Let's be careful about declaring something a "decision" or that
> > > something was "determined" when it happened off-list. I think that,
> > > in the case below, you were really saying "we agreed to make this
> > > the proposal to the list", so no harm there.
> > >
> > > The last bit, about a meeting in person, is a little concerning to
> > > me though... It really needs to be open to anyone that might want
> > > to participate if at all possible (and this means folks that aren't
> > > physically there). Also, please be sure to bring back a summary of
> > > the discussions to the list, so that those not there have an
> > > opportunity to see what was discussed (and comment if they have
> > > comments). Think of the outcome of the discussions as "proposals"
> > > that will be brought back to the list.
> > >
> > > Sorry for sounding preachy, but it's important to keep remembering
> > > that the list is where decisions are made... and discussions
> > > shouldn't be closed to community members that may have geographic or
> > > temporal constraints.
> > >
> > > -chip
> > [Animesh>] Chip I think by now everyone knows that all decisions need
> to happen on the list. John B's proposal to have a face-face meeting I
> think is good to bring in a good proposal which can be discussed and
> decided on the mailing list.
>
> It needed to be said, only because of some of the wording. I'm sure you
> guys are going to do the right thing.
[Animesh>] I was not sure if we have an open slot in Collab for design session on storage discussed above but now that Joe mentioned the HackDay being an open collaborative day in other email it's probably best to have that discussion as one of the session on HACK day.
Re: Object_Store storage refactor Meeting Notes
Posted by Chip Childers <ch...@sungard.com>.
On Thu, Jun 13, 2013 at 05:52:01PM +0000, Animesh Chaturvedi wrote:
>
>
> > -----Original Message-----
> > From: Chip Childers [mailto:chip.childers@sungard.com]
> > Sent: Thursday, June 13, 2013 10:43 AM
> > To: dev@cloudstack.apache.org
> > Subject: Re: Object_Store storage refactor Meeting Notes
> >
> > First, thanks for bringing this back to the list. I'm +1 on the
> > technical approach.
> >
> > A couple of thoughts though, just so that we make sure that we keep
> > operating in the right manner as an Apache project:
> >
> > Let's be careful about declaring something a "decision" or that
> > something was "determined" when it happened off-list. I think that, in
> > the case below, you were really saying "we agreed to make this the
> > proposal to the list", so no harm there.
> >
> > The last bit, about a meeting in person, is a little concerning to me
> > though... It really needs to be open to anyone that might want to
> > participate if at all possible (and this means folks that aren't
> > physically there). Also, please be sure to bring back a summary of the
> > discussions to the list, so that those not there have an opportunity to
> > see what was discussed (and comment if they have comments). Think of
> > the outcome of the discussions as "proposals" that will be brought back
> > to the list.
> >
> > Sorry for sounding preachy, but it's important to keep remembering that
> > the list is where decisions are made... and discussions shouldn't be
> > closed to community members that may have geographic or temporal
> > constraints.
> >
> > -chip
> [Animesh>] Chip I think by now everyone knows that all decisions need to happen on the list. John B's proposal to have a face-face meeting I think is good to bring in a good proposal which can be discussed and decided on the mailing list.
It needed to be said, only because of some of the wording. I'm sure you
guys are going to do the right thing.
RE: Object_Store storage refactor Meeting Notes
Posted by Animesh Chaturvedi <an...@citrix.com>.
> -----Original Message-----
> From: Chip Childers [mailto:chip.childers@sungard.com]
> Sent: Thursday, June 13, 2013 10:43 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Object_Store storage refactor Meeting Notes
>
> First, thanks for bringing this back to the list. I'm +1 on the
> technical approach.
>
> A couple of thoughts though, just so that we make sure that we keep
> operating in the right manner as an Apache project:
>
> Let's be careful about declaring something a "decision" or that
> something was "determined" when it happened off-list. I think that, in
> the case below, you were really saying "we agreed to make this the
> proposal to the list", so no harm there.
>
> The last bit, about a meeting in person, is a little concerning to me
> though... It really needs to be open to anyone that might want to
> participate if at all possible (and this means folks that aren't
> physically there). Also, please be sure to bring back a summary of the
> discussions to the list, so that those not there have an opportunity to
> see what was discussed (and comment if they have comments). Think of
> the outcome of the discussions as "proposals" that will be brought back
> to the list.
>
> Sorry for sounding preachy, but it's important to keep remembering that
> the list is where decisions are made... and discussions shouldn't be
> closed to community members that may have geographic or temporal
> constraints.
>
> -chip
[Animesh>] Chip I think by now everyone knows that all decisions need to happen on the list. John B's proposal to have a face-face meeting I think is good to bring in a good proposal which can be discussed and decided on the mailing list.
>
> On Thu, Jun 13, 2013 at 12:38:16PM -0400, John Burwell wrote:
> > All,
> >
> > Edison Su, Min Chen, Animesh Chaturvedi, and myself met via
> teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting
> was determine the path forward for merging the object_store branch by
> the 4.2 freeze date, 30 June 2013. The conversation focused on the
> following topics:
> >
> > * Staging area mechanism
> > * Removing dependencies from the Storage to the Hypervisor layer
> > * Dependencies of other patches on object_store
> > * QA's desire to start testing the patch ASAP
> >
> > Min, Edison, and I agreed that the staging mechanism must age out
> files and use a reference count to ensure that file in-use are not
> prematurely purged. While we agree that some form of reservation system
> is required, Edison is concerned that it will be too conservative and
> create bottlenecks.
> >
> > As we delved deeper into the subject of the storage to hypervisor
> dependencies and the reservation mechanism, we determined that NFS
> storage would still need to be the size of the secondary storage data
> set. Since the hypervisor layer has not been completely fitted to the
> new storage layer, NFS would be still required for a number of
> operations. Based on this realization, we decided to de-scope the
> staging mechanism, and leave the 4.2 object store functionality the same
> as 4.1. Therefore, NFS will remain the secondary storage of record, and
> object storage will serve as backup/multi-zone sync. In 4.3, we will
> fit the hypervisor layer for the new storage layer which will allow
> object stores to server as secondary storage. This work will include
> removing the storage to hypervisor dependencies. For 4.2, Edison and
> Min have implemented the critical foundation necessary to establish our
> next generation storage layer. There simply was not enough time in this
> development cycle to make these changes and fit the hypervisor layer.
> >
> > Due to the size of the patch, Animesh voiced QA's concerned regarding
> test scope and impact. As such, we want to get the merge completed as
> soon as possible to allow testing to begin. We discussed breaking up
> the patch, but we could not devise a reasonable set of chunks there were
> both isolated and significantly testable. Therefore, the patch can only
> be delivered in its current state. We also walked through potential
> dependencies between the storage framework changes and the solidfire
> branch. It was determined that these two merges could occur
> independently.
> >
> > Finally, Animesh is going to setup a meeting at Citrix's Santa Clara
> office on 26 June 2013 (the day after Collab ends) to discuss the path
> forward for 4.3 and work through a high-level design/approach to fitting
> the hypervisor layer to exploit the new storage capabilities. Details
> will be published to the dev mailing list.
> >
> > Thanks,
> > -John
Re: Object_Store storage refactor Meeting Notes
Posted by Chip Childers <ch...@sungard.com>.
First, thanks for bringing this back to the list. I'm +1 on the technical approach.
A couple of thoughts though, just so that we make sure that we keep
operating in the right manner as an Apache project:
Let's be careful about declaring something a "decision" or that
something was "determined" when it happened off-list. I think that,
in the case below, you were really saying "we agreed to make this the
proposal to the list", so no harm there.
The last bit, about a meeting in person, is a little concerning to me
though... It really needs to be open to anyone that might want to
participate if at all possible (and this means folks that aren't
physically there). Also, please be sure to bring back a summary of the
discussions to the list, so that those not there have an opportunity to
see what was discussed (and comment if they have comments). Think of
the outcome of the discussions as "proposals" that will be brought back
to the list.
Sorry for sounding preachy, but it's important to keep remembering that
the list is where decisions are made... and discussions shouldn't be
closed to community members that may have geographic or temporal constraints.
-chip
On Thu, Jun 13, 2013 at 12:38:16PM -0400, John Burwell wrote:
> All,
>
> Edison Su, Min Chen, Animesh Chaturvedi, and myself met via teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting was determine the path forward for merging the object_store branch by the 4.2 freeze date, 30 June 2013. The conversation focused on the following topics:
>
> * Staging area mechanism
> * Removing dependencies from the Storage to the Hypervisor layer
> * Dependencies of other patches on object_store
> * QA's desire to start testing the patch ASAP
>
> Min, Edison, and I agreed that the staging mechanism must age out files and use a reference count to ensure that file in-use are not prematurely purged. While we agree that some form of reservation system is required, Edison is concerned that it will be too conservative and create bottlenecks.
>
> As we delved deeper into the subject of the storage to hypervisor dependencies and the reservation mechanism, we determined that NFS storage would still need to be the size of the secondary storage data set. Since the hypervisor layer has not been completely fitted to the new storage layer, NFS would be still required for a number of operations. Based on this realization, we decided to de-scope the staging mechanism, and leave the 4.2 object store functionality the same as 4.1. Therefore, NFS will remain the secondary storage of record, and object storage will serve as backup/multi-zone sync. In 4.3, we will fit the hypervisor layer for the new storage layer which will allow object stores to server as secondary storage. This work will include removing the storage to hypervisor dependencies. For 4.2, Edison and Min have implemented the critical foundation necessary to establish our next generation storage layer. There simply was not enough time in this development cycle to make these changes and fit the hypervisor layer.
>
> Due to the size of the patch, Animesh voiced QA's concerned regarding test scope and impact. As such, we want to get the merge completed as soon as possible to allow testing to begin. We discussed breaking up the patch, but we could not devise a reasonable set of chunks there were both isolated and significantly testable. Therefore, the patch can only be delivered in its current state. We also walked through potential dependencies between the storage framework changes and the solidfire branch. It was determined that these two merges could occur independently.
>
> Finally, Animesh is going to setup a meeting at Citrix's Santa Clara office on 26 June 2013 (the day after Collab ends) to discuss the path forward for 4.3 and work through a high-level design/approach to fitting the hypervisor layer to exploit the new storage capabilities. Details will be published to the dev mailing list.
>
> Thanks,
> -John
RE: Object_Store storage refactor Meeting Notes
Posted by Sudha Ponnaganti <su...@citrix.com>.
Sure John!
I am referring to the point that even for one time merge, let us go through the planned QA cycle.
Thanks
/sudha
-----Original Message-----
From: John Burwell [mailto:jburwell@basho.com]
Sent: Thursday, June 13, 2013 10:14 AM
To: dev@cloudstack.apache.org
Subject: Re: Object_Store storage refactor Meeting Notes
Sudha,
The current plan is to merge once. We explored the feasibility of decomposing it into independent, testable chunks, and determined it was not possible.
Thanks,
-John
On Jun 13, 2013, at 12:54 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
> Thanks John for summary. From QA stand point it would make sense to
> merge once
>
> - assigned test cases are executed and pass rate is on par with
> release criteria ( test plans published and execution results are
> being posted)
> - automation runs are successful and shows same pass rate as Master
> - blockers are fixed before merge
>
> Let me know if this would be agreeable. QA usually would not test features completely on feature branches but this one is exception given the nature of changes.
>
> Thanks
> /sudha
>
> -----Original Message-----
> From: John Burwell [mailto:jburwell@basho.com]
> Sent: Thursday, June 13, 2013 9:38 AM
> To: dev@cloudstack.apache.org
> Subject: Object_Store storage refactor Meeting Notes
>
> All,
>
> Edison Su, Min Chen, Animesh Chaturvedi, and myself met via teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting was determine the path forward for merging the object_store branch by the 4.2 freeze date, 30 June 2013. The conversation focused on the following topics:
>
> * Staging area mechanism
> * Removing dependencies from the Storage to the Hypervisor layer
> * Dependencies of other patches on object_store
> * QA's desire to start testing the patch ASAP
>
> Min, Edison, and I agreed that the staging mechanism must age out files and use a reference count to ensure that file in-use are not prematurely purged. While we agree that some form of reservation system is required, Edison is concerned that it will be too conservative and create bottlenecks.
>
> As we delved deeper into the subject of the storage to hypervisor dependencies and the reservation mechanism, we determined that NFS storage would still need to be the size of the secondary storage data set. Since the hypervisor layer has not been completely fitted to the new storage layer, NFS would be still required for a number of operations. Based on this realization, we decided to de-scope the staging mechanism, and leave the 4.2 object store functionality the same as 4.1. Therefore, NFS will remain the secondary storage of record, and object storage will serve as backup/multi-zone sync. In 4.3, we will fit the hypervisor layer for the new storage layer which will allow object stores to server as secondary storage. This work will include removing the storage to hypervisor dependencies. For 4.2, Edison and Min have implemented the critical foundation necessary to establish our next generation storage layer. There simply was not enough time in this development cycle to make these changes and fit the hypervisor layer.
>
> Due to the size of the patch, Animesh voiced QA's concerned regarding test scope and impact. As such, we want to get the merge completed as soon as possible to allow testing to begin. We discussed breaking up the patch, but we could not devise a reasonable set of chunks there were both isolated and significantly testable. Therefore, the patch can only be delivered in its current state. We also walked through potential dependencies between the storage framework changes and the solidfire branch. It was determined that these two merges could occur independently.
>
> Finally, Animesh is going to setup a meeting at Citrix's Santa Clara office on 26 June 2013 (the day after Collab ends) to discuss the path forward for 4.3 and work through a high-level design/approach to fitting the hypervisor layer to exploit the new storage capabilities. Details will be published to the dev mailing list.
>
> Thanks,
> -John
>
> On Jun 11, 2013, at 2:08 AM, Min Chen <mi...@citrix.com> wrote:
>
>> It is 11th June. John is not free between 9:15am to 10am, that is why
>> we schedule it at 10:30am.
>>
>> Thanks
>> -min
>>
>> On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>>
>>> Hi Min,
>>> When you say tomorrow, what date is it 11th June or 12th ? Can the
>>> time be preponed by an hour please - its too late here ?
>>>
>>> Thanks,
>>> -Nitin
>>>
>>> On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>>>
>>>> Hi there,
>>>>
>>>> To reach consensus on some remaining NFS cache issues on
>>>> object_store storage refactor work in a more effective manner,
>>>> John, Edison and I have scheduled a GoToMeeting tomorrow to discuss
>>>> them over the phone, any interested parties are welcome to join and
>>>> brainstorm. Here are detailed GTM information:
>>>>
>>>> Meeting Time: 10:30 AM 12:30 PM PST
>>>>
>>>> Meeting Details:
>>>>
>>>> 1. Please join my meeting.
>>>> https://www1.gotomeeting.com/join/188620897
>>>>
>>>> 2. Use your microphone and speakers (VoIP) - a headset is recommended.
>>>> Or, call in using your telephone.
>>>>
>>>> United States: +1 (626) 521-0017
>>>> United States (toll-free): 1 877 309 2070
>>>>
>>>> Access Code: 188-620-897
>>>> Audio PIN: Shown after joining the meeting
>>>>
>>>> Meeting ID: 188-620-897
>>>>
>>>> GoToMeeting(r)
>>>> Online Meetings Made Easy(r)
>>>>
>>>> Not at your computer? Click the link to join this meeting from your
>>>> iPhone(r), iPad(r) or Android(r) device via the GoToMeeting app.
>>>>
>>>> Thanks
>>>> -min
>>>
>>
>
Re: Object_Store storage refactor Meeting Notes
Posted by John Burwell <jb...@basho.com>.
Sudha,
The current plan is to merge once. We explored the feasibility of decomposing it into independent, testable chunks, and determined it was not possible.
Thanks,
-John
On Jun 13, 2013, at 12:54 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
> Thanks John for summary. From QA stand point it would make sense to merge once
>
> - assigned test cases are executed and pass rate is on par with release criteria ( test plans published and execution results are being posted)
> - automation runs are successful and shows same pass rate as Master
> - blockers are fixed before merge
>
> Let me know if this would be agreeable. QA usually would not test features completely on feature branches but this one is exception given the nature of changes.
>
> Thanks
> /sudha
>
> -----Original Message-----
> From: John Burwell [mailto:jburwell@basho.com]
> Sent: Thursday, June 13, 2013 9:38 AM
> To: dev@cloudstack.apache.org
> Subject: Object_Store storage refactor Meeting Notes
>
> All,
>
> Edison Su, Min Chen, Animesh Chaturvedi, and myself met via teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting was determine the path forward for merging the object_store branch by the 4.2 freeze date, 30 June 2013. The conversation focused on the following topics:
>
> * Staging area mechanism
> * Removing dependencies from the Storage to the Hypervisor layer
> * Dependencies of other patches on object_store
> * QA's desire to start testing the patch ASAP
>
> Min, Edison, and I agreed that the staging mechanism must age out files and use a reference count to ensure that file in-use are not prematurely purged. While we agree that some form of reservation system is required, Edison is concerned that it will be too conservative and create bottlenecks.
>
> As we delved deeper into the subject of the storage to hypervisor dependencies and the reservation mechanism, we determined that NFS storage would still need to be the size of the secondary storage data set. Since the hypervisor layer has not been completely fitted to the new storage layer, NFS would be still required for a number of operations. Based on this realization, we decided to de-scope the staging mechanism, and leave the 4.2 object store functionality the same as 4.1. Therefore, NFS will remain the secondary storage of record, and object storage will serve as backup/multi-zone sync. In 4.3, we will fit the hypervisor layer for the new storage layer which will allow object stores to server as secondary storage. This work will include removing the storage to hypervisor dependencies. For 4.2, Edison and Min have implemented the critical foundation necessary to establish our next generation storage layer. There simply was not enough time in this development cycle to make these changes and fit the hypervisor layer.
>
> Due to the size of the patch, Animesh voiced QA's concerned regarding test scope and impact. As such, we want to get the merge completed as soon as possible to allow testing to begin. We discussed breaking up the patch, but we could not devise a reasonable set of chunks there were both isolated and significantly testable. Therefore, the patch can only be delivered in its current state. We also walked through potential dependencies between the storage framework changes and the solidfire branch. It was determined that these two merges could occur independently.
>
> Finally, Animesh is going to setup a meeting at Citrix's Santa Clara office on 26 June 2013 (the day after Collab ends) to discuss the path forward for 4.3 and work through a high-level design/approach to fitting the hypervisor layer to exploit the new storage capabilities. Details will be published to the dev mailing list.
>
> Thanks,
> -John
>
> On Jun 11, 2013, at 2:08 AM, Min Chen <mi...@citrix.com> wrote:
>
>> It is 11th June. John is not free between 9:15am to 10am, that is why
>> we schedule it at 10:30am.
>>
>> Thanks
>> -min
>>
>> On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>>
>>> Hi Min,
>>> When you say tomorrow, what date is it 11th June or 12th ? Can the
>>> time be preponed by an hour please - its too late here ?
>>>
>>> Thanks,
>>> -Nitin
>>>
>>> On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>>>
>>>> Hi there,
>>>>
>>>> To reach consensus on some remaining NFS cache issues on
>>>> object_store storage refactor work in a more effective manner, John,
>>>> Edison and I have scheduled a GoToMeeting tomorrow to discuss them
>>>> over the phone, any interested parties are welcome to join and
>>>> brainstorm. Here are detailed GTM information:
>>>>
>>>> Meeting Time: 10:30 AM 12:30 PM PST
>>>>
>>>> Meeting Details:
>>>>
>>>> 1. Please join my meeting.
>>>> https://www1.gotomeeting.com/join/188620897
>>>>
>>>> 2. Use your microphone and speakers (VoIP) - a headset is recommended.
>>>> Or, call in using your telephone.
>>>>
>>>> United States: +1 (626) 521-0017
>>>> United States (toll-free): 1 877 309 2070
>>>>
>>>> Access Code: 188-620-897
>>>> Audio PIN: Shown after joining the meeting
>>>>
>>>> Meeting ID: 188-620-897
>>>>
>>>> GoToMeeting(r)
>>>> Online Meetings Made Easy(r)
>>>>
>>>> Not at your computer? Click the link to join this meeting from your
>>>> iPhone(r), iPad(r) or Android(r) device via the GoToMeeting app.
>>>>
>>>> Thanks
>>>> -min
>>>
>>
>
RE: Object_Store storage refactor Meeting Notes
Posted by Sudha Ponnaganti <su...@citrix.com>.
Thanks John for summary. From QA stand point it would make sense to merge once
- assigned test cases are executed and pass rate is on par with release criteria ( test plans published and execution results are being posted)
- automation runs are successful and shows same pass rate as Master
- blockers are fixed before merge
Let me know if this would be agreeable. QA usually would not test features completely on feature branches but this one is exception given the nature of changes.
Thanks
/sudha
-----Original Message-----
From: John Burwell [mailto:jburwell@basho.com]
Sent: Thursday, June 13, 2013 9:38 AM
To: dev@cloudstack.apache.org
Subject: Object_Store storage refactor Meeting Notes
All,
Edison Su, Min Chen, Animesh Chaturvedi, and myself met via teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting was determine the path forward for merging the object_store branch by the 4.2 freeze date, 30 June 2013. The conversation focused on the following topics:
* Staging area mechanism
* Removing dependencies from the Storage to the Hypervisor layer
* Dependencies of other patches on object_store
* QA's desire to start testing the patch ASAP
Min, Edison, and I agreed that the staging mechanism must age out files and use a reference count to ensure that file in-use are not prematurely purged. While we agree that some form of reservation system is required, Edison is concerned that it will be too conservative and create bottlenecks.
As we delved deeper into the subject of the storage to hypervisor dependencies and the reservation mechanism, we determined that NFS storage would still need to be the size of the secondary storage data set. Since the hypervisor layer has not been completely fitted to the new storage layer, NFS would be still required for a number of operations. Based on this realization, we decided to de-scope the staging mechanism, and leave the 4.2 object store functionality the same as 4.1. Therefore, NFS will remain the secondary storage of record, and object storage will serve as backup/multi-zone sync. In 4.3, we will fit the hypervisor layer for the new storage layer which will allow object stores to server as secondary storage. This work will include removing the storage to hypervisor dependencies. For 4.2, Edison and Min have implemented the critical foundation necessary to establish our next generation storage layer. There simply was not enough time in this development cycle to make these changes and fit the hypervisor layer.
Due to the size of the patch, Animesh voiced QA's concerned regarding test scope and impact. As such, we want to get the merge completed as soon as possible to allow testing to begin. We discussed breaking up the patch, but we could not devise a reasonable set of chunks there were both isolated and significantly testable. Therefore, the patch can only be delivered in its current state. We also walked through potential dependencies between the storage framework changes and the solidfire branch. It was determined that these two merges could occur independently.
Finally, Animesh is going to setup a meeting at Citrix's Santa Clara office on 26 June 2013 (the day after Collab ends) to discuss the path forward for 4.3 and work through a high-level design/approach to fitting the hypervisor layer to exploit the new storage capabilities. Details will be published to the dev mailing list.
Thanks,
-John
On Jun 11, 2013, at 2:08 AM, Min Chen <mi...@citrix.com> wrote:
> It is 11th June. John is not free between 9:15am to 10am, that is why
> we schedule it at 10:30am.
>
> Thanks
> -min
>
> On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>
>> Hi Min,
>> When you say tomorrow, what date is it 11th June or 12th ? Can the
>> time be preponed by an hour please - its too late here ?
>>
>> Thanks,
>> -Nitin
>>
>> On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>>
>>> Hi there,
>>>
>>> To reach consensus on some remaining NFS cache issues on
>>> object_store storage refactor work in a more effective manner, John,
>>> Edison and I have scheduled a GoToMeeting tomorrow to discuss them
>>> over the phone, any interested parties are welcome to join and
>>> brainstorm. Here are detailed GTM information:
>>>
>>> Meeting Time: 10:30 AM 12:30 PM PST
>>>
>>> Meeting Details:
>>>
>>> 1. Please join my meeting.
>>> https://www1.gotomeeting.com/join/188620897
>>>
>>> 2. Use your microphone and speakers (VoIP) - a headset is recommended.
>>> Or, call in using your telephone.
>>>
>>> United States: +1 (626) 521-0017
>>> United States (toll-free): 1 877 309 2070
>>>
>>> Access Code: 188-620-897
>>> Audio PIN: Shown after joining the meeting
>>>
>>> Meeting ID: 188-620-897
>>>
>>> GoToMeeting(r)
>>> Online Meetings Made Easy(r)
>>>
>>> Not at your computer? Click the link to join this meeting from your
>>> iPhone(r), iPad(r) or Android(r) device via the GoToMeeting app.
>>>
>>> Thanks
>>> -min
>>
>
Re: Object_Store storage refactor Meeting Notes
Posted by Min Chen <mi...@citrix.com>.
One comment regarding upgrade path: due to internal DB schema change
documented in our FS:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backup+Objec
t+Store+Plugin+Framework, we do need to handle upgrade cases in 4.2, which
are mainly related to data migration in DB level. I am working on that
right now, will check in a version to object_branch today.
Thanks
-min
On 6/14/13 7:17 AM, "John Burwell" <jb...@basho.com> wrote:
>Nitin,
>
>Please see my comments in-line below.
>
>Thanks,
>-John
>
>On Jun 14, 2013, at 4:16 AM, Nitin Mehta <Ni...@citrix.com> wrote:
>
>>
>>
>> On 13/06/13 10:08 PM, "John Burwell" <jb...@basho.com> wrote:
>>
>>> All,
>>>
>>> Edison Su, Min Chen, Animesh Chaturvedi, and myself met via
>>> teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting
>>> was determine the path forward for merging the object_store branch by
>>>the
>>> 4.2 freeze date, 30 June 2013. The conversation focused on the
>>>following
>>> topics:
>>>
>>> * Staging area mechanism
>>> * Removing dependencies from the Storage to the Hypervisor layer
>>> * Dependencies of other patches on object_store
>>> * QA's desire to start testing the patch ASAP
>>>
>>> Min, Edison, and I agreed that the staging mechanism must age out files
>>> and use a reference count to ensure that file in-use are not
>>>prematurely
>>> purged. While we agree that some form of reservation system is
>>>required,
>>> Edison is concerned that it will be too conservative and create
>>> bottlenecks.
>>
>> Can you please elaborate on the fact that it is too conservative - we
>>just
>> can't purge the files when they are still in use correct ? We can use a
>> combination of LRU + reference count, trying to purge the LRU files if
>> their reference count <= 0 as a start ?
>
>The issue is not around determining when to purge a file. The issue
>emerges around reservation sizes. Currently, if we take a snapshot of a
>2 TB volume, we would have to reserve 2 TB in the staging area to ensure
>that we would have enough space for the maximum potential size of the
>snapshot. However, it is very unlikely that the snapshot will actually
>be this size. The concern becomes that large reservations would start
>starving out other processes. For 4.2, we didn't feel there was enough
>time to devise a "smarter" reservation mechanism. Therefore, in 4.3,
>there should be time to think the implications of various approaches
>through and devise a more efficient approach.
>
>>
>>>
>>> As we delved deeper into the subject of the storage to hypervisor
>>> dependencies and the reservation mechanism, we determined that NFS
>>> storage would still need to be the size of the secondary storage data
>>> set. Since the hypervisor layer has not been completely fitted to the
>>> new storage layer, NFS would be still required for a number of
>>> operations. Based on this realization, we decided to de-scope the
>>> staging mechanism, and leave the 4.2 object store functionality the
>>>same
>>> as 4.1. Therefore, NFS will remain the secondary storage of record,
>>>and
>>> object storage will serve as backup/multi-zone sync.
>>
>> I am not sure how we can comment its going to be the same as 4.1 - is it
>> from the end user perspective ? The internal semantics and their flow
>>have
>> changed. This needs to be elaborated and properly documented. Also I am
>> not sure if the feature is supported on the upgrade path or is it ? Need
>> more documentation here.
>
>From an end user perspective, object stores will remain a backup of
>secondary storage. The user interface will likely be a bit nicer, but in
>terms of system architecture, the roles of object storage and NFS remain
>the same in 4.2 and 4.1. To my mind, when we support object stores as
>native secondary storage targets, it will be a new feature, and we should
>continue to support the backup model as well. Therefore, I don't see an
>upgrade path issue.
>
>>
>>
>>> In 4.3, we will fit the hypervisor layer for the new storage layer
>>>which
>>> will allow object stores to server as secondary storage. This work
>>>will
>>> include removing the storage to hypervisor dependencies. For 4.2,
>>>Edison
>>> and Min have implemented the critical foundation necessary to establish
>>> our next generation storage layer. There simply was not enough time in
>>> this development cycle to make these changes and fit the hypervisor
>>>layer.
>>>
>>> Due to the size of the patch, Animesh voiced QA's concerned regarding
>>> test scope and impact. As such, we want to get the merge completed as
>>> soon as possible to allow testing to begin. We discussed breaking up
>>>the
>>> patch, but we could not devise a reasonable set of chunks there were
>>>both
>>> isolated and significantly testable. Therefore, the patch can only be
>>> delivered in its current state. We also walked through potential
>>> dependencies between the storage framework changes and the solidfire
>>> branch. It was determined that these two merges could occur
>>> independently.
>>>
>>> Finally, Animesh is going to setup a meeting at Citrix's Santa Clara
>>> office on 26 June 2013 (the day after Collab ends) to discuss the path
>>> forward for 4.3 and work through a high-level design/approach to
>>>fitting
>>> the hypervisor layer to exploit the new storage capabilities. Details
>>> will be published to the dev mailing list.
>>>
>>> Thanks,
>>> -John
>>>
>>> On Jun 11, 2013, at 2:08 AM, Min Chen <mi...@citrix.com> wrote:
>>>
>>>> It is 11th June. John is not free between 9:15am to 10am, that is why
>>>>we
>>>> schedule it at 10:30am.
>>>>
>>>> Thanks
>>>> -min
>>>>
>>>> On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>>>>
>>>>> Hi Min,
>>>>> When you say tomorrow, what date is it 11th June or 12th ? Can the
>>>>> time be
>>>>> preponed by an hour please - its too late here ?
>>>>>
>>>>> Thanks,
>>>>> -Nitin
>>>>>
>>>>> On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> To reach consensus on some remaining NFS cache issues on
>>>>>>object_store
>>>>>> storage refactor work in a more effective manner, John, Edison and I
>>>>>> have
>>>>>> scheduled a GoToMeeting tomorrow to discuss them over the phone, any
>>>>>> interested parties are welcome to join and brainstorm. Here are
>>>>>> detailed
>>>>>> GTM information:
>>>>>>
>>>>>> Meeting Time: 10:30 AM 12:30 PM PST
>>>>>>
>>>>>> Meeting Details:
>>>>>>
>>>>>> 1. Please join my meeting.
>>>>>> https://www1.gotomeeting.com/join/188620897
>>>>>>
>>>>>> 2. Use your microphone and speakers (VoIP) - a headset is
>>>>>> recommended.
>>>>>> Or, call in using your telephone.
>>>>>>
>>>>>> United States: +1 (626) 521-0017
>>>>>> United States (toll-free): 1 877 309 2070
>>>>>>
>>>>>> Access Code: 188-620-897
>>>>>> Audio PIN: Shown after joining the meeting
>>>>>>
>>>>>> Meeting ID: 188-620-897
>>>>>>
>>>>>> GoToMeeting®
>>>>>> Online Meetings Made Easy®
>>>>>>
>>>>>> Not at your computer? Click the link to join this meeting from your
>>>>>> iPhone®, iPad® or Android® device via the GoToMeeting app.
>>>>>>
>>>>>> Thanks
>>>>>> -min
>>>>>
>>>>
>>>
>>
>
Re: Object_Store storage refactor Meeting Notes
Posted by John Burwell <jb...@basho.com>.
Nitin,
Please see my comments in-line below.
Thanks,
-John
On Jun 14, 2013, at 4:16 AM, Nitin Mehta <Ni...@citrix.com> wrote:
>
>
> On 13/06/13 10:08 PM, "John Burwell" <jb...@basho.com> wrote:
>
>> All,
>>
>> Edison Su, Min Chen, Animesh Chaturvedi, and myself met via
>> teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting
>> was determine the path forward for merging the object_store branch by the
>> 4.2 freeze date, 30 June 2013. The conversation focused on the following
>> topics:
>>
>> * Staging area mechanism
>> * Removing dependencies from the Storage to the Hypervisor layer
>> * Dependencies of other patches on object_store
>> * QA's desire to start testing the patch ASAP
>>
>> Min, Edison, and I agreed that the staging mechanism must age out files
>> and use a reference count to ensure that file in-use are not prematurely
>> purged. While we agree that some form of reservation system is required,
>> Edison is concerned that it will be too conservative and create
>> bottlenecks.
>
> Can you please elaborate on the fact that it is too conservative - we just
> can't purge the files when they are still in use correct ? We can use a
> combination of LRU + reference count, trying to purge the LRU files if
> their reference count <= 0 as a start ?
The issue is not around determining when to purge a file. The issue emerges around reservation sizes. Currently, if we take a snapshot of a 2 TB volume, we would have to reserve 2 TB in the staging area to ensure that we would have enough space for the maximum potential size of the snapshot. However, it is very unlikely that the snapshot will actually be this size. The concern becomes that large reservations would start starving out other processes. For 4.2, we didn't feel there was enough time to devise a "smarter" reservation mechanism. Therefore, in 4.3, there should be time to think the implications of various approaches through and devise a more efficient approach.
>
>>
>> As we delved deeper into the subject of the storage to hypervisor
>> dependencies and the reservation mechanism, we determined that NFS
>> storage would still need to be the size of the secondary storage data
>> set. Since the hypervisor layer has not been completely fitted to the
>> new storage layer, NFS would be still required for a number of
>> operations. Based on this realization, we decided to de-scope the
>> staging mechanism, and leave the 4.2 object store functionality the same
>> as 4.1. Therefore, NFS will remain the secondary storage of record, and
>> object storage will serve as backup/multi-zone sync.
>
> I am not sure how we can comment its going to be the same as 4.1 - is it
> from the end user perspective ? The internal semantics and their flow have
> changed. This needs to be elaborated and properly documented. Also I am
> not sure if the feature is supported on the upgrade path or is it ? Need
> more documentation here.
From an end user perspective, object stores will remain a backup of secondary storage. The user interface will likely be a bit nicer, but in terms of system architecture, the roles of object storage and NFS remain the same in 4.2 and 4.1. To my mind, when we support object stores as native secondary storage targets, it will be a new feature, and we should continue to support the backup model as well. Therefore, I don't see an upgrade path issue.
>
>
>> In 4.3, we will fit the hypervisor layer for the new storage layer which
>> will allow object stores to server as secondary storage. This work will
>> include removing the storage to hypervisor dependencies. For 4.2, Edison
>> and Min have implemented the critical foundation necessary to establish
>> our next generation storage layer. There simply was not enough time in
>> this development cycle to make these changes and fit the hypervisor layer.
>>
>> Due to the size of the patch, Animesh voiced QA's concerned regarding
>> test scope and impact. As such, we want to get the merge completed as
>> soon as possible to allow testing to begin. We discussed breaking up the
>> patch, but we could not devise a reasonable set of chunks there were both
>> isolated and significantly testable. Therefore, the patch can only be
>> delivered in its current state. We also walked through potential
>> dependencies between the storage framework changes and the solidfire
>> branch. It was determined that these two merges could occur
>> independently.
>>
>> Finally, Animesh is going to setup a meeting at Citrix's Santa Clara
>> office on 26 June 2013 (the day after Collab ends) to discuss the path
>> forward for 4.3 and work through a high-level design/approach to fitting
>> the hypervisor layer to exploit the new storage capabilities. Details
>> will be published to the dev mailing list.
>>
>> Thanks,
>> -John
>>
>> On Jun 11, 2013, at 2:08 AM, Min Chen <mi...@citrix.com> wrote:
>>
>>> It is 11th June. John is not free between 9:15am to 10am, that is why we
>>> schedule it at 10:30am.
>>>
>>> Thanks
>>> -min
>>>
>>> On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>>>
>>>> Hi Min,
>>>> When you say tomorrow, what date is it 11th June or 12th ? Can the
>>>> time be
>>>> preponed by an hour please - its too late here ?
>>>>
>>>> Thanks,
>>>> -Nitin
>>>>
>>>> On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> To reach consensus on some remaining NFS cache issues on object_store
>>>>> storage refactor work in a more effective manner, John, Edison and I
>>>>> have
>>>>> scheduled a GoToMeeting tomorrow to discuss them over the phone, any
>>>>> interested parties are welcome to join and brainstorm. Here are
>>>>> detailed
>>>>> GTM information:
>>>>>
>>>>> Meeting Time: 10:30 AM 12:30 PM PST
>>>>>
>>>>> Meeting Details:
>>>>>
>>>>> 1. Please join my meeting.
>>>>> https://www1.gotomeeting.com/join/188620897
>>>>>
>>>>> 2. Use your microphone and speakers (VoIP) - a headset is
>>>>> recommended.
>>>>> Or, call in using your telephone.
>>>>>
>>>>> United States: +1 (626) 521-0017
>>>>> United States (toll-free): 1 877 309 2070
>>>>>
>>>>> Access Code: 188-620-897
>>>>> Audio PIN: Shown after joining the meeting
>>>>>
>>>>> Meeting ID: 188-620-897
>>>>>
>>>>> GoToMeeting®
>>>>> Online Meetings Made Easy®
>>>>>
>>>>> Not at your computer? Click the link to join this meeting from your
>>>>> iPhone®, iPad® or Android® device via the GoToMeeting app.
>>>>>
>>>>> Thanks
>>>>> -min
>>>>
>>>
>>
>
Re: Object_Store storage refactor Meeting Notes
Posted by Nitin Mehta <Ni...@citrix.com>.
On 13/06/13 10:08 PM, "John Burwell" <jb...@basho.com> wrote:
>All,
>
>Edison Su, Min Chen, Animesh Chaturvedi, and myself met via
>teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting
>was determine the path forward for merging the object_store branch by the
>4.2 freeze date, 30 June 2013. The conversation focused on the following
>topics:
>
> * Staging area mechanism
> * Removing dependencies from the Storage to the Hypervisor layer
> * Dependencies of other patches on object_store
> * QA's desire to start testing the patch ASAP
>
>Min, Edison, and I agreed that the staging mechanism must age out files
>and use a reference count to ensure that file in-use are not prematurely
>purged. While we agree that some form of reservation system is required,
>Edison is concerned that it will be too conservative and create
>bottlenecks.
Can you please elaborate on the fact that it is too conservative - we just
can't purge the files when they are still in use correct ? We can use a
combination of LRU + reference count, trying to purge the LRU files if
their reference count <= 0 as a start ?
>
>As we delved deeper into the subject of the storage to hypervisor
>dependencies and the reservation mechanism, we determined that NFS
>storage would still need to be the size of the secondary storage data
>set. Since the hypervisor layer has not been completely fitted to the
>new storage layer, NFS would be still required for a number of
>operations. Based on this realization, we decided to de-scope the
>staging mechanism, and leave the 4.2 object store functionality the same
>as 4.1. Therefore, NFS will remain the secondary storage of record, and
>object storage will serve as backup/multi-zone sync.
I am not sure how we can comment its going to be the same as 4.1 - is it
from the end user perspective ? The internal semantics and their flow have
changed. This needs to be elaborated and properly documented. Also I am
not sure if the feature is supported on the upgrade path or is it ? Need
more documentation here.
>In 4.3, we will fit the hypervisor layer for the new storage layer which
>will allow object stores to server as secondary storage. This work will
>include removing the storage to hypervisor dependencies. For 4.2, Edison
>and Min have implemented the critical foundation necessary to establish
>our next generation storage layer. There simply was not enough time in
>this development cycle to make these changes and fit the hypervisor layer.
>
>Due to the size of the patch, Animesh voiced QA's concerned regarding
>test scope and impact. As such, we want to get the merge completed as
>soon as possible to allow testing to begin. We discussed breaking up the
>patch, but we could not devise a reasonable set of chunks there were both
>isolated and significantly testable. Therefore, the patch can only be
>delivered in its current state. We also walked through potential
>dependencies between the storage framework changes and the solidfire
>branch. It was determined that these two merges could occur
>independently.
>
>Finally, Animesh is going to setup a meeting at Citrix's Santa Clara
>office on 26 June 2013 (the day after Collab ends) to discuss the path
>forward for 4.3 and work through a high-level design/approach to fitting
>the hypervisor layer to exploit the new storage capabilities. Details
>will be published to the dev mailing list.
>
>Thanks,
>-John
>
>On Jun 11, 2013, at 2:08 AM, Min Chen <mi...@citrix.com> wrote:
>
>> It is 11th June. John is not free between 9:15am to 10am, that is why we
>> schedule it at 10:30am.
>>
>> Thanks
>> -min
>>
>> On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>>
>>> Hi Min,
>>> When you say tomorrow, what date is it 11th June or 12th ? Can the
>>>time be
>>> preponed by an hour please - its too late here ?
>>>
>>> Thanks,
>>> -Nitin
>>>
>>> On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>>>
>>>> Hi there,
>>>>
>>>> To reach consensus on some remaining NFS cache issues on object_store
>>>> storage refactor work in a more effective manner, John, Edison and I
>>>>have
>>>> scheduled a GoToMeeting tomorrow to discuss them over the phone, any
>>>> interested parties are welcome to join and brainstorm. Here are
>>>>detailed
>>>> GTM information:
>>>>
>>>> Meeting Time: 10:30 AM 12:30 PM PST
>>>>
>>>> Meeting Details:
>>>>
>>>> 1. Please join my meeting.
>>>> https://www1.gotomeeting.com/join/188620897
>>>>
>>>> 2. Use your microphone and speakers (VoIP) - a headset is
>>>>recommended.
>>>> Or, call in using your telephone.
>>>>
>>>> United States: +1 (626) 521-0017
>>>> United States (toll-free): 1 877 309 2070
>>>>
>>>> Access Code: 188-620-897
>>>> Audio PIN: Shown after joining the meeting
>>>>
>>>> Meeting ID: 188-620-897
>>>>
>>>> GoToMeeting®
>>>> Online Meetings Made Easy®
>>>>
>>>> Not at your computer? Click the link to join this meeting from your
>>>> iPhone®, iPad® or Android® device via the GoToMeeting app.
>>>>
>>>> Thanks
>>>> -min
>>>
>>
>
Object_Store storage refactor Meeting Notes
Posted by John Burwell <jb...@basho.com>.
All,
Edison Su, Min Chen, Animesh Chaturvedi, and myself met via teleconference on 11 June 2013 @ 1:30 PM EDT. The goal of the meeting was determine the path forward for merging the object_store branch by the 4.2 freeze date, 30 June 2013. The conversation focused on the following topics:
* Staging area mechanism
* Removing dependencies from the Storage to the Hypervisor layer
* Dependencies of other patches on object_store
* QA's desire to start testing the patch ASAP
Min, Edison, and I agreed that the staging mechanism must age out files and use a reference count to ensure that file in-use are not prematurely purged. While we agree that some form of reservation system is required, Edison is concerned that it will be too conservative and create bottlenecks.
As we delved deeper into the subject of the storage to hypervisor dependencies and the reservation mechanism, we determined that NFS storage would still need to be the size of the secondary storage data set. Since the hypervisor layer has not been completely fitted to the new storage layer, NFS would be still required for a number of operations. Based on this realization, we decided to de-scope the staging mechanism, and leave the 4.2 object store functionality the same as 4.1. Therefore, NFS will remain the secondary storage of record, and object storage will serve as backup/multi-zone sync. In 4.3, we will fit the hypervisor layer for the new storage layer which will allow object stores to server as secondary storage. This work will include removing the storage to hypervisor dependencies. For 4.2, Edison and Min have implemented the critical foundation necessary to establish our next generation storage layer. There simply was not enough time in this development cycle to make these changes and fit the hypervisor layer.
Due to the size of the patch, Animesh voiced QA's concerned regarding test scope and impact. As such, we want to get the merge completed as soon as possible to allow testing to begin. We discussed breaking up the patch, but we could not devise a reasonable set of chunks there were both isolated and significantly testable. Therefore, the patch can only be delivered in its current state. We also walked through potential dependencies between the storage framework changes and the solidfire branch. It was determined that these two merges could occur independently.
Finally, Animesh is going to setup a meeting at Citrix's Santa Clara office on 26 June 2013 (the day after Collab ends) to discuss the path forward for 4.3 and work through a high-level design/approach to fitting the hypervisor layer to exploit the new storage capabilities. Details will be published to the dev mailing list.
Thanks,
-John
On Jun 11, 2013, at 2:08 AM, Min Chen <mi...@citrix.com> wrote:
> It is 11th June. John is not free between 9:15am to 10am, that is why we
> schedule it at 10:30am.
>
> Thanks
> -min
>
> On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>
>> Hi Min,
>> When you say tomorrow, what date is it 11th June or 12th ? Can the time be
>> preponed by an hour please - its too late here ?
>>
>> Thanks,
>> -Nitin
>>
>> On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>>
>>> Hi there,
>>>
>>> To reach consensus on some remaining NFS cache issues on object_store
>>> storage refactor work in a more effective manner, John, Edison and I have
>>> scheduled a GoToMeeting tomorrow to discuss them over the phone, any
>>> interested parties are welcome to join and brainstorm. Here are detailed
>>> GTM information:
>>>
>>> Meeting Time: 10:30 AM 12:30 PM PST
>>>
>>> Meeting Details:
>>>
>>> 1. Please join my meeting.
>>> https://www1.gotomeeting.com/join/188620897
>>>
>>> 2. Use your microphone and speakers (VoIP) - a headset is recommended.
>>> Or, call in using your telephone.
>>>
>>> United States: +1 (626) 521-0017
>>> United States (toll-free): 1 877 309 2070
>>>
>>> Access Code: 188-620-897
>>> Audio PIN: Shown after joining the meeting
>>>
>>> Meeting ID: 188-620-897
>>>
>>> GoToMeeting®
>>> Online Meetings Made Easy®
>>>
>>> Not at your computer? Click the link to join this meeting from your
>>> iPhone®, iPad® or Android® device via the GoToMeeting app.
>>>
>>> Thanks
>>> -min
>>
>
Re: Object_Store storage refactor GoToMeeting Tomorrow
Posted by Min Chen <mi...@citrix.com>.
It is 11th June. John is not free between 9:15am to 10am, that is why we
schedule it at 10:30am.
Thanks
-min
On 6/10/13 10:52 PM, "Nitin Mehta" <Ni...@citrix.com> wrote:
>Hi Min,
>When you say tomorrow, what date is it 11th June or 12th ? Can the time be
>preponed by an hour please - its too late here ?
>
>Thanks,
>-Nitin
>
>On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>
>>Hi there,
>>
>>To reach consensus on some remaining NFS cache issues on object_store
>>storage refactor work in a more effective manner, John, Edison and I have
>>scheduled a GoToMeeting tomorrow to discuss them over the phone, any
>>interested parties are welcome to join and brainstorm. Here are detailed
>>GTM information:
>>
>>Meeting Time: 10:30 AM 12:30 PM PST
>>
>>Meeting Details:
>>
>>1. Please join my meeting.
>>https://www1.gotomeeting.com/join/188620897
>>
>>2. Use your microphone and speakers (VoIP) - a headset is recommended.
>>Or, call in using your telephone.
>>
>>United States: +1 (626) 521-0017
>>United States (toll-free): 1 877 309 2070
>>
>>Access Code: 188-620-897
>>Audio PIN: Shown after joining the meeting
>>
>>Meeting ID: 188-620-897
>>
>>GoToMeeting®
>>Online Meetings Made Easy®
>>
>>Not at your computer? Click the link to join this meeting from your
>>iPhone®, iPad® or Android® device via the GoToMeeting app.
>>
>>Thanks
>>-min
>
Re: Object_Store storage refactor GoToMeeting Tomorrow
Posted by Nitin Mehta <Ni...@citrix.com>.
Hi Min,
When you say tomorrow, what date is it 11th June or 12th ? Can the time be
preponed by an hour please - its too late here ?
Thanks,
-Nitin
On 11/06/13 11:06 AM, "Min Chen" <mi...@citrix.com> wrote:
>Hi there,
>
>To reach consensus on some remaining NFS cache issues on object_store
>storage refactor work in a more effective manner, John, Edison and I have
>scheduled a GoToMeeting tomorrow to discuss them over the phone, any
>interested parties are welcome to join and brainstorm. Here are detailed
>GTM information:
>
>Meeting Time: 10:30 AM 12:30 PM PST
>
>Meeting Details:
>
>1. Please join my meeting.
>https://www1.gotomeeting.com/join/188620897
>
>2. Use your microphone and speakers (VoIP) - a headset is recommended.
>Or, call in using your telephone.
>
>United States: +1 (626) 521-0017
>United States (toll-free): 1 877 309 2070
>
>Access Code: 188-620-897
>Audio PIN: Shown after joining the meeting
>
>Meeting ID: 188-620-897
>
>GoToMeeting®
>Online Meetings Made Easy®
>
>Not at your computer? Click the link to join this meeting from your
>iPhone®, iPad® or Android® device via the GoToMeeting app.
>
>Thanks
>-min