You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bloodhound.apache.org by Gary <ga...@physics.org> on 2018/03/11 21:21:44 UTC
[Proposal] Core Bloodhound - basic concepts
Hi everyone,
Sorry it has taken so long to get this written. With the understanding that we have expressed the desire to move away from Trac as the base for the project, I'd like to put forward the following as an initial proposal for the way we progress. You should not expect that this is high on details. I only intend to provide some guidance around the concepts that I would like to see used. I am hoping that this will be enough to get a reasonable idea of what the core model might look like.
In broad terms I am proposing that we
* take the opportunity to design the project from the ground up as opposed to attempting to build up the existing interfaces
* build the project using the Django web framework
* separate out a core sub-project with a well defined REST api for a frontend UI sub-project to build upon
The rest of this proposal sets out to define parts of the core sub-project.
The core should provide enough services to support the main part of a issue tracking UI. I am also expecting there to be further sub-projects to support additional optional functionality.
The core sub-project will probably provide a basic UI to make progress ahead of work on the UI sub-project. I am not looking to define the UI sub-project any further in this proposal although it is likely that it will be highly dynamic, perhaps making use of websockets. The basic UI of the core project should remain well-maintained throughout the life of the project but at the moment my expectation would be that it is a no-thrills interface.
Back to the core sub-project, I am looking for this to be built on the following concepts:
#0 a change in nomenclature from product to tracker as the top level
#1 uuids for pretty much everything
#2 tickets as the accumulation of ticket change events
#3 'labels' and 'label types' as a unifying concept for categorisation of tickets
One of the early decisions made in handling of multiple projects in bloodhound was to call this feature multi-product. The reasoning for this was that the term 'project' is overloaded (if I recall, there may have been an excuse to add projects as a layer in bloodhound that could be used to group tickets across products.) I would like to see the basic functionality brought in to handle the ability to host multiple projects (in the top level organisation sense) but change the nomenclature to tracker to be as generic as possible. To be fair, the name of the feature may not matter so much as I would also like to see users given the ability to define their levels of organisation to a greater extent. More on this in the discussion of #3.
Note that I will use the term tracker in the rest of this proposal where you might prefer to see product or project.
The idea behind #1, uuids, is that it may be useful to be able to unambiguously identify instances of objects even across bloodhound instances. Pending decisions on how tickets are referenced, these uuids could be used as a reference that is unchanging as it moved between trackers.
How tickets are stored as a whole is also worth tackling, which is what #2 is trying to broadly decide. Here I am suggesting that the state of a ticket can be built up from a query that collects all the change events and applies the deltas in order. This could prove to be a slow process so at some point we may want to look at keeping a record of the current state or a checkpoint but, given that ticket views are expected to show the history, these are details that are required anyway. I suggest that we can look at optimising for speed later. Knowing that we can update a ticket from deltas may be useful.
Finally, I am suggesting in #3 that as much as possible we generalise ticket categorisation. Categorisation is a central concept to capture and, in trac we inherited categorisations such as statuses (open, in progress, etc), types (bug, enhancement, task, etc), milestones, versions, etc, and these had separate implementations. What I would like to see is that, while we might provide an initial set of ways to categorise tickets, we should allow users to add the categorisations that they require. I'm suggesting calling this generalisation 'labels' and 'label types' (though we can bikeshed that if we like!)
There are a few ways to look at implementing labels as we could look to capture a set of common kinds of collections so that there are types of 'label types', as it were. What I mean by this is that you can have labels that are simply a tag, perhaps some types of labels should have descriptions, some might have dates associated with them. Beyond that we would need to know where a label can be allowed to coexist on a ticket with others of the same 'label type'. A lot of these aspects could influence how you might want to display them.
At the moment I believe that we can probably get away with a common model across all label types and we can provide users with the tools to define what you can do with labels of a given label type via the properties of that label type.
As you can see I have a lot to say about these labels. Another thing that could be useful would be to be able to have label types and labels that are available to a specific tracker, globally or even to a set of trackers. To deliver this, label types and labels need to be able to be associated with a variable number of trackers.
We also need to be able to define workflows in terms of transitions between labels and potentially how they are affected by other labels. I think it is suffice to say that I would like to see an implementation where users are able to define workflow transitions with a great deal of flexibility in the label types that are involved.
From the point of view of the transition from the trac model to the desired model, we can attempt to make use of Django's migrations from autogenerated models as brought to our attention by Daniel Brownridge. I do want to ensure that there is a viable migration path in place from the old bloodhound to the new.
I think this is about as far as I want the proposal to go for now. Obviously there is going to be plenty more to decide.
Please let me know your thoughts.
Cheers,
Gary
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Gary <ga...@physics.org>.
Hi Allan,
Thank you for the suggestions. Please take my word for it that the input is definitely appreciated.
I do want to delay discussion on how the front-end could work, or at least limit that discussion in this thread. The main thing I want to focus on here is getting agreement on the core platform. OK, temptation gets the better of me.. hopefully it is enough to say that I would see React as a reasonable choice for helping build the web front-end but Allan appears to have the gist of what I am expecting.
I'm also not hugely worried about anything being particularly slow for the moment. Slow operations will be possible to speed up later. I will attempt to keep the scheduler method in mind for the future as I might not have come up with that as an obvious option!
Cheers,
Gary
On Sun, 11 Mar 2018, at 9:35 PM, Allan Swanepoel wrote:
> Possibly take a look at what has already been done using Django in this
> arena?
> One project that jumps out with some Google foo is
> https://djacket.github.io/
>
> Also, if you want to separate (loosely couple) the front-end and the
> back-end, possibly build the whole front-end with some js framework like
> react?
> All the functionality would be provided by the api, and all you need is
> some Ajax calls.
>
> For the slow stuff, possibly make these more asynchronous and use some form
> of scheduler (celery / ampq bus) to offload the work. Again, the result is
> posted back via the api.
>
> Just my 2c's worth
>
> On 11 Mar 2018 23:21, "Gary" <ga...@physics.org> wrote:
>
> > Hi everyone,
> >
> > Sorry it has taken so long to get this written. With the understanding
> > that we have expressed the desire to move away from Trac as the base for
> > the project, I'd like to put forward the following as an initial proposal
> > for the way we progress. You should not expect that this is high on
> > details. I only intend to provide some guidance around the concepts that I
> > would like to see used. I am hoping that this will be enough to get a
> > reasonable idea of what the core model might look like.
> >
> > In broad terms I am proposing that we
> > * take the opportunity to design the project from the ground up as opposed
> > to attempting to build up the existing interfaces
> > * build the project using the Django web framework
> > * separate out a core sub-project with a well defined REST api for a
> > frontend UI sub-project to build upon
> >
> > The rest of this proposal sets out to define parts of the core sub-project.
> >
> > The core should provide enough services to support the main part of a
> > issue tracking UI. I am also expecting there to be further sub-projects to
> > support additional optional functionality.
> >
> > The core sub-project will probably provide a basic UI to make progress
> > ahead of work on the UI sub-project. I am not looking to define the UI
> > sub-project any further in this proposal although it is likely that it will
> > be highly dynamic, perhaps making use of websockets. The basic UI of the
> > core project should remain well-maintained throughout the life of the
> > project but at the moment my expectation would be that it is a no-thrills
> > interface.
> >
> > Back to the core sub-project, I am looking for this to be built on the
> > following concepts:
> >
> > #0 a change in nomenclature from product to tracker as the top level
> > #1 uuids for pretty much everything
> > #2 tickets as the accumulation of ticket change events
> > #3 'labels' and 'label types' as a unifying concept for categorisation of
> > tickets
> >
> > One of the early decisions made in handling of multiple projects in
> > bloodhound was to call this feature multi-product. The reasoning for this
> > was that the term 'project' is overloaded (if I recall, there may have been
> > an excuse to add projects as a layer in bloodhound that could be used to
> > group tickets across products.) I would like to see the basic functionality
> > brought in to handle the ability to host multiple projects (in the top
> > level organisation sense) but change the nomenclature to tracker to be as
> > generic as possible. To be fair, the name of the feature may not matter so
> > much as I would also like to see users given the ability to define their
> > levels of organisation to a greater extent. More on this in the discussion
> > of #3.
> >
> > Note that I will use the term tracker in the rest of this proposal where
> > you might prefer to see product or project.
> >
> > The idea behind #1, uuids, is that it may be useful to be able to
> > unambiguously identify instances of objects even across bloodhound
> > instances. Pending decisions on how tickets are referenced, these uuids
> > could be used as a reference that is unchanging as it moved between
> > trackers.
> >
> > How tickets are stored as a whole is also worth tackling, which is what #2
> > is trying to broadly decide. Here I am suggesting that the state of a
> > ticket can be built up from a query that collects all the change events and
> > applies the deltas in order. This could prove to be a slow process so at
> > some point we may want to look at keeping a record of the current state or
> > a checkpoint but, given that ticket views are expected to show the history,
> > these are details that are required anyway. I suggest that we can look at
> > optimising for speed later. Knowing that we can update a ticket from deltas
> > may be useful.
> >
> > Finally, I am suggesting in #3 that as much as possible we generalise
> > ticket categorisation. Categorisation is a central concept to capture and,
> > in trac we inherited categorisations such as statuses (open, in progress,
> > etc), types (bug, enhancement, task, etc), milestones, versions, etc, and
> > these had separate implementations. What I would like to see is that, while
> > we might provide an initial set of ways to categorise tickets, we should
> > allow users to add the categorisations that they require. I'm suggesting
> > calling this generalisation 'labels' and 'label types' (though we can
> > bikeshed that if we like!)
> >
> > There are a few ways to look at implementing labels as we could look to
> > capture a set of common kinds of collections so that there are types of
> > 'label types', as it were. What I mean by this is that you can have labels
> > that are simply a tag, perhaps some types of labels should have
> > descriptions, some might have dates associated with them. Beyond that we
> > would need to know where a label can be allowed to coexist on a ticket with
> > others of the same 'label type'. A lot of these aspects could influence how
> > you might want to display them.
> >
> > At the moment I believe that we can probably get away with a common model
> > across all label types and we can provide users with the tools to define
> > what you can do with labels of a given label type via the properties of
> > that label type.
> >
> > As you can see I have a lot to say about these labels. Another thing that
> > could be useful would be to be able to have label types and labels that are
> > available to a specific tracker, globally or even to a set of trackers. To
> > deliver this, label types and labels need to be able to be associated with
> > a variable number of trackers.
> >
> > We also need to be able to define workflows in terms of transitions
> > between labels and potentially how they are affected by other labels. I
> > think it is suffice to say that I would like to see an implementation where
> > users are able to define workflow transitions with a great deal of
> > flexibility in the label types that are involved.
> >
> > From the point of view of the transition from the trac model to the
> > desired model, we can attempt to make use of Django's migrations from
> > autogenerated models as brought to our attention by Daniel Brownridge. I do
> > want to ensure that there is a viable migration path in place from the old
> > bloodhound to the new.
> >
> > I think this is about as far as I want the proposal to go for now.
> > Obviously there is going to be plenty more to decide.
> >
> > Please let me know your thoughts.
> >
> > Cheers,
> > Gary
> >
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Gary Martin <ga...@physics.org>.
On Mon, 19 Mar 2018, at 2:09 PM, Greg Stein wrote:
> On Mon, Mar 19, 2018 at 8:43 AM, Gary Martin <ga...@physics.org>
> wrote:
>
> > Hi everyone,
> >
> > I am going to start building out a basic core implementation to play
> > around with some of the ideas we have discussed. I'll try not to take
> > things too far before getting this into source control.
>
>
> I would recommend using an svn branch, and committing "hourly". Why not?
> You won't be affecting trunk, and you'll get early feedback.
That is true. I can obviously avoid the need to discover how to create the git repo by just using a branch for the experiment of course.
> >...
>
> > On that last item, I believe I have the ability to create a git repo for
> > the project at the asf. I would expect the project to also be available via
> > github.
> >
>
> Infra would like to see some community consensus, if you plan to switch
> from svn to git. (ie. discuss it first)
You may well be right that there has not been enough discussion on this although it has been mentioned a number of times. I will make it a firm proposal again before taking that suggestion any further.
Cheers,
Gary
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Greg Stein <gs...@gmail.com>.
On Mon, Mar 19, 2018 at 8:43 AM, Gary Martin <ga...@physics.org>
wrote:
> Hi everyone,
>
> I am going to start building out a basic core implementation to play
> around with some of the ideas we have discussed. I'll try not to take
> things too far before getting this into source control.
I would recommend using an svn branch, and committing "hourly". Why not?
You won't be affecting trunk, and you'll get early feedback.
>...
> On that last item, I believe I have the ability to create a git repo for
> the project at the asf. I would expect the project to also be available via
> github.
>
Infra would like to see some community consensus, if you plan to switch
from svn to git. (ie. discuss it first)
>...
Cheers,
-g
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Gary Martin <ga...@physics.org>.
Hi everyone,
I am going to start building out a basic core implementation to play around with some of the ideas we have discussed. I'll try not to take things too far before getting this into source control. I am having a quick play with pipenv [1] as the new base for helping to develop and install the project. I'll also be hoping to get unit testing and some selenium tests in place relatively early along with some basic instructions for getting started with development.
At the moment I am expecting that we will be working with python >=3.5, we are using django >= 2.0, unit testing probably to be based on pytest [2] and that there is a desire to use a git repo for the work.
On that last item, I believe I have the ability to create a git repo for the project at the asf. I would expect the project to also be available via github.
Anything else I may have missed?
Cheers,
Gary
[1] https://docs.pipenv.org/
[2] https://docs.pytest.org/
On Thu, 15 Mar 2018, at 1:01 AM, Gary Martin wrote:
> On Mon, 12 Mar 2018, at 12:10 PM, Jason Morgan wrote:
> > Rather than Celery, [...]
>
> I was assuming that Allan was referring to the work of constructing the
> current state of a ticket. I may have misunderstood of course. I was
> more expecting to go down the route of cached state when we want to
> start addressing the limitations of speed but this is not to say that
> making use of some kind of message queue could be in scope. I'm seeing
> this for the moment as an implementation detail.
>
> That said, using a message queue could be one effective way of bringing
> a separation between a core project and a ui project. In that model we
> could allow there to be a server for the ui that acts as an intermediary
> and provides a websockets implementation assuming that is required. A
> message queue could also be an effective way of implementing
> notifications for ticket change events and similar.
>
> I have played with zeromq a bit which is good fun... but I understand we
> can't use it due to its current choice of license. I mostly mention it
> as at the moment I sort of prefer the socket library approach rather
> than requiring a message broker. I wouldn't be sure that nanomsg is a
> viable alternative.
>
> > [...] I've had a lot of experience and success with Django
> > Channels (websockets).
> >
> > See https://channels.readthedocs.io
> > And my post here:
> > https://stackoverflow.com/questions/42815835/how-to-make-webpage-to-show-real-time-values-sent-from-raspberry-pi/42817627#42817627
> >
> > Websockets are supported natively by all modern browsers so it makes a
> > very portable solution with a low client overhead and with Channels, all
> > the benefits of Django.
>
> Good stuff. I am certainly happy with use of websockets for the main
> frontend project. Again, as I suggest above, websockets might not make
> it into the core project but I don't think that should be seen as a big
> deal. If a frontend project wants to make use of them then we can make
> that happen in some part of the architecture.
>
> One thing that this does bring up is the support for reduced browsers
> with reduced functionality. I can see the benefit of not having to
> consider older browsers but it would be nice if ticket management is
> possible in some form for older browsers. The lack of support for some
> ways of interacting might not be seen as all that problematic though.
>
> > j.
> >
> >
> > -----Original Message-----
> > From: Allan Swanepoel [mailto:allanice001@gmail.com]
> > Sent: 11 March 2018 21:36
> > To: dev@bloodhound.apache.org
> > Subject: Re: [Proposal] Core Bloodhound - basic concepts
> >
> > Possibly take a look at what has already been done using Django in this arena?
> > One project that jumps out with some Google foo is https://djacket.github.io/
> >
> > Also, if you want to separate (loosely couple) the front-end and the
> > back-end, possibly build the whole front-end with some js framework like
> > react?
> > All the functionality would be provided by the api, and all you need is
> > some Ajax calls.
> >
> > For the slow stuff, possibly make these more asynchronous and use some
> > form of scheduler (celery / ampq bus) to offload the work. Again, the
> > result is posted back via the api.
> >
> > Just my 2c's worth
> >
> > On 11 Mar 2018 23:21, "Gary" <ga...@physics.org> wrote:
> >
> > > Hi everyone,
> > >
> > > Sorry it has taken so long to get this written. With the understanding
> > > that we have expressed the desire to move away from Trac as the base
> > > for the project, I'd like to put forward the following as an initial
> > > proposal for the way we progress. You should not expect that this is
> > > high on details. I only intend to provide some guidance around the
> > > concepts that I would like to see used. I am hoping that this will be
> > > enough to get a reasonable idea of what the core model might look like.
> > >
> > > In broad terms I am proposing that we
> > > * take the opportunity to design the project from the ground up as
> > > opposed to attempting to build up the existing interfaces
> > > * build the project using the Django web framework
> > > * separate out a core sub-project with a well defined REST api for a
> > > frontend UI sub-project to build upon
> > >
> > > The rest of this proposal sets out to define parts of the core sub-project.
> > >
> > > The core should provide enough services to support the main part of a
> > > issue tracking UI. I am also expecting there to be further
> > > sub-projects to support additional optional functionality.
> > >
> > > The core sub-project will probably provide a basic UI to make progress
> > > ahead of work on the UI sub-project. I am not looking to define the UI
> > > sub-project any further in this proposal although it is likely that it
> > > will be highly dynamic, perhaps making use of websockets. The basic UI
> > > of the core project should remain well-maintained throughout the life
> > > of the project but at the moment my expectation would be that it is a
> > > no-thrills interface.
> > >
> > > Back to the core sub-project, I am looking for this to be built on the
> > > following concepts:
> > >
> > > #0 a change in nomenclature from product to tracker as the top level
> > > #1 uuids for pretty much everything
> > > #2 tickets as the accumulation of ticket change events
> > > #3 'labels' and 'label types' as a unifying concept for
> > > categorisation of tickets
> > >
> > > One of the early decisions made in handling of multiple projects in
> > > bloodhound was to call this feature multi-product. The reasoning for
> > > this was that the term 'project' is overloaded (if I recall, there may
> > > have been an excuse to add projects as a layer in bloodhound that
> > > could be used to group tickets across products.) I would like to see
> > > the basic functionality brought in to handle the ability to host
> > > multiple projects (in the top level organisation sense) but change the
> > > nomenclature to tracker to be as generic as possible. To be fair, the
> > > name of the feature may not matter so much as I would also like to see
> > > users given the ability to define their levels of organisation to a
> > > greater extent. More on this in the discussion of #3.
> > >
> > > Note that I will use the term tracker in the rest of this proposal
> > > where you might prefer to see product or project.
> > >
> > > The idea behind #1, uuids, is that it may be useful to be able to
> > > unambiguously identify instances of objects even across bloodhound
> > > instances. Pending decisions on how tickets are referenced, these
> > > uuids could be used as a reference that is unchanging as it moved
> > > between trackers.
> > >
> > > How tickets are stored as a whole is also worth tackling, which is
> > > what #2 is trying to broadly decide. Here I am suggesting that the
> > > state of a ticket can be built up from a query that collects all the
> > > change events and applies the deltas in order. This could prove to be
> > > a slow process so at some point we may want to look at keeping a
> > > record of the current state or a checkpoint but, given that ticket
> > > views are expected to show the history, these are details that are
> > > required anyway. I suggest that we can look at optimising for speed
> > > later. Knowing that we can update a ticket from deltas may be useful.
> > >
> > > Finally, I am suggesting in #3 that as much as possible we generalise
> > > ticket categorisation. Categorisation is a central concept to capture
> > > and, in trac we inherited categorisations such as statuses (open, in
> > > progress, etc), types (bug, enhancement, task, etc), milestones,
> > > versions, etc, and these had separate implementations. What I would
> > > like to see is that, while we might provide an initial set of ways to
> > > categorise tickets, we should allow users to add the categorisations
> > > that they require. I'm suggesting calling this generalisation 'labels'
> > > and 'label types' (though we can bikeshed that if we like!)
> > >
> > > There are a few ways to look at implementing labels as we could look
> > > to capture a set of common kinds of collections so that there are
> > > types of 'label types', as it were. What I mean by this is that you
> > > can have labels that are simply a tag, perhaps some types of labels
> > > should have descriptions, some might have dates associated with them.
> > > Beyond that we would need to know where a label can be allowed to
> > > coexist on a ticket with others of the same 'label type'. A lot of
> > > these aspects could influence how you might want to display them.
> > >
> > > At the moment I believe that we can probably get away with a common
> > > model across all label types and we can provide users with the tools
> > > to define what you can do with labels of a given label type via the
> > > properties of that label type.
> > >
> > > As you can see I have a lot to say about these labels. Another thing
> > > that could be useful would be to be able to have label types and
> > > labels that are available to a specific tracker, globally or even to a
> > > set of trackers. To deliver this, label types and labels need to be
> > > able to be associated with a variable number of trackers.
> > >
> > > We also need to be able to define workflows in terms of transitions
> > > between labels and potentially how they are affected by other labels.
> > > I think it is suffice to say that I would like to see an
> > > implementation where users are able to define workflow transitions
> > > with a great deal of flexibility in the label types that are involved.
> > >
> > > From the point of view of the transition from the trac model to the
> > > desired model, we can attempt to make use of Django's migrations from
> > > autogenerated models as brought to our attention by Daniel Brownridge.
> > > I do want to ensure that there is a viable migration path in place
> > > from the old bloodhound to the new.
> > >
> > > I think this is about as far as I want the proposal to go for now.
> > > Obviously there is going to be plenty more to decide.
> > >
> > > Please let me know your thoughts.
> > >
> > > Cheers,
> > > Gary
> > >
> >
>
> Cheers,
> Gary
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Gary Martin <ga...@physics.org>.
On Mon, 12 Mar 2018, at 12:10 PM, Jason Morgan wrote:
> Rather than Celery, [...]
I was assuming that Allan was referring to the work of constructing the current state of a ticket. I may have misunderstood of course. I was more expecting to go down the route of cached state when we want to start addressing the limitations of speed but this is not to say that making use of some kind of message queue could be in scope. I'm seeing this for the moment as an implementation detail.
That said, using a message queue could be one effective way of bringing a separation between a core project and a ui project. In that model we could allow there to be a server for the ui that acts as an intermediary and provides a websockets implementation assuming that is required. A message queue could also be an effective way of implementing notifications for ticket change events and similar.
I have played with zeromq a bit which is good fun... but I understand we can't use it due to its current choice of license. I mostly mention it as at the moment I sort of prefer the socket library approach rather than requiring a message broker. I wouldn't be sure that nanomsg is a viable alternative.
> [...] I've had a lot of experience and success with Django
> Channels (websockets).
>
> See https://channels.readthedocs.io
> And my post here:
> https://stackoverflow.com/questions/42815835/how-to-make-webpage-to-show-real-time-values-sent-from-raspberry-pi/42817627#42817627
>
> Websockets are supported natively by all modern browsers so it makes a
> very portable solution with a low client overhead and with Channels, all
> the benefits of Django.
Good stuff. I am certainly happy with use of websockets for the main frontend project. Again, as I suggest above, websockets might not make it into the core project but I don't think that should be seen as a big deal. If a frontend project wants to make use of them then we can make that happen in some part of the architecture.
One thing that this does bring up is the support for reduced browsers with reduced functionality. I can see the benefit of not having to consider older browsers but it would be nice if ticket management is possible in some form for older browsers. The lack of support for some ways of interacting might not be seen as all that problematic though.
> j.
>
>
> -----Original Message-----
> From: Allan Swanepoel [mailto:allanice001@gmail.com]
> Sent: 11 March 2018 21:36
> To: dev@bloodhound.apache.org
> Subject: Re: [Proposal] Core Bloodhound - basic concepts
>
> Possibly take a look at what has already been done using Django in this arena?
> One project that jumps out with some Google foo is https://djacket.github.io/
>
> Also, if you want to separate (loosely couple) the front-end and the
> back-end, possibly build the whole front-end with some js framework like
> react?
> All the functionality would be provided by the api, and all you need is
> some Ajax calls.
>
> For the slow stuff, possibly make these more asynchronous and use some
> form of scheduler (celery / ampq bus) to offload the work. Again, the
> result is posted back via the api.
>
> Just my 2c's worth
>
> On 11 Mar 2018 23:21, "Gary" <ga...@physics.org> wrote:
>
> > Hi everyone,
> >
> > Sorry it has taken so long to get this written. With the understanding
> > that we have expressed the desire to move away from Trac as the base
> > for the project, I'd like to put forward the following as an initial
> > proposal for the way we progress. You should not expect that this is
> > high on details. I only intend to provide some guidance around the
> > concepts that I would like to see used. I am hoping that this will be
> > enough to get a reasonable idea of what the core model might look like.
> >
> > In broad terms I am proposing that we
> > * take the opportunity to design the project from the ground up as
> > opposed to attempting to build up the existing interfaces
> > * build the project using the Django web framework
> > * separate out a core sub-project with a well defined REST api for a
> > frontend UI sub-project to build upon
> >
> > The rest of this proposal sets out to define parts of the core sub-project.
> >
> > The core should provide enough services to support the main part of a
> > issue tracking UI. I am also expecting there to be further
> > sub-projects to support additional optional functionality.
> >
> > The core sub-project will probably provide a basic UI to make progress
> > ahead of work on the UI sub-project. I am not looking to define the UI
> > sub-project any further in this proposal although it is likely that it
> > will be highly dynamic, perhaps making use of websockets. The basic UI
> > of the core project should remain well-maintained throughout the life
> > of the project but at the moment my expectation would be that it is a
> > no-thrills interface.
> >
> > Back to the core sub-project, I am looking for this to be built on the
> > following concepts:
> >
> > #0 a change in nomenclature from product to tracker as the top level
> > #1 uuids for pretty much everything
> > #2 tickets as the accumulation of ticket change events
> > #3 'labels' and 'label types' as a unifying concept for
> > categorisation of tickets
> >
> > One of the early decisions made in handling of multiple projects in
> > bloodhound was to call this feature multi-product. The reasoning for
> > this was that the term 'project' is overloaded (if I recall, there may
> > have been an excuse to add projects as a layer in bloodhound that
> > could be used to group tickets across products.) I would like to see
> > the basic functionality brought in to handle the ability to host
> > multiple projects (in the top level organisation sense) but change the
> > nomenclature to tracker to be as generic as possible. To be fair, the
> > name of the feature may not matter so much as I would also like to see
> > users given the ability to define their levels of organisation to a
> > greater extent. More on this in the discussion of #3.
> >
> > Note that I will use the term tracker in the rest of this proposal
> > where you might prefer to see product or project.
> >
> > The idea behind #1, uuids, is that it may be useful to be able to
> > unambiguously identify instances of objects even across bloodhound
> > instances. Pending decisions on how tickets are referenced, these
> > uuids could be used as a reference that is unchanging as it moved
> > between trackers.
> >
> > How tickets are stored as a whole is also worth tackling, which is
> > what #2 is trying to broadly decide. Here I am suggesting that the
> > state of a ticket can be built up from a query that collects all the
> > change events and applies the deltas in order. This could prove to be
> > a slow process so at some point we may want to look at keeping a
> > record of the current state or a checkpoint but, given that ticket
> > views are expected to show the history, these are details that are
> > required anyway. I suggest that we can look at optimising for speed
> > later. Knowing that we can update a ticket from deltas may be useful.
> >
> > Finally, I am suggesting in #3 that as much as possible we generalise
> > ticket categorisation. Categorisation is a central concept to capture
> > and, in trac we inherited categorisations such as statuses (open, in
> > progress, etc), types (bug, enhancement, task, etc), milestones,
> > versions, etc, and these had separate implementations. What I would
> > like to see is that, while we might provide an initial set of ways to
> > categorise tickets, we should allow users to add the categorisations
> > that they require. I'm suggesting calling this generalisation 'labels'
> > and 'label types' (though we can bikeshed that if we like!)
> >
> > There are a few ways to look at implementing labels as we could look
> > to capture a set of common kinds of collections so that there are
> > types of 'label types', as it were. What I mean by this is that you
> > can have labels that are simply a tag, perhaps some types of labels
> > should have descriptions, some might have dates associated with them.
> > Beyond that we would need to know where a label can be allowed to
> > coexist on a ticket with others of the same 'label type'. A lot of
> > these aspects could influence how you might want to display them.
> >
> > At the moment I believe that we can probably get away with a common
> > model across all label types and we can provide users with the tools
> > to define what you can do with labels of a given label type via the
> > properties of that label type.
> >
> > As you can see I have a lot to say about these labels. Another thing
> > that could be useful would be to be able to have label types and
> > labels that are available to a specific tracker, globally or even to a
> > set of trackers. To deliver this, label types and labels need to be
> > able to be associated with a variable number of trackers.
> >
> > We also need to be able to define workflows in terms of transitions
> > between labels and potentially how they are affected by other labels.
> > I think it is suffice to say that I would like to see an
> > implementation where users are able to define workflow transitions
> > with a great deal of flexibility in the label types that are involved.
> >
> > From the point of view of the transition from the trac model to the
> > desired model, we can attempt to make use of Django's migrations from
> > autogenerated models as brought to our attention by Daniel Brownridge.
> > I do want to ensure that there is a viable migration path in place
> > from the old bloodhound to the new.
> >
> > I think this is about as far as I want the proposal to go for now.
> > Obviously there is going to be plenty more to decide.
> >
> > Please let me know your thoughts.
> >
> > Cheers,
> > Gary
> >
>
Cheers,
Gary
RE: [Proposal] Core Bloodhound - basic concepts
Posted by Jason Morgan <ja...@cropwell.net>.
Rather than Celery, I've had a lot of experience and success with Django Channels (websockets).
See https://channels.readthedocs.io
And my post here: https://stackoverflow.com/questions/42815835/how-to-make-webpage-to-show-real-time-values-sent-from-raspberry-pi/42817627#42817627
Websockets are supported natively by all modern browsers so it makes a very portable solution with a low client overhead and with Channels, all the benefits of Django.
j.
-----Original Message-----
From: Allan Swanepoel [mailto:allanice001@gmail.com]
Sent: 11 March 2018 21:36
To: dev@bloodhound.apache.org
Subject: Re: [Proposal] Core Bloodhound - basic concepts
Possibly take a look at what has already been done using Django in this arena?
One project that jumps out with some Google foo is https://djacket.github.io/
Also, if you want to separate (loosely couple) the front-end and the back-end, possibly build the whole front-end with some js framework like react?
All the functionality would be provided by the api, and all you need is some Ajax calls.
For the slow stuff, possibly make these more asynchronous and use some form of scheduler (celery / ampq bus) to offload the work. Again, the result is posted back via the api.
Just my 2c's worth
On 11 Mar 2018 23:21, "Gary" <ga...@physics.org> wrote:
> Hi everyone,
>
> Sorry it has taken so long to get this written. With the understanding
> that we have expressed the desire to move away from Trac as the base
> for the project, I'd like to put forward the following as an initial
> proposal for the way we progress. You should not expect that this is
> high on details. I only intend to provide some guidance around the
> concepts that I would like to see used. I am hoping that this will be
> enough to get a reasonable idea of what the core model might look like.
>
> In broad terms I am proposing that we
> * take the opportunity to design the project from the ground up as
> opposed to attempting to build up the existing interfaces
> * build the project using the Django web framework
> * separate out a core sub-project with a well defined REST api for a
> frontend UI sub-project to build upon
>
> The rest of this proposal sets out to define parts of the core sub-project.
>
> The core should provide enough services to support the main part of a
> issue tracking UI. I am also expecting there to be further
> sub-projects to support additional optional functionality.
>
> The core sub-project will probably provide a basic UI to make progress
> ahead of work on the UI sub-project. I am not looking to define the UI
> sub-project any further in this proposal although it is likely that it
> will be highly dynamic, perhaps making use of websockets. The basic UI
> of the core project should remain well-maintained throughout the life
> of the project but at the moment my expectation would be that it is a
> no-thrills interface.
>
> Back to the core sub-project, I am looking for this to be built on the
> following concepts:
>
> #0 a change in nomenclature from product to tracker as the top level
> #1 uuids for pretty much everything
> #2 tickets as the accumulation of ticket change events
> #3 'labels' and 'label types' as a unifying concept for
> categorisation of tickets
>
> One of the early decisions made in handling of multiple projects in
> bloodhound was to call this feature multi-product. The reasoning for
> this was that the term 'project' is overloaded (if I recall, there may
> have been an excuse to add projects as a layer in bloodhound that
> could be used to group tickets across products.) I would like to see
> the basic functionality brought in to handle the ability to host
> multiple projects (in the top level organisation sense) but change the
> nomenclature to tracker to be as generic as possible. To be fair, the
> name of the feature may not matter so much as I would also like to see
> users given the ability to define their levels of organisation to a
> greater extent. More on this in the discussion of #3.
>
> Note that I will use the term tracker in the rest of this proposal
> where you might prefer to see product or project.
>
> The idea behind #1, uuids, is that it may be useful to be able to
> unambiguously identify instances of objects even across bloodhound
> instances. Pending decisions on how tickets are referenced, these
> uuids could be used as a reference that is unchanging as it moved
> between trackers.
>
> How tickets are stored as a whole is also worth tackling, which is
> what #2 is trying to broadly decide. Here I am suggesting that the
> state of a ticket can be built up from a query that collects all the
> change events and applies the deltas in order. This could prove to be
> a slow process so at some point we may want to look at keeping a
> record of the current state or a checkpoint but, given that ticket
> views are expected to show the history, these are details that are
> required anyway. I suggest that we can look at optimising for speed
> later. Knowing that we can update a ticket from deltas may be useful.
>
> Finally, I am suggesting in #3 that as much as possible we generalise
> ticket categorisation. Categorisation is a central concept to capture
> and, in trac we inherited categorisations such as statuses (open, in
> progress, etc), types (bug, enhancement, task, etc), milestones,
> versions, etc, and these had separate implementations. What I would
> like to see is that, while we might provide an initial set of ways to
> categorise tickets, we should allow users to add the categorisations
> that they require. I'm suggesting calling this generalisation 'labels'
> and 'label types' (though we can bikeshed that if we like!)
>
> There are a few ways to look at implementing labels as we could look
> to capture a set of common kinds of collections so that there are
> types of 'label types', as it were. What I mean by this is that you
> can have labels that are simply a tag, perhaps some types of labels
> should have descriptions, some might have dates associated with them.
> Beyond that we would need to know where a label can be allowed to
> coexist on a ticket with others of the same 'label type'. A lot of
> these aspects could influence how you might want to display them.
>
> At the moment I believe that we can probably get away with a common
> model across all label types and we can provide users with the tools
> to define what you can do with labels of a given label type via the
> properties of that label type.
>
> As you can see I have a lot to say about these labels. Another thing
> that could be useful would be to be able to have label types and
> labels that are available to a specific tracker, globally or even to a
> set of trackers. To deliver this, label types and labels need to be
> able to be associated with a variable number of trackers.
>
> We also need to be able to define workflows in terms of transitions
> between labels and potentially how they are affected by other labels.
> I think it is suffice to say that I would like to see an
> implementation where users are able to define workflow transitions
> with a great deal of flexibility in the label types that are involved.
>
> From the point of view of the transition from the trac model to the
> desired model, we can attempt to make use of Django's migrations from
> autogenerated models as brought to our attention by Daniel Brownridge.
> I do want to ensure that there is a viable migration path in place
> from the old bloodhound to the new.
>
> I think this is about as far as I want the proposal to go for now.
> Obviously there is going to be plenty more to decide.
>
> Please let me know your thoughts.
>
> Cheers,
> Gary
>
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Allan Swanepoel <al...@gmail.com>.
Possibly take a look at what has already been done using Django in this
arena?
One project that jumps out with some Google foo is
https://djacket.github.io/
Also, if you want to separate (loosely couple) the front-end and the
back-end, possibly build the whole front-end with some js framework like
react?
All the functionality would be provided by the api, and all you need is
some Ajax calls.
For the slow stuff, possibly make these more asynchronous and use some form
of scheduler (celery / ampq bus) to offload the work. Again, the result is
posted back via the api.
Just my 2c's worth
On 11 Mar 2018 23:21, "Gary" <ga...@physics.org> wrote:
> Hi everyone,
>
> Sorry it has taken so long to get this written. With the understanding
> that we have expressed the desire to move away from Trac as the base for
> the project, I'd like to put forward the following as an initial proposal
> for the way we progress. You should not expect that this is high on
> details. I only intend to provide some guidance around the concepts that I
> would like to see used. I am hoping that this will be enough to get a
> reasonable idea of what the core model might look like.
>
> In broad terms I am proposing that we
> * take the opportunity to design the project from the ground up as opposed
> to attempting to build up the existing interfaces
> * build the project using the Django web framework
> * separate out a core sub-project with a well defined REST api for a
> frontend UI sub-project to build upon
>
> The rest of this proposal sets out to define parts of the core sub-project.
>
> The core should provide enough services to support the main part of a
> issue tracking UI. I am also expecting there to be further sub-projects to
> support additional optional functionality.
>
> The core sub-project will probably provide a basic UI to make progress
> ahead of work on the UI sub-project. I am not looking to define the UI
> sub-project any further in this proposal although it is likely that it will
> be highly dynamic, perhaps making use of websockets. The basic UI of the
> core project should remain well-maintained throughout the life of the
> project but at the moment my expectation would be that it is a no-thrills
> interface.
>
> Back to the core sub-project, I am looking for this to be built on the
> following concepts:
>
> #0 a change in nomenclature from product to tracker as the top level
> #1 uuids for pretty much everything
> #2 tickets as the accumulation of ticket change events
> #3 'labels' and 'label types' as a unifying concept for categorisation of
> tickets
>
> One of the early decisions made in handling of multiple projects in
> bloodhound was to call this feature multi-product. The reasoning for this
> was that the term 'project' is overloaded (if I recall, there may have been
> an excuse to add projects as a layer in bloodhound that could be used to
> group tickets across products.) I would like to see the basic functionality
> brought in to handle the ability to host multiple projects (in the top
> level organisation sense) but change the nomenclature to tracker to be as
> generic as possible. To be fair, the name of the feature may not matter so
> much as I would also like to see users given the ability to define their
> levels of organisation to a greater extent. More on this in the discussion
> of #3.
>
> Note that I will use the term tracker in the rest of this proposal where
> you might prefer to see product or project.
>
> The idea behind #1, uuids, is that it may be useful to be able to
> unambiguously identify instances of objects even across bloodhound
> instances. Pending decisions on how tickets are referenced, these uuids
> could be used as a reference that is unchanging as it moved between
> trackers.
>
> How tickets are stored as a whole is also worth tackling, which is what #2
> is trying to broadly decide. Here I am suggesting that the state of a
> ticket can be built up from a query that collects all the change events and
> applies the deltas in order. This could prove to be a slow process so at
> some point we may want to look at keeping a record of the current state or
> a checkpoint but, given that ticket views are expected to show the history,
> these are details that are required anyway. I suggest that we can look at
> optimising for speed later. Knowing that we can update a ticket from deltas
> may be useful.
>
> Finally, I am suggesting in #3 that as much as possible we generalise
> ticket categorisation. Categorisation is a central concept to capture and,
> in trac we inherited categorisations such as statuses (open, in progress,
> etc), types (bug, enhancement, task, etc), milestones, versions, etc, and
> these had separate implementations. What I would like to see is that, while
> we might provide an initial set of ways to categorise tickets, we should
> allow users to add the categorisations that they require. I'm suggesting
> calling this generalisation 'labels' and 'label types' (though we can
> bikeshed that if we like!)
>
> There are a few ways to look at implementing labels as we could look to
> capture a set of common kinds of collections so that there are types of
> 'label types', as it were. What I mean by this is that you can have labels
> that are simply a tag, perhaps some types of labels should have
> descriptions, some might have dates associated with them. Beyond that we
> would need to know where a label can be allowed to coexist on a ticket with
> others of the same 'label type'. A lot of these aspects could influence how
> you might want to display them.
>
> At the moment I believe that we can probably get away with a common model
> across all label types and we can provide users with the tools to define
> what you can do with labels of a given label type via the properties of
> that label type.
>
> As you can see I have a lot to say about these labels. Another thing that
> could be useful would be to be able to have label types and labels that are
> available to a specific tracker, globally or even to a set of trackers. To
> deliver this, label types and labels need to be able to be associated with
> a variable number of trackers.
>
> We also need to be able to define workflows in terms of transitions
> between labels and potentially how they are affected by other labels. I
> think it is suffice to say that I would like to see an implementation where
> users are able to define workflow transitions with a great deal of
> flexibility in the label types that are involved.
>
> From the point of view of the transition from the trac model to the
> desired model, we can attempt to make use of Django's migrations from
> autogenerated models as brought to our attention by Daniel Brownridge. I do
> want to ensure that there is a viable migration path in place from the old
> bloodhound to the new.
>
> I think this is about as far as I want the proposal to go for now.
> Obviously there is going to be plenty more to decide.
>
> Please let me know your thoughts.
>
> Cheers,
> Gary
>
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Gary <ga...@physics.org>.
On Mon, 12 Mar 2018, at 2:00 AM, Greg Stein wrote:
> On Sun, Mar 11, 2018 at 4:21 PM, Gary <ga...@physics.org> wrote:
> >...
>
> > How tickets are stored as a whole is also worth tackling, which is what #2
> > is trying to broadly decide. Here I am suggesting that the state of a
> > ticket can be built up from a query that collects all the change events and
> > applies the deltas in order. This could prove to be a slow process so at
> > some point we may want to look at keeping a record of the current state or
> > a checkpoint but, given that ticket views are expected to show the history,
> > these are details that are required anyway. I suggest that we can look at
> > optimising for speed later. Knowing that we can update a ticket from deltas
> > may be useful.
> >
>
> The Apache Subversion team has a LOT of experience in this kind of storage
> :-)
>
> When we started, we stored the "latest" in full text, and then had a series
> of "reverse deltas" to go backwards in time. We switched that around, and
> store the original in full text, and then apply "forward" deltas to reach
> "latest".
>
> We have two mechanisms to speed this up: a sophisticated cache system.
> Invariably, "latest" will be cached and fast to retrieve. The more
> important part of our "series of delta" system, allowing us to rapidly
> construct any point in history is to use "skip deltas". This is analogous
> to the "skip list"[1] concept, where we assemble N deltas into a single
> delta allowing us to skip over many deltas with a single application.
> Generally speaking, if there are M deltas in a file's entire history, then
> we can reassemble any point in history by applying log(M) deltas.
>
> (note we used the skip delta mechanism for both directions; it is effective
> in both directions)
Well, it is good to confirm that this aspect of the proposal is on relatively solid footing. Given that retention of ticket history has to be there, I thought it would be better to go with a solution that avoided modifying existing records. Beyond that, you have translated what I was saying pretty well! I kind of see the cache of the latest state of a ticket as being something that does not require being kept constantly up to date given the ability to tell when it is behind. Anyway, these are implementation details that do not need to be set in stone yet.
Regardless, I will of course welcome contributions or suggestions from those with experience of this!
> Finally, I am suggesting in #3 that as much as possible we generalise
> > ticket categorisation. Categorisation is a central concept to capture and,
> > in trac we inherited categorisations such as statuses (open, in progress,
> > etc), types (bug, enhancement, task, etc), milestones, versions, etc, and
> > these had separate implementations.
>
>
> When my team designed the issue tracker for Google Code's project hosting,
> we did the same thing. Most issue trackers are highly-structured with a
> field or this and that. It complicates issue creation, issue management,
> querying, etc. I recall sitting down with 20 fields from a typical Bugzilla
> install, and reducing it to about 8, where we used labels for "everything".
> We did not bother with "issue types".
>
> A second thing that we did is allow labels such as "milestone-14", and then
> enable our "list display" to be configured for a "milestone" column that
> would extract any labels that started with "milestone-", showing just the
> "14".
>
> It made for a very robust, easy to understand, and flexible metadata system
> for each issue.
>
> You can still see some of these design choices a decade later in Monorail,
> a descendent of our tracker, now being used by the Chromium project.
I like this :)
The main objectives around labelling (classification, characterisation, whatever) as I described them was to have a flexible model where labels can be shared across trackers or be localised to a specific tracker and admins can setup label types that they feel are appropriate.
We could consider peeling this way back to basics so that you just label things similarly to how Greg describes. This can simplify core logic at the expense of being able to formalise behavioural differences between kinds of labels (like whether a ticket can support more than one of the same type of label at a time.)
We could look at allowing these constraints to be added on outside of core functionality or look at adding it into the core at a later point.
Given lightweight labels, whether a label is shared across trackers or is only available in a specific tracker may be less important.
Certainly food for thought. I would be up for starting from the simplified model and seeing how far that gets us in meeting the core ticketing needs.
>
> >...
>
> Cheers,
> -g
>
> [1] https://en.wikipedia.org/wiki/Skip_list
Thanks for the input!
Cheers,
Gary
Re: [Proposal] Core Bloodhound - basic concepts
Posted by Greg Stein <gs...@gmail.com>.
On Sun, Mar 11, 2018 at 4:21 PM, Gary <ga...@physics.org> wrote:
>...
> How tickets are stored as a whole is also worth tackling, which is what #2
> is trying to broadly decide. Here I am suggesting that the state of a
> ticket can be built up from a query that collects all the change events and
> applies the deltas in order. This could prove to be a slow process so at
> some point we may want to look at keeping a record of the current state or
> a checkpoint but, given that ticket views are expected to show the history,
> these are details that are required anyway. I suggest that we can look at
> optimising for speed later. Knowing that we can update a ticket from deltas
> may be useful.
>
The Apache Subversion team has a LOT of experience in this kind of storage
:-)
When we started, we stored the "latest" in full text, and then had a series
of "reverse deltas" to go backwards in time. We switched that around, and
store the original in full text, and then apply "forward" deltas to reach
"latest".
We have two mechanisms to speed this up: a sophisticated cache system.
Invariably, "latest" will be cached and fast to retrieve. The more
important part of our "series of delta" system, allowing us to rapidly
construct any point in history is to use "skip deltas". This is analogous
to the "skip list"[1] concept, where we assemble N deltas into a single
delta allowing us to skip over many deltas with a single application.
Generally speaking, if there are M deltas in a file's entire history, then
we can reassemble any point in history by applying log(M) deltas.
(note we used the skip delta mechanism for both directions; it is effective
in both directions)
Finally, I am suggesting in #3 that as much as possible we generalise
> ticket categorisation. Categorisation is a central concept to capture and,
> in trac we inherited categorisations such as statuses (open, in progress,
> etc), types (bug, enhancement, task, etc), milestones, versions, etc, and
> these had separate implementations.
When my team designed the issue tracker for Google Code's project hosting,
we did the same thing. Most issue trackers are highly-structured with a
field or this and that. It complicates issue creation, issue management,
querying, etc. I recall sitting down with 20 fields from a typical Bugzilla
install, and reducing it to about 8, where we used labels for "everything".
We did not bother with "issue types".
A second thing that we did is allow labels such as "milestone-14", and then
enable our "list display" to be configured for a "milestone" column that
would extract any labels that started with "milestone-", showing just the
"14".
It made for a very robust, easy to understand, and flexible metadata system
for each issue.
You can still see some of these design choices a decade later in Monorail,
a descendent of our tracker, now being used by the Chromium project.
>...
Cheers,
-g
[1] https://en.wikipedia.org/wiki/Skip_list