You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@superset.apache.org by Fabian Menges <fm...@twitter.com.INVALID> on 2017/09/23 15:40:39 UTC
Annotations and Overlays
Hi everyone,
Just wanted to give everyone heads up, I'm working on adding Annotations
and Overlays to time series line charts in Superset.
I'll probably have something done by next week. This is how it will work:
There are 3 different types of annotations:
1. Events. Events are a list of dates with descriptions. They will be
displayed as vertical lines. Their description will show up in the tooltips
if you hover over them. They are fetched from a "table" slice, that
contains the data.
2. Timeseries. Timeseries are are added as an additional line on the
line chart. The name and the display properties can be configured. The data
is fetched from any timeseries slce.
3. Formulas. Formulas are added as an additional line to the chart. You
will be able to enter a mathematical expression, that will be evaluated on
the client side using "mathjs". You will be able to specify display
properties. You will be able to specify any mathematical formula e.g
- y = 500 (horizontal line)
- y = 0.2x (line with inclination of 0.2)
- y = x^2 (parabola)
Let me know if you have any concerns or can think of missing features.
Fabian
Re: Annotations and Overlays
Posted by Fabian Menges <fm...@twitter.com.INVALID>.
Bringing this conversation to the mailing list,
for reference: https://github.com/apache/incubator-superset/pull/3521 and
https://github.com/apache/incubator-superset/pull/3518
Maxime wrote the quoted text, I'll respond inline:
so I'm a little confused of the distinction between 'slice-defined' and
'data-based'. The only annotation type I implemented that is not pulling
data from a database using a superset "slice" as a means to query data is
"Formula". Formulas are evaluated client side.
> we may need to talk to sync up about annotations, we all really like the
> work that you've done and do want to support the stuff you added:
>
> - slice-defined formulas
>
>
> - slice-defined vertical lines
>
> The *vertical lines are data-based from slices *only** right now (since
vertical lines are not functions), I can obviously add another vertical
line annotation type that you can specify through the UI, but that should
be stored in your annotations table in
https://github.com/apache/incubator-superset/pull/3521/files#diff-42109fa9f588a4bcfbf015e5383a7f7d
(Superset-managed
Annotation Layers). I still believe that is the only legitimate use case
for that table since "annotation" data from any other source should be
directly read from that source and not copied to a table within the
superset DB first.
Obviously I can't prevent you from loading data into that table that
originates from somewhere else but I think it is a bad idea ;P
> - data-based annotations, using a specific slice-types with the right
> constraints and guarantees, reusable across slices
>
> I don't see a good reason for limiting the slice types much. Obviously we
should limit the slices that you can choose from in the UI to derivations
of line chart and table with time series. We will probably just end up with
a copy of the table and the line chart slice if we create new slice types
for that.
The format for the data coming from the server side is already standardized
since the client side needs to be able to handle it no matter what. I might
need to add logic to the "Annotation Layer" dialog so you can pick the time
column and pass down some parameters, but on the backend I have that
already working.
An example for not adding too many constrains would be: If you look at my
video Events <https://youtu.be/My6ScXyAxIY> the "LOL" annotations (end of
the video) don't contain any metrics, while "Lots of Crhistophers"
annotation does.
> Throw in:
>
> - slice-defined data points (X/Y) annotations using
> http://d3-annotation.susielu.com/
>
> Again I think these should be data-based from slices. If you want to
create such annotations through the UI they should be stored in the
annotations table you are proposing (Superset-managed Annotation Layers),
that table would require a Y value column then. The data-based
implementation I should be able to add in a few minutes.
On top of this PR, which I would describe as:
>
> - Superset-managed Annotation Layers, to be renamed "Time Annotation
> Layers" to clarify, these are highly re-usage across slices.
>
> All this should be composable.
> I'd like to break down 1-2-3 from 4 potentially, though if we can agree on
> the vision we could make your current PR evolve into 1-2-4 combo. That's a
> pretty massive feature set though.
Could you specify what 1-2-3 and 4 are ?
Thanks,
Fabian
On Tue, Sep 26, 2017 at 12:33 PM, Fabian Menges <fm...@twitter.com> wrote:
> Just to close the loop, I just created some videos for how my annotations
> work:
>
> Events: https://youtu.be/My6ScXyAxIY
> Time Series: https://youtu.be/EQm7Kk1D1Cw
> Formulas: https://youtu.be/vBSaUNxLE0c
>
> I'll probably need another few days to clean up and expose a couple more
> features in the UI.
> I think it does make sense for me to integrate your annotations
> implementation, e.g. as an additional type.
>
> Fabian
>
>
>
> On Sat, Sep 23, 2017 at 2:15 PM, Fabian Menges <fm...@twitter.com>
> wrote:
>
>> Ok, I think we are solving two different but related problems.
>>
>> From your DB data model it looks like you will give users the ability to
>> add annotations through the UI.
>>
>> My work enables users to overlay data from other datasources, like
>> annotations, onto line charts. I think both of these are complementary.
>>
>> I'm happy to take on the UI work for both of these use cases if you are
>> fine with that.
>> Let me know how you envisioned the user experience for adding annotations
>> and I'm happy to give it shot.
>>
>> Fabian
>>
>> On Sat, Sep 23, 2017 at 1:39 PM, Fabian Menges <fm...@twitter.com>
>> wrote:
>>
>>> Nice, I'm almost done with my implementation and it looks like our stuff
>>> is compatible and we haven't done much duplicate work....
>>>
>>> I didn't really do anything server side, check out my code that add a
>>> annotation layer to the d3.svg...
>>>
>>> https://github.com/apache/incubator-superset/pull/3518/files
>>>
>>> Let me know what you think.
>>>
>>> On Sat, Sep 23, 2017 at 12:38 PM, Maxime Beauchemin <
>>> maximebeauchemin@gmail.com> wrote:
>>>
>>>> Wait up! Grace and I have been working on annotations last week.
>>>>
>>>> Here's the backend part I wrote, Grace has code that shows one or many
>>>> "annotation layers" on the nvd3 time series charts
>>>>
>>>> https://github.com/apache/incubator-superset/compare/master.
>>>> ..mistercrunch:annotations?expand=1
>>>>
>>>> Those are time-related annotations, namespaced into arbitrary "layers"
>>>> stored in the backend. We have plans to support other types of
>>>> annotation
>>>> like "point in place" in the future. We're open to collaborate on all
>>>> that.
>>>>
>>>> I like the idea of formulas, curious how you're planning on safely
>>>> "eval-ing" the formulas too, I'm thinking users should also be able to
>>>> pick
>>>> their color and style for that special series.
>>>>
>>>> Max
>>>>
>>>> On Sep 23, 2017 8:41 AM, "Fabian Menges" <fm...@twitter.com.invalid>
>>>> wrote:
>>>>
>>>> > Hi everyone,
>>>> >
>>>> > Just wanted to give everyone heads up, I'm working on adding
>>>> Annotations
>>>> > and Overlays to time series line charts in Superset.
>>>> >
>>>> > I'll probably have something done by next week. This is how it will
>>>> work:
>>>> >
>>>> > There are 3 different types of annotations:
>>>> >
>>>> > 1. Events. Events are a list of dates with descriptions. They will
>>>> be
>>>> > displayed as vertical lines. Their description will show up in the
>>>> > tooltips
>>>> > if you hover over them. They are fetched from a "table" slice, that
>>>> > contains the data.
>>>> > 2. Timeseries. Timeseries are are added as an additional line on
>>>> the
>>>> > line chart. The name and the display properties can be configured.
>>>> The
>>>> > data
>>>> > is fetched from any timeseries slce.
>>>> > 3. Formulas. Formulas are added as an additional line to the
>>>> chart. You
>>>> > will be able to enter a mathematical expression, that will be
>>>> evaluated
>>>> > on
>>>> > the client side using "mathjs". You will be able to specify display
>>>> > properties. You will be able to specify any mathematical formula
>>>> e.g
>>>> > - y = 500 (horizontal line)
>>>> > - y = 0.2x (line with inclination of 0.2)
>>>> > - y = x^2 (parabola)
>>>> >
>>>> > Let me know if you have any concerns or can think of missing features.
>>>> >
>>>> >
>>>> > Fabian
>>>> >
>>>>
>>>
>>>
>>
>
Re: Annotations and Overlays
Posted by Fabian Menges <fm...@twitter.com.INVALID>.
Just to close the loop, I just created some videos for how my annotations
work:
Events: https://youtu.be/My6ScXyAxIY
Time Series: https://youtu.be/EQm7Kk1D1Cw
Formulas: https://youtu.be/vBSaUNxLE0c
I'll probably need another few days to clean up and expose a couple more
features in the UI.
I think it does make sense for me to integrate your annotations
implementation, e.g. as an additional type.
Fabian
On Sat, Sep 23, 2017 at 2:15 PM, Fabian Menges <fm...@twitter.com> wrote:
> Ok, I think we are solving two different but related problems.
>
> From your DB data model it looks like you will give users the ability to
> add annotations through the UI.
>
> My work enables users to overlay data from other datasources, like
> annotations, onto line charts. I think both of these are complementary.
>
> I'm happy to take on the UI work for both of these use cases if you are
> fine with that.
> Let me know how you envisioned the user experience for adding annotations
> and I'm happy to give it shot.
>
> Fabian
>
> On Sat, Sep 23, 2017 at 1:39 PM, Fabian Menges <fm...@twitter.com>
> wrote:
>
>> Nice, I'm almost done with my implementation and it looks like our stuff
>> is compatible and we haven't done much duplicate work....
>>
>> I didn't really do anything server side, check out my code that add a
>> annotation layer to the d3.svg...
>>
>> https://github.com/apache/incubator-superset/pull/3518/files
>>
>> Let me know what you think.
>>
>> On Sat, Sep 23, 2017 at 12:38 PM, Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>>> Wait up! Grace and I have been working on annotations last week.
>>>
>>> Here's the backend part I wrote, Grace has code that shows one or many
>>> "annotation layers" on the nvd3 time series charts
>>>
>>> https://github.com/apache/incubator-superset/compare/master.
>>> ..mistercrunch:annotations?expand=1
>>>
>>> Those are time-related annotations, namespaced into arbitrary "layers"
>>> stored in the backend. We have plans to support other types of annotation
>>> like "point in place" in the future. We're open to collaborate on all
>>> that.
>>>
>>> I like the idea of formulas, curious how you're planning on safely
>>> "eval-ing" the formulas too, I'm thinking users should also be able to
>>> pick
>>> their color and style for that special series.
>>>
>>> Max
>>>
>>> On Sep 23, 2017 8:41 AM, "Fabian Menges" <fm...@twitter.com.invalid>
>>> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > Just wanted to give everyone heads up, I'm working on adding
>>> Annotations
>>> > and Overlays to time series line charts in Superset.
>>> >
>>> > I'll probably have something done by next week. This is how it will
>>> work:
>>> >
>>> > There are 3 different types of annotations:
>>> >
>>> > 1. Events. Events are a list of dates with descriptions. They will
>>> be
>>> > displayed as vertical lines. Their description will show up in the
>>> > tooltips
>>> > if you hover over them. They are fetched from a "table" slice, that
>>> > contains the data.
>>> > 2. Timeseries. Timeseries are are added as an additional line on the
>>> > line chart. The name and the display properties can be configured.
>>> The
>>> > data
>>> > is fetched from any timeseries slce.
>>> > 3. Formulas. Formulas are added as an additional line to the chart.
>>> You
>>> > will be able to enter a mathematical expression, that will be
>>> evaluated
>>> > on
>>> > the client side using "mathjs". You will be able to specify display
>>> > properties. You will be able to specify any mathematical formula e.g
>>> > - y = 500 (horizontal line)
>>> > - y = 0.2x (line with inclination of 0.2)
>>> > - y = x^2 (parabola)
>>> >
>>> > Let me know if you have any concerns or can think of missing features.
>>> >
>>> >
>>> > Fabian
>>> >
>>>
>>
>>
>
Re: Annotations and Overlays
Posted by Fabian Menges <fm...@twitter.com.INVALID>.
Ok, I think we are solving two different but related problems.
From your DB data model it looks like you will give users the ability to
add annotations through the UI.
My work enables users to overlay data from other datasources, like
annotations, onto line charts. I think both of these are complementary.
I'm happy to take on the UI work for both of these use cases if you are
fine with that.
Let me know how you envisioned the user experience for adding annotations
and I'm happy to give it shot.
Fabian
On Sat, Sep 23, 2017 at 1:39 PM, Fabian Menges <fm...@twitter.com> wrote:
> Nice, I'm almost done with my implementation and it looks like our stuff
> is compatible and we haven't done much duplicate work....
>
> I didn't really do anything server side, check out my code that add a
> annotation layer to the d3.svg...
>
> https://github.com/apache/incubator-superset/pull/3518/files
>
> Let me know what you think.
>
> On Sat, Sep 23, 2017 at 12:38 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
>> Wait up! Grace and I have been working on annotations last week.
>>
>> Here's the backend part I wrote, Grace has code that shows one or many
>> "annotation layers" on the nvd3 time series charts
>>
>> https://github.com/apache/incubator-superset/compare/master.
>> ..mistercrunch:annotations?expand=1
>>
>> Those are time-related annotations, namespaced into arbitrary "layers"
>> stored in the backend. We have plans to support other types of annotation
>> like "point in place" in the future. We're open to collaborate on all
>> that.
>>
>> I like the idea of formulas, curious how you're planning on safely
>> "eval-ing" the formulas too, I'm thinking users should also be able to
>> pick
>> their color and style for that special series.
>>
>> Max
>>
>> On Sep 23, 2017 8:41 AM, "Fabian Menges" <fm...@twitter.com.invalid>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > Just wanted to give everyone heads up, I'm working on adding Annotations
>> > and Overlays to time series line charts in Superset.
>> >
>> > I'll probably have something done by next week. This is how it will
>> work:
>> >
>> > There are 3 different types of annotations:
>> >
>> > 1. Events. Events are a list of dates with descriptions. They will be
>> > displayed as vertical lines. Their description will show up in the
>> > tooltips
>> > if you hover over them. They are fetched from a "table" slice, that
>> > contains the data.
>> > 2. Timeseries. Timeseries are are added as an additional line on the
>> > line chart. The name and the display properties can be configured.
>> The
>> > data
>> > is fetched from any timeseries slce.
>> > 3. Formulas. Formulas are added as an additional line to the chart.
>> You
>> > will be able to enter a mathematical expression, that will be
>> evaluated
>> > on
>> > the client side using "mathjs". You will be able to specify display
>> > properties. You will be able to specify any mathematical formula e.g
>> > - y = 500 (horizontal line)
>> > - y = 0.2x (line with inclination of 0.2)
>> > - y = x^2 (parabola)
>> >
>> > Let me know if you have any concerns or can think of missing features.
>> >
>> >
>> > Fabian
>> >
>>
>
>
Re: Annotations and Overlays
Posted by Fabian Menges <fm...@twitter.com.INVALID>.
Nice, I'm almost done with my implementation and it looks like our stuff is
compatible and we haven't done much duplicate work....
I didn't really do anything server side, check out my code that add a
annotation layer to the d3.svg...
https://github.com/apache/incubator-superset/pull/3518/files
Let me know what you think.
On Sat, Sep 23, 2017 at 12:38 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> Wait up! Grace and I have been working on annotations last week.
>
> Here's the backend part I wrote, Grace has code that shows one or many
> "annotation layers" on the nvd3 time series charts
>
> https://github.com/apache/incubator-superset/compare/
> master...mistercrunch:annotations?expand=1
>
> Those are time-related annotations, namespaced into arbitrary "layers"
> stored in the backend. We have plans to support other types of annotation
> like "point in place" in the future. We're open to collaborate on all that.
>
> I like the idea of formulas, curious how you're planning on safely
> "eval-ing" the formulas too, I'm thinking users should also be able to pick
> their color and style for that special series.
>
> Max
>
> On Sep 23, 2017 8:41 AM, "Fabian Menges" <fm...@twitter.com.invalid>
> wrote:
>
> > Hi everyone,
> >
> > Just wanted to give everyone heads up, I'm working on adding Annotations
> > and Overlays to time series line charts in Superset.
> >
> > I'll probably have something done by next week. This is how it will work:
> >
> > There are 3 different types of annotations:
> >
> > 1. Events. Events are a list of dates with descriptions. They will be
> > displayed as vertical lines. Their description will show up in the
> > tooltips
> > if you hover over them. They are fetched from a "table" slice, that
> > contains the data.
> > 2. Timeseries. Timeseries are are added as an additional line on the
> > line chart. The name and the display properties can be configured. The
> > data
> > is fetched from any timeseries slce.
> > 3. Formulas. Formulas are added as an additional line to the chart.
> You
> > will be able to enter a mathematical expression, that will be
> evaluated
> > on
> > the client side using "mathjs". You will be able to specify display
> > properties. You will be able to specify any mathematical formula e.g
> > - y = 500 (horizontal line)
> > - y = 0.2x (line with inclination of 0.2)
> > - y = x^2 (parabola)
> >
> > Let me know if you have any concerns or can think of missing features.
> >
> >
> > Fabian
> >
>
Re: Annotations and Overlays
Posted by Maxime Beauchemin <ma...@gmail.com>.
Wait up! Grace and I have been working on annotations last week.
Here's the backend part I wrote, Grace has code that shows one or many
"annotation layers" on the nvd3 time series charts
https://github.com/apache/incubator-superset/compare/master...mistercrunch:annotations?expand=1
Those are time-related annotations, namespaced into arbitrary "layers"
stored in the backend. We have plans to support other types of annotation
like "point in place" in the future. We're open to collaborate on all that.
I like the idea of formulas, curious how you're planning on safely
"eval-ing" the formulas too, I'm thinking users should also be able to pick
their color and style for that special series.
Max
On Sep 23, 2017 8:41 AM, "Fabian Menges" <fm...@twitter.com.invalid>
wrote:
> Hi everyone,
>
> Just wanted to give everyone heads up, I'm working on adding Annotations
> and Overlays to time series line charts in Superset.
>
> I'll probably have something done by next week. This is how it will work:
>
> There are 3 different types of annotations:
>
> 1. Events. Events are a list of dates with descriptions. They will be
> displayed as vertical lines. Their description will show up in the
> tooltips
> if you hover over them. They are fetched from a "table" slice, that
> contains the data.
> 2. Timeseries. Timeseries are are added as an additional line on the
> line chart. The name and the display properties can be configured. The
> data
> is fetched from any timeseries slce.
> 3. Formulas. Formulas are added as an additional line to the chart. You
> will be able to enter a mathematical expression, that will be evaluated
> on
> the client side using "mathjs". You will be able to specify display
> properties. You will be able to specify any mathematical formula e.g
> - y = 500 (horizontal line)
> - y = 0.2x (line with inclination of 0.2)
> - y = x^2 (parabola)
>
> Let me know if you have any concerns or can think of missing features.
>
>
> Fabian
>