You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@superset.apache.org by Artem Moskvin <mo...@gmail.com> on 2017/09/24 09:31:54 UTC
Druid metrics are not automatically detected in Superset
Hi all. Do I always need to add Druid metrics manually? Aren't they
supposed to be detected automatically? I just had to update 20+ metrics
manually because they were not detected automatically. That was so tedious!
--
Respectfully,
Art Moskvin
Re: Druid metrics are not automatically detected in Superset
Posted by Maxime Beauchemin <ma...@gmail.com>.
On our side we have a python script that reads yaml configurations and
talks syncs the metadata db using the ORM.
I'll take another look at that PR, let's get it through.
Max
On Mon, Sep 25, 2017 at 1:13 PM, Fabian Menges <fm...@twitter.com.invalid>
wrote:
> Are you writing the metadata straight into SQL? Our plan is to go a similar
> route but using YAML as an intermediate format so we can have the
> datasource configuration changes (more importantly the postagg metrics) in
> source control.
>
> We are bout 80% there and its based on this change set.
> https://github.com/apache/incubator-superset/pull/2993
>
> I'm internally keeping this MR up to date and we will eventually try to go
> get it upstream again once we have internally ironed the details (its
> already in use but not automated yet).
>
> Fabian
>
>
>
> On Mon, Sep 25, 2017 at 3:35 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > At Airbnb we have an internal framework to load data into Druid, and we
> use
> > the metadata needed by the framework to maintain the Druid datasource
> > metadata in Superset (as opposed to running the auto-refresh).
> >
> > That means the auto-refresh logic was kind of orphaned by us so it's not
> as
> > great as it could be. Work/PRs in that area are welcomed of course.
> >
> > Max
> >
> > On Sun, Sep 24, 2017 at 9:53 PM, Artem Moskvin <mo...@gmail.com>
> > wrote:
> >
> > > Hi Fabian,
> > >
> > > Thanks for response. Glad to know that I'm not alone here :)
> > > Ping me if you need any help. I'm ready to spend some time solving this
> > > problem too.
> > >
> > > On Mon, Sep 25, 2017 at 2:26 AM Fabian Menges
> > <fmenges@twitter.com.invalid
> > > >
> > > wrote:
> > >
> > > > Hi Artem,
> > > >
> > > > We usually have the opposite problem. I spend a lot of time deleting
> > > > automatically generated druid metrics everytime I run the refresh...
> > > >
> > > > Anyways you have two options, either to you run it through the UI
> > > (Sources
> > > > => Refresh Druid Meta Data) or you use the command line...
> > > > 'superset refresh_druid'. If I remember correctly you can specify a
> > > > specific datasource to update if you use the command line.
> > > >
> > > > I will probably spend some time working on making this process better
> > in
> > > > the next few weeks. Our druid datasources tend to have > 50 columns
> > and a
> > > > lot of post aggregated metrics and its pretty painful right now to
> > > maintain
> > > > that.
> > > >
> > > > Hope this helps...
> > > >
> > > > Fabian
> > > >
> > > > On Sun, Sep 24, 2017 at 5:31 AM, Artem Moskvin <
> moscowart99@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi all. Do I always need to add Druid metrics manually? Aren't they
> > > >> supposed to be detected automatically? I just had to update 20+
> > metrics
> > > >> manually because they were not detected automatically. That was so
> > > >> tedious!
> > > >> --
> > > >> Respectfully,
> > > >> Art Moskvin
> > > >>
> > > >
> > > > --
> > > Respectfully,
> > > Art Moskvin
> > >
> >
>
Re: Druid metrics are not automatically detected in Superset
Posted by Fabian Menges <fm...@twitter.com.INVALID>.
Are you writing the metadata straight into SQL? Our plan is to go a similar
route but using YAML as an intermediate format so we can have the
datasource configuration changes (more importantly the postagg metrics) in
source control.
We are bout 80% there and its based on this change set.
https://github.com/apache/incubator-superset/pull/2993
I'm internally keeping this MR up to date and we will eventually try to go
get it upstream again once we have internally ironed the details (its
already in use but not automated yet).
Fabian
On Mon, Sep 25, 2017 at 3:35 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> At Airbnb we have an internal framework to load data into Druid, and we use
> the metadata needed by the framework to maintain the Druid datasource
> metadata in Superset (as opposed to running the auto-refresh).
>
> That means the auto-refresh logic was kind of orphaned by us so it's not as
> great as it could be. Work/PRs in that area are welcomed of course.
>
> Max
>
> On Sun, Sep 24, 2017 at 9:53 PM, Artem Moskvin <mo...@gmail.com>
> wrote:
>
> > Hi Fabian,
> >
> > Thanks for response. Glad to know that I'm not alone here :)
> > Ping me if you need any help. I'm ready to spend some time solving this
> > problem too.
> >
> > On Mon, Sep 25, 2017 at 2:26 AM Fabian Menges
> <fmenges@twitter.com.invalid
> > >
> > wrote:
> >
> > > Hi Artem,
> > >
> > > We usually have the opposite problem. I spend a lot of time deleting
> > > automatically generated druid metrics everytime I run the refresh...
> > >
> > > Anyways you have two options, either to you run it through the UI
> > (Sources
> > > => Refresh Druid Meta Data) or you use the command line...
> > > 'superset refresh_druid'. If I remember correctly you can specify a
> > > specific datasource to update if you use the command line.
> > >
> > > I will probably spend some time working on making this process better
> in
> > > the next few weeks. Our druid datasources tend to have > 50 columns
> and a
> > > lot of post aggregated metrics and its pretty painful right now to
> > maintain
> > > that.
> > >
> > > Hope this helps...
> > >
> > > Fabian
> > >
> > > On Sun, Sep 24, 2017 at 5:31 AM, Artem Moskvin <mo...@gmail.com>
> > > wrote:
> > >
> > >> Hi all. Do I always need to add Druid metrics manually? Aren't they
> > >> supposed to be detected automatically? I just had to update 20+
> metrics
> > >> manually because they were not detected automatically. That was so
> > >> tedious!
> > >> --
> > >> Respectfully,
> > >> Art Moskvin
> > >>
> > >
> > > --
> > Respectfully,
> > Art Moskvin
> >
>
Re: Druid metrics are not automatically detected in Superset
Posted by Maxime Beauchemin <ma...@gmail.com>.
At Airbnb we have an internal framework to load data into Druid, and we use
the metadata needed by the framework to maintain the Druid datasource
metadata in Superset (as opposed to running the auto-refresh).
That means the auto-refresh logic was kind of orphaned by us so it's not as
great as it could be. Work/PRs in that area are welcomed of course.
Max
On Sun, Sep 24, 2017 at 9:53 PM, Artem Moskvin <mo...@gmail.com>
wrote:
> Hi Fabian,
>
> Thanks for response. Glad to know that I'm not alone here :)
> Ping me if you need any help. I'm ready to spend some time solving this
> problem too.
>
> On Mon, Sep 25, 2017 at 2:26 AM Fabian Menges <fmenges@twitter.com.invalid
> >
> wrote:
>
> > Hi Artem,
> >
> > We usually have the opposite problem. I spend a lot of time deleting
> > automatically generated druid metrics everytime I run the refresh...
> >
> > Anyways you have two options, either to you run it through the UI
> (Sources
> > => Refresh Druid Meta Data) or you use the command line...
> > 'superset refresh_druid'. If I remember correctly you can specify a
> > specific datasource to update if you use the command line.
> >
> > I will probably spend some time working on making this process better in
> > the next few weeks. Our druid datasources tend to have > 50 columns and a
> > lot of post aggregated metrics and its pretty painful right now to
> maintain
> > that.
> >
> > Hope this helps...
> >
> > Fabian
> >
> > On Sun, Sep 24, 2017 at 5:31 AM, Artem Moskvin <mo...@gmail.com>
> > wrote:
> >
> >> Hi all. Do I always need to add Druid metrics manually? Aren't they
> >> supposed to be detected automatically? I just had to update 20+ metrics
> >> manually because they were not detected automatically. That was so
> >> tedious!
> >> --
> >> Respectfully,
> >> Art Moskvin
> >>
> >
> > --
> Respectfully,
> Art Moskvin
>
Re: Druid metrics are not automatically detected in Superset
Posted by Artem Moskvin <mo...@gmail.com>.
Hi Fabian,
Thanks for response. Glad to know that I'm not alone here :)
Ping me if you need any help. I'm ready to spend some time solving this
problem too.
On Mon, Sep 25, 2017 at 2:26 AM Fabian Menges <fm...@twitter.com.invalid>
wrote:
> Hi Artem,
>
> We usually have the opposite problem. I spend a lot of time deleting
> automatically generated druid metrics everytime I run the refresh...
>
> Anyways you have two options, either to you run it through the UI (Sources
> => Refresh Druid Meta Data) or you use the command line...
> 'superset refresh_druid'. If I remember correctly you can specify a
> specific datasource to update if you use the command line.
>
> I will probably spend some time working on making this process better in
> the next few weeks. Our druid datasources tend to have > 50 columns and a
> lot of post aggregated metrics and its pretty painful right now to maintain
> that.
>
> Hope this helps...
>
> Fabian
>
> On Sun, Sep 24, 2017 at 5:31 AM, Artem Moskvin <mo...@gmail.com>
> wrote:
>
>> Hi all. Do I always need to add Druid metrics manually? Aren't they
>> supposed to be detected automatically? I just had to update 20+ metrics
>> manually because they were not detected automatically. That was so
>> tedious!
>> --
>> Respectfully,
>> Art Moskvin
>>
>
> --
Respectfully,
Art Moskvin
Re: Druid metrics are not automatically detected in Superset
Posted by Fabian Menges <fm...@twitter.com.INVALID>.
Hi Artem,
We usually have the opposite problem. I spend a lot of time deleting
automatically generated druid metrics everytime I run the refresh...
Anyways you have two options, either to you run it through the UI (Sources
=> Refresh Druid Meta Data) or you use the command line...
'superset refresh_druid'. If I remember correctly you can specify a
specific datasource to update if you use the command line.
I will probably spend some time working on making this process better in
the next few weeks. Our druid datasources tend to have > 50 columns and a
lot of post aggregated metrics and its pretty painful right now to maintain
that.
Hope this helps...
Fabian
On Sun, Sep 24, 2017 at 5:31 AM, Artem Moskvin <mo...@gmail.com>
wrote:
> Hi all. Do I always need to add Druid metrics manually? Aren't they
> supposed to be detected automatically? I just had to update 20+ metrics
> manually because they were not detected automatically. That was so tedious!
> --
> Respectfully,
> Art Moskvin
>