You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by Chun Chang <cc...@maprtech.com> on 2015/09/24 23:18:41 UTC

How deep can Drill drill?

For JSON data format, what is the recommended/supported nesting levels
Drill can handle? I think we should establish a limit and make it into the
documentation.

I tried querying a json file with 1800 nesting levels. The query never came
back, even after three hours.

Thanks,
Chun

Re: How deep can Drill drill?

Posted by Chun Chang <cc...@maprtech.com>.

Hi Stefan,

Yeah, that many level doesn't make too much practical sense. It's just an
example how people may test it with.

Chun

{
  "level0_lab0_bool" : false,
  "level0_lab1_float" : 0.03461426496505737,
  "level0_lab2_int" : -1001861650,
  "level0_lab3_double" : 0.015721726429861027,
  "level0_lab4_string" : "61fee88e-195b-4fd3-8cfa-50952e4a66a7",
  "level0_lab5_long" : 7258421602631813798,
  "level0_lab6_date" : "2015-09-24",
  "level0_lab7_null" : null,
  "level0_lab8_byte" : 85,
  "level0_lab9_short" : 9369,
  "level0_lab10_time" : "01:15:54",
  "level0_lab11_timestamp" : null,
  "stairway_to_level1" : {
    "level1_lab12_bool" : false,
    "level1_lab13_float" : 0.5304490327835083,
    "level1_lab14_int" : 1520835057,
    "level1_lab15_double" : 0.9255150816426463,
    "level1_lab16_string" : "229f74c6-c20d-40d7-bf1c-c87e84f6200f",
    "level1_lab17_long" : -5283026701033140543,
    "level1_lab18_date" : "2015-09-24",
    "level1_lab19_null" : null,
    "level1_lab20_byte" : 85,
    "level1_lab21_short" : 12957,
    "level1_lab22_time" : "01:15:54",
    "level1_lab23_timestamp" : null,
    "stairway_to_level2" : {
      "level2_lab24_bool" : true,
      "level2_lab25_float" : 0.7355604767799377,
      "level2_lab26_int" : 1518663456,
      "level2_lab27_double" : 0.2988853136068288,
      "level2_lab28_string" : "034e3a3e-224c-489c-975b-68fe4833c45c",
      "level2_lab29_long" : -7191075049054166440,
      "level2_lab30_date" : "2015-09-24",
      "level2_lab31_null" : null,
      "level2_lab32_byte" : 62,
      "level2_lab33_short" : 26995,
      "level2_lab34_time" : "01:15:54",
      "level2_lab35_timestamp" : null,
      "stairway_to_level3" : {
...
...
           "stairway_to_level1800" : {
...
...
}}...}

On Thu, Sep 24, 2015 at 3:26 PM, Stefán Baxter <st...@activitystream.com>
wrote:

> Hi Chun,
>
> Can you please explain to me what you mean by nesting levels?
>
> Because the way I understand it, having 1800 nesting levels makes no sense
> :).
>
> Regards,
>  -Stefan
>
> On Thu, Sep 24, 2015 at 9:18 PM, Chun Chang <cc...@maprtech.com> wrote:
>
> > For JSON data format, what is the recommended/supported nesting levels
> > Drill can handle? I think we should establish a limit and make it into
> the
> > documentation.
> >
> > I tried querying a json file with 1800 nesting levels. The query never
> came
> > back, even after three hours.
> >
> > Thanks,
> > Chun
> >
>

Re: How deep can Drill drill?

Posted by rahul challapalli <ch...@gmail.com>.

I am not sure about the impact that nesting would cause, but if you have a
lot of fields (nested or not) in one record, then drill might have some
trouble in allocating memory for each field. If I just want to test how
deep drill can dig, I will create a json document which has a single field
at depth "n". Increase "n" until it breaks.

- Rahul

On Thu, Sep 24, 2015 at 3:26 PM, Stefán Baxter <st...@activitystream.com>
wrote:

> Hi Chun,
>
> Can you please explain to me what you mean by nesting levels?
>
> Because the way I understand it, having 1800 nesting levels makes no sense
> :).
>
> Regards,
>  -Stefan
>
> On Thu, Sep 24, 2015 at 9:18 PM, Chun Chang <cc...@maprtech.com> wrote:
>
> > For JSON data format, what is the recommended/supported nesting levels
> > Drill can handle? I think we should establish a limit and make it into
> the
> > documentation.
> >
> > I tried querying a json file with 1800 nesting levels. The query never
> came
> > back, even after three hours.
> >
> > Thanks,
> > Chun
> >
>

Re: How deep can Drill drill?

Posted by Stefán Baxter <st...@activitystream.com>.

Hi Chun,

Can you please explain to me what you mean by nesting levels?

Because the way I understand it, having 1800 nesting levels makes no sense
:).

Regards,
 -Stefan

On Thu, Sep 24, 2015 at 9:18 PM, Chun Chang <cc...@maprtech.com> wrote:

> For JSON data format, what is the recommended/supported nesting levels
> Drill can handle? I think we should establish a limit and make it into the
> documentation.
>
> I tried querying a json file with 1800 nesting levels. The query never came
> back, even after three hours.
>
> Thanks,
> Chun
>