You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Sudip Mukherjee <sm...@commvault.com> on 2015/09/07 13:49:36 UTC
Drill Sql Max row size
Hi Devs,
Is there a max row limit which I can configure while pulling data from underlying datasource? If there is a large data-set would drill fetch like page by page?
Thanks,
Sudip
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
Re: Drill Sql Max row size
Posted by Jacques Nadeau <ja...@dremio.com>.
Yes, a reader works a batch at a time. Typically, we use 4095 as the
default number of records to produce. I believe the Elastic approach is
based on using cursormark.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Tue, Sep 8, 2015 at 7:53 AM, Sudip Mukherjee <sm...@commvault.com>
wrote:
> Yes, there is no problem in that . Was just wondering if there is a
> mechanism of getting data by batches from the record reader.
> Actually , I wrote a basic plugin to solr and was thinking if I paginate
> or use cursormark params in the recordreader.
>
> Thanks,
> Sudip
>
> -----Original Message-----
> From: Tomer Shiran [mailto:tshiran@dremio.com]
> Sent: 08 September 2015 PM 08:16
> To: dev@drill.apache.org
> Subject: Re: Drill Sql Max row size
>
> That won't be a problem. There's actually no limit on how many
> records/rows you can have.
>
> > On Sep 8, 2015, at 2:00 AM, Sudip Mukherjee <sm...@commvault.com>
> wrote:
> >
> > Hi,
> > I have somewhere around a million records. But columns are less in
> > numbers (up to 10)
> >
> > Thanks,
> > Sudip
> >
> > -----Original Message-----
> > From: Jacques Nadeau [mailto:jacques@dremio.com]
> > Sent: 08 September 2015 AM 06:58
> > To: dev@drill.apache.org
> > Subject: Re: Drill Sql Max row size
> >
> > Generally, no. That being said, Drill will probably struggle if you
> start reading records where one or more cells is greater than a few hundred
> kilobytes (or mbs) or more than several hundred columns/fields. What size
> records are you working with?
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee
> > <sm...@commvault.com>
> > wrote:
> >
> >> Hi Devs,
> >>
> >> Is there a max row limit which I can configure while pulling data
> >> from underlying datasource? If there is a large data-set would drill
> >> fetch like page by page?
> >>
> >> Thanks,
> >> Sudip
> >>
> >>
> >>
> >> ***************************Legal
> >> Disclaimer***************************
> >> "This communication may contain confidential and privileged material
> >> for the sole use of the intended recipient. Any unauthorized review,
> >> use or distribution by others is strictly prohibited. If you have
> >> received the message by mistake, please advise the sender by reply
> >> email and delete the message. Thank you."
> >> *********************************************************************
> >> *
> >
> >
> >
> > ***************************Legal Disclaimer***************************
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply email
> and delete the message. Thank you."
> > **********************************************************************
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
>
RE: Drill Sql Max row size
Posted by Sudip Mukherjee <sm...@commvault.com>.
Yes, there is no problem in that . Was just wondering if there is a mechanism of getting data by batches from the record reader.
Actually , I wrote a basic plugin to solr and was thinking if I paginate or use cursormark params in the recordreader.
Thanks,
Sudip
-----Original Message-----
From: Tomer Shiran [mailto:tshiran@dremio.com]
Sent: 08 September 2015 PM 08:16
To: dev@drill.apache.org
Subject: Re: Drill Sql Max row size
That won't be a problem. There's actually no limit on how many records/rows you can have.
> On Sep 8, 2015, at 2:00 AM, Sudip Mukherjee <sm...@commvault.com> wrote:
>
> Hi,
> I have somewhere around a million records. But columns are less in
> numbers (up to 10)
>
> Thanks,
> Sudip
>
> -----Original Message-----
> From: Jacques Nadeau [mailto:jacques@dremio.com]
> Sent: 08 September 2015 AM 06:58
> To: dev@drill.apache.org
> Subject: Re: Drill Sql Max row size
>
> Generally, no. That being said, Drill will probably struggle if you start reading records where one or more cells is greater than a few hundred kilobytes (or mbs) or more than several hundred columns/fields. What size records are you working with?
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee
> <sm...@commvault.com>
> wrote:
>
>> Hi Devs,
>>
>> Is there a max row limit which I can configure while pulling data
>> from underlying datasource? If there is a large data-set would drill
>> fetch like page by page?
>>
>> Thanks,
>> Sudip
>>
>>
>>
>> ***************************Legal
>> Disclaimer***************************
>> "This communication may contain confidential and privileged material
>> for the sole use of the intended recipient. Any unauthorized review,
>> use or distribution by others is strictly prohibited. If you have
>> received the message by mistake, please advise the sender by reply
>> email and delete the message. Thank you."
>> *********************************************************************
>> *
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message by mistake, please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
Re: Drill Sql Max row size
Posted by Tomer Shiran <ts...@dremio.com>.
That won't be a problem. There's actually no limit on how many records/rows you can have.
> On Sep 8, 2015, at 2:00 AM, Sudip Mukherjee <sm...@commvault.com> wrote:
>
> Hi,
> I have somewhere around a million records. But columns are less in numbers (up to 10)
>
> Thanks,
> Sudip
>
> -----Original Message-----
> From: Jacques Nadeau [mailto:jacques@dremio.com]
> Sent: 08 September 2015 AM 06:58
> To: dev@drill.apache.org
> Subject: Re: Drill Sql Max row size
>
> Generally, no. That being said, Drill will probably struggle if you start reading records where one or more cells is greater than a few hundred kilobytes (or mbs) or more than several hundred columns/fields. What size records are you working with?
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee <sm...@commvault.com>
> wrote:
>
>> Hi Devs,
>>
>> Is there a max row limit which I can configure while pulling data from
>> underlying datasource? If there is a large data-set would drill fetch
>> like page by page?
>>
>> Thanks,
>> Sudip
>>
>>
>>
>> ***************************Legal Disclaimer***************************
>> "This communication may contain confidential and privileged material
>> for the sole use of the intended recipient. Any unauthorized review,
>> use or distribution by others is strictly prohibited. If you have
>> received the message by mistake, please advise the sender by reply
>> email and delete the message. Thank you."
>> **********************************************************************
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for the
> sole use of the intended recipient. Any unauthorized review, use or distribution
> by others is strictly prohibited. If you have received the message by mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
RE: Drill Sql Max row size
Posted by Sudip Mukherjee <sm...@commvault.com>.
Hi,
I have somewhere around a million records. But columns are less in numbers (up to 10)
Thanks,
Sudip
-----Original Message-----
From: Jacques Nadeau [mailto:jacques@dremio.com]
Sent: 08 September 2015 AM 06:58
To: dev@drill.apache.org
Subject: Re: Drill Sql Max row size
Generally, no. That being said, Drill will probably struggle if you start reading records where one or more cells is greater than a few hundred kilobytes (or mbs) or more than several hundred columns/fields. What size records are you working with?
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee <sm...@commvault.com>
wrote:
> Hi Devs,
>
> Is there a max row limit which I can configure while pulling data from
> underlying datasource? If there is a large data-set would drill fetch
> like page by page?
>
> Thanks,
> Sudip
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material
> for the sole use of the intended recipient. Any unauthorized review,
> use or distribution by others is strictly prohibited. If you have
> received the message by mistake, please advise the sender by reply
> email and delete the message. Thank you."
> **********************************************************************
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************
Re: Drill Sql Max row size
Posted by Jacques Nadeau <ja...@dremio.com>.
Generally, no. That being said, Drill will probably struggle if you start
reading records where one or more cells is greater than a few hundred
kilobytes (or mbs) or more than several hundred columns/fields. What size
records are you working with?
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee <sm...@commvault.com>
wrote:
> Hi Devs,
>
> Is there a max row limit which I can configure while pulling data from
> underlying datasource? If there is a large data-set would drill fetch like
> page by page?
>
> Thanks,
> Sudip
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************