You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by anil gupta <an...@gmail.com> on 2014/05/04 08:19:58 UTC
Phoenix creates ResultSpoolerxxxxxx.bin files in /tmp
Hi,
I would like to know why phoenix creates "ResultSpoolerxxxxxxxx.bin" file?
What is the content of these?
I was using sqlline with phoenix3.0 and ran a "select * from table" query.
That table had around 10 million rows. Is phoenix really trying to read
entire table?
Eventually this exception was thrown by phoenix:
java.lang.Runtime Exception:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.phoenix.exception.PhoenixIOException: No space left on device
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
at sqlline.SqlLine.print(SqlLine.java:1735)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
at sqlline.SqlLine.dispatch(SqlLine.java:821)
at sqlline.SqlLine.begin(SqlLine.java:699)
at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
at sqlline.SqlLine.main(SqlLine.java:424)
--
Thanks & Regards,
Anil Gupta
Re: Phoenix creates ResultSpoolerxxxxxx.bin files in /tmp
Posted by James Taylor <ja...@apache.org>.
Your table is spooled to disk once a memory threshold is exceeded. The data
will be spooled up to a configurable max and then the query would fail. We
have a JIRA to instead pull the data over from the server as needed and
it'd be great to have a volunteer to implement it:
https://issues.apache.org/jira/browse/PHOENIX-539
Thanks,
James
On Tue, May 6, 2014 at 8:53 PM, anil gupta <an...@gmail.com> wrote:
> Thanks for the response.
> Why do we need to store this data on disk? In some environments this is
> undesirable because of security constraints/reasons.
> If we do "select * from test_table;" then i am assuming that we are not
> trying to pull the entire table at client side. We will just show first
> 50-100 rows(1 page) and keep on serving subsequent pages on next page
> requests.
> Note: I was using sqlline to do that query. In Squirell, i think by default
> for every query limit is set to 100.
>
>
> On Tue, May 6, 2014 at 12:10 PM, James Taylor <jamestaylor@apache.org
> >wrote:
>
> > Yes, and just to add to that, those files should be deleted when the scan
> > is closed. If that's not the case, please file a JIRA.
> > Thanks,
> > James
> >
> >
> > On Tue, May 6, 2014 at 12:08 AM, Gabriel Reid <gabriel.reid@gmail.com
> > >wrote:
> >
> > > Yes, Phoenix writes the results of scanners to an in-memory store that
> > > spools into file storage once it gets too big (see [1]).
> > >
> > > 1.
> > >
> >
> https://github.com/apache/incubator-phoenix/blob/c22539d940fa0afc790cc0f25c2cdbbb255f80a3/phoenix-core/src/main/java/org/apache/phoenix/iterate/SpoolingResultIterator.java#L95
> > >
> > > On Sun, May 4, 2014 at 8:19 AM, anil gupta <an...@gmail.com>
> > wrote:
> > > > Hi,
> > > >
> > > > I would like to know why phoenix creates "ResultSpoolerxxxxxxxx.bin"
> > > file?
> > > > What is the content of these?
> > > >
> > > > I was using sqlline with phoenix3.0 and ran a "select * from table"
> > > query.
> > > > That table had around 10 million rows. Is phoenix really trying to
> read
> > > > entire table?
> > > >
> > > > Eventually this exception was thrown by phoenix:
> > > > java.lang.Runtime Exception:
> > > > org.apache.phoenix.exception.PhoenixIOException:
> > > > org.apache.phoenix.exception.PhoenixIOException: No space left on
> > device
> > > > at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
> > > > at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
> > > > at sqlline.SqlLine.print(SqlLine.java:1735)
> > > > at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
> > > > at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
> > > > at sqlline.SqlLine.dispatch(SqlLine.java:821)
> > > > at sqlline.SqlLine.begin(SqlLine.java:699)
> > > > at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
> > > > at sqlline.SqlLine.main(SqlLine.java:424)
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > >
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Phoenix creates ResultSpoolerxxxxxx.bin files in /tmp
Posted by anil gupta <an...@gmail.com>.
Thanks for the response.
Why do we need to store this data on disk? In some environments this is
undesirable because of security constraints/reasons.
If we do "select * from test_table;" then i am assuming that we are not
trying to pull the entire table at client side. We will just show first
50-100 rows(1 page) and keep on serving subsequent pages on next page
requests.
Note: I was using sqlline to do that query. In Squirell, i think by default
for every query limit is set to 100.
On Tue, May 6, 2014 at 12:10 PM, James Taylor <ja...@apache.org>wrote:
> Yes, and just to add to that, those files should be deleted when the scan
> is closed. If that's not the case, please file a JIRA.
> Thanks,
> James
>
>
> On Tue, May 6, 2014 at 12:08 AM, Gabriel Reid <gabriel.reid@gmail.com
> >wrote:
>
> > Yes, Phoenix writes the results of scanners to an in-memory store that
> > spools into file storage once it gets too big (see [1]).
> >
> > 1.
> >
> https://github.com/apache/incubator-phoenix/blob/c22539d940fa0afc790cc0f25c2cdbbb255f80a3/phoenix-core/src/main/java/org/apache/phoenix/iterate/SpoolingResultIterator.java#L95
> >
> > On Sun, May 4, 2014 at 8:19 AM, anil gupta <an...@gmail.com>
> wrote:
> > > Hi,
> > >
> > > I would like to know why phoenix creates "ResultSpoolerxxxxxxxx.bin"
> > file?
> > > What is the content of these?
> > >
> > > I was using sqlline with phoenix3.0 and ran a "select * from table"
> > query.
> > > That table had around 10 million rows. Is phoenix really trying to read
> > > entire table?
> > >
> > > Eventually this exception was thrown by phoenix:
> > > java.lang.Runtime Exception:
> > > org.apache.phoenix.exception.PhoenixIOException:
> > > org.apache.phoenix.exception.PhoenixIOException: No space left on
> device
> > > at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
> > > at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
> > > at sqlline.SqlLine.print(SqlLine.java:1735)
> > > at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
> > > at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
> > > at sqlline.SqlLine.dispatch(SqlLine.java:821)
> > > at sqlline.SqlLine.begin(SqlLine.java:699)
> > > at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
> > > at sqlline.SqlLine.main(SqlLine.java:424)
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Phoenix creates ResultSpoolerxxxxxx.bin files in /tmp
Posted by James Taylor <ja...@apache.org>.
Yes, and just to add to that, those files should be deleted when the scan
is closed. If that's not the case, please file a JIRA.
Thanks,
James
On Tue, May 6, 2014 at 12:08 AM, Gabriel Reid <ga...@gmail.com>wrote:
> Yes, Phoenix writes the results of scanners to an in-memory store that
> spools into file storage once it gets too big (see [1]).
>
> 1.
> https://github.com/apache/incubator-phoenix/blob/c22539d940fa0afc790cc0f25c2cdbbb255f80a3/phoenix-core/src/main/java/org/apache/phoenix/iterate/SpoolingResultIterator.java#L95
>
> On Sun, May 4, 2014 at 8:19 AM, anil gupta <an...@gmail.com> wrote:
> > Hi,
> >
> > I would like to know why phoenix creates "ResultSpoolerxxxxxxxx.bin"
> file?
> > What is the content of these?
> >
> > I was using sqlline with phoenix3.0 and ran a "select * from table"
> query.
> > That table had around 10 million rows. Is phoenix really trying to read
> > entire table?
> >
> > Eventually this exception was thrown by phoenix:
> > java.lang.Runtime Exception:
> > org.apache.phoenix.exception.PhoenixIOException:
> > org.apache.phoenix.exception.PhoenixIOException: No space left on device
> > at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
> > at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
> > at sqlline.SqlLine.print(SqlLine.java:1735)
> > at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
> > at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
> > at sqlline.SqlLine.dispatch(SqlLine.java:821)
> > at sqlline.SqlLine.begin(SqlLine.java:699)
> > at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
> > at sqlline.SqlLine.main(SqlLine.java:424)
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
Re: Phoenix creates ResultSpoolerxxxxxx.bin files in /tmp
Posted by Gabriel Reid <ga...@gmail.com>.
Yes, Phoenix writes the results of scanners to an in-memory store that
spools into file storage once it gets too big (see [1]).
1. https://github.com/apache/incubator-phoenix/blob/c22539d940fa0afc790cc0f25c2cdbbb255f80a3/phoenix-core/src/main/java/org/apache/phoenix/iterate/SpoolingResultIterator.java#L95
On Sun, May 4, 2014 at 8:19 AM, anil gupta <an...@gmail.com> wrote:
> Hi,
>
> I would like to know why phoenix creates "ResultSpoolerxxxxxxxx.bin" file?
> What is the content of these?
>
> I was using sqlline with phoenix3.0 and ran a "select * from table" query.
> That table had around 10 million rows. Is phoenix really trying to read
> entire table?
>
> Eventually this exception was thrown by phoenix:
> java.lang.Runtime Exception:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: No space left on device
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
> at sqlline.SqlLine.print(SqlLine.java:1735)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
> at sqlline.SqlLine.dispatch(SqlLine.java:821)
> at sqlline.SqlLine.begin(SqlLine.java:699)
> at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
> at sqlline.SqlLine.main(SqlLine.java:424)
>
> --
> Thanks & Regards,
> Anil Gupta