You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "James Taylor (JIRA)" <ji...@apache.org> on 2014/08/28 09:55:00 UTC
[jira] [Commented] (PHOENIX-1217) Spool GROUP BY to disk using
HBase itself
[ https://issues.apache.org/jira/browse/PHOENIX-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113510#comment-14113510 ]
James Taylor commented on PHOENIX-1217:
---------------------------------------
Comments exploring this option via an email conversation between myself, Lars Hofhansl, and Marcel Kutsch:
Date: Tue, 25 Feb 2014 11:00:08 -0800
Subject: Re: using HBase to do spill
From: Lars Hofhansl <lh...@salesforce.com>
To: James Taylor <jt...@salesforce.com>
Cc: Marcel Kutsch <ku...@gmail.com>
Ah. So if you had direct access to the Memstore through the HRegion you
could it? Something like HRegion.getMemstore().
Maybe we can add that to HRegion going forward.
-- Lars
On Tue, Feb 25, 2014 at 10:47 AM, James Taylor <jt...@salesforce.com>wrote:
> It's ok if the last aggregation is on disk. A raw scan runs to do a final
> aggregation on the region server which combines them together. This is just
> an optimization, as in most cases, since the number of distinct values is
> low, you'd end up with a single batch of Puts that might never even be
> flushed. If you don't do this optimization, though, you end up writing one
> row per row in the region (even if there's say three distinct values among
> 10M rows). Functionally it'd still work, but perf would suck.
>
>
>
>
> On Tue, Feb 25, 2014 at 10:42 AM, Lars Hofhansl <lh...@salesforce.com>wrote:
>
>> Yep. That way you control what you keep in memory. Relying HRegion for
>> that you can only indirectly control when it gets flushed to disk, so on
>> each lookup you'd need to check the disk since the current value might be
>> found there.
>>
>> I guess even in your approach, how would you know that the last value of
>> the aggregation is not on disk? I guess each time you batch write the data
>> to the HRegion you'd need to seed the next batch with the former value.
>> What it that does not fit into memory? (I'm probably not understanding this
>> correctly...)
>>
>>
>> On Tue, Feb 25, 2014 at 10:02 AM, James Taylor <jt...@salesforce.com>wrote:
>>
>>> WFH today. I think my other alternative is simple and would work: batch
>>> the mutations yourself instead of using hregion.put() with each upsert.
>>> Then you can lookup the Put in your batch to combine them together. When
>>> the batch gets big enough, do a hregion.batchMutate() and set the flush
>>> size threshold very small (as essentially we'd want to have the flush occur
>>> when the batchMutate happens).
>>>
>>> Does that sound reasonable?
>>>
>>>
>>> On Tue, Feb 25, 2014 at 9:56 AM, Lars Hofhansl <lhofhansl@salesforce.com
>>> > wrote:
>>>
>>>> Hmm... Not sure the HRegion stuff would be right solution then, at
>>>> least not without further changes to the HRegion class. Let's chat when you
>>>> get in today.
>>>>
>>>>
>>>> On Mon, Feb 24, 2014 at 11:35 PM, James Taylor <jt...@salesforce.com>wrote:
>>>>
>>>>> I'd like to be able to opportunistically peek into the
>>>>> "not-yet-flushed" data as a poor-mans cache. The idea being that if we have
>>>>> lots of rows, but few distinct values, we'd never flush. I don't want to
>>>>> continually go to disk when the rows are being Put, though. I thought I'd
>>>>> be better to let HBase do it's merge sort when we do the raw scan before
>>>>> sending the results back to the client (so that we're not doing a bunch of
>>>>> random reads/seeks).
>>>>>
>>>>> I guess we could use a block cache, but then we're wasting the memory
>>>>> holding the "unflushed" data. There's no way to get to this?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 24, 2014 at 1:44 PM, Lars Hofhansl <
>>>>> lhofhansl@salesforce.com> wrote:
>>>>>
>>>>>> hregion.getStore(cf).get(rowKey), will also hit disk if the key is
>>>>>> not found in the memstore.
>>>>>>
>>>>>> Isn't that what you want, what if you actually had to spill to disk
>>>>>> at some point?
>>>>>> You mean is had to hit disk, just to check whether the KV exists? You
>>>>>> can mitigate this with a row or rowcol bloomfitler. Or would might need to
>>>>>> initialize a block cache a separate block cache.
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 24, 2014 at 12:26 PM, James Taylor <
>>>>>> jtaylor@salesforce.com> wrote:
>>>>>>
>>>>>>> On Mon, Feb 24, 2014 at 11:53 AM, Lars Hofhansl <
>>>>>>> lhofhansl@salesforce.com> wrote:
>>>>>>>
>>>>>>>> Assuming you'd only use one column family, you can do this:
>>>>>>>> 1. get() the old value from the HRegion
>>>>>>>>
>>>>>>> Wouldn't the hregion.get() hit disk if it doesn't find it in the
>>>>>>> memstore? That's what I want to avoid. I'd like to be able to do something
>>>>>>> along these lines:
>>>>>>> KeyValue kv = hregion.getStore(cf).get(rowKey);
>>>>>>> if (kv == null) {
>>>>>>> hregion.put(aggregatedKv);
>>>>>>> } else {
>>>>>>> aggregatedKv = aggregate(aggregatedKv, kv);
>>>>>>> hregion.getStore().upsert(aggregatedKv);
>>>>>>> }
>>>>>>> Is that possible?
>>>>>>>
>>>>>>>
>>>>>>>> 2. get the store from the HRegion (there's one per column family).
>>>>>>>> Either call getStore(<cf>) or getStores()
>>>>>>>> 3. update your key values accordingly and call Store.upsert. If the
>>>>>>>> KVs are still in the memstore they'll be updated in place.
>>>>>>>>
>>>>>>>> Upserts are not friendly to the GC since we cannot use the
>>>>>>>> memstoreLAB allocation for those (too much fragmentation inside the LABs).
>>>>>>>> So they should be used only where needed (like in aggregate, but not to
>>>>>>>> just overflow a result set that does not change). Upsert also is not MVCC
>>>>>>>> save (0.94), but that is not an issue for you.
>>>>>>>> (That reminds me... At some point we have to look through the
>>>>>>>> Phoenix code and make sure it's GC friendly.)
>>>>>>>>
>>>>>>>> -- Lars
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 24, 2014 at 11:27 AM, James Taylor <
>>>>>>>> jtaylor@salesforce.com> wrote:
>>>>>>>>
>>>>>>>>> Yes, the latter. The aggregate value is a "running total" for all
>>>>>>>>> rows with the same row key (formed from evaluating the group by and forming
>>>>>>>>> a row key). We need to be able to get back the current value from the
>>>>>>>>> memstore so we can aggregate it together with the new value and then upsert
>>>>>>>>> it back. We don't care about old versions in this case - it's better to not
>>>>>>>>> keep the old values as it'll consume less memory.
>>>>>>>>>
>>>>>>>>> Do we have to write our own version of an incrementColumnValue
>>>>>>>>> method (kind of an aggregateColumnValue)? How do we get the old value from
>>>>>>>>> the memstore (ensuring that we only look in memstore and return null if
>>>>>>>>> it's not found)?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> James
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 24, 2014 at 10:26 AM, Lars Hofhansl <
>>>>>>>>> lhofhansl@salesforce.com> wrote:
>>>>>>>>>
>>>>>>>>>> The memstore has an upsert() method that does exactly that. Note,
>>>>>>>>>> though, that that was specifically designed for incrementColumnValue,
>>>>>>>>>> append, etc. Is that what you mean?
>>>>>>>>>>
>>>>>>>>>> But I am not sure I follow what the problem is. Whether you use
>>>>>>>>>> your own data structure or an HRegion for spilling has no bearing on how
>>>>>>>>>> much data you write, no?
>>>>>>>>>> Is it because in an aggregation you update the same value over
>>>>>>>>>> and over again and you do not care about the old versions?
>>>>>>>>>>
>>>>>>>>>> -- Lars
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Feb 22, 2014 at 12:07 PM, James Taylor <
>>>>>>>>>> jtaylor@salesforce.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> (Lars - can you give advice on how to do the item in bold below)?
>>>>>>>>>>>
>>>>>>>>>>> Marcel, one thing I was thinking about was that in using this
>>>>>>>>>>> new algorithm, we'll end up essentially re-writing the entire region's
>>>>>>>>>>> worth of data, even in the case where there are only a few distinct values.
>>>>>>>>>>> That's because we'd be doing a Put for each row that we processes while
>>>>>>>>>>> scanning. One way to fix this would be to keep our own batch of Puts, and
>>>>>>>>>>> then instead of doing an hregion.put() each time, we'd do an
>>>>>>>>>>> hregion.batchMutate() when our batch becomes "big enough". Then we could
>>>>>>>>>>> check if a group by key is in our batch and aggregate them together (like
>>>>>>>>>>> we do today). That way, we'd only be writing the distinct values.
>>>>>>>>>>>
>>>>>>>>>>> *Another, more HBase-y way to do this would be to continue doing
>>>>>>>>>>> the hregion.put(), but for the lookup, we'd check the memstore and if
>>>>>>>>>>> found, replace that memstore kv with the aggregated kv. I'm not sure how to
>>>>>>>>>>> do it this way, but Lars would know.*
>>>>>>>>>>>
>>>>>>>>>>> The only problem with the former way is that we'd have our own
>>>>>>>>>>> batch and then they'd continue to be held in memory in the memstore after
>>>>>>>>>>> we do our hregion.batchMutate(), so we'd be double-buffering.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> James
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Feb 4, 2014 at 10:25 AM, Lars Hofhansl <
>>>>>>>>>>> lhofhansl@salesforce.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> This sounds right.
>>>>>>>>>>>> The only extra thing you need to do is creating a local
>>>>>>>>>>>> filesystem.
>>>>>>>>>>>> Something like FileSystem.get("file://<your temp location>",
>>>>>>>>>>>> new Configuration());
>>>>>>>>>>>> Might need to tweak the configuration, otherwise I'll pick up
>>>>>>>>>>>> the HBase default.
>>>>>>>>>>>>
>>>>>>>>>>>> Then - after looking a bit more - the best API to use is (in
>>>>>>>>>>>> HRegion):
>>>>>>>>>>>> public static HRegion newHRegion(Path tableDir, HLog log,
>>>>>>>>>>>> FileSystem fs,
>>>>>>>>>>>> Configuration conf, HRegionInfo regionInfo, final
>>>>>>>>>>>> HTableDescriptor htd,
>>>>>>>>>>>> RegionServerServices rsServices) {
>>>>>>>>>>>>
>>>>>>>>>>>> It can be called something like this:
>>>>>>>>>>>>
>>>>>>>>>>>> HRegion r = HRegion.newHRegion(tableDir, null, fs, conf, hri, htd, null);
>>>>>>>>>>>> r.initialize(null);
>>>>>>>>>>>> An HRegionInfo can be created simply with new
>>>>>>>>>>>> HRegionInfo(byte[] tableName). This will set both start and end key to
>>>>>>>>>>>> null, which means all keys fall in this region.
>>>>>>>>>>>>
>>>>>>>>>>>> Let me know how this works. If there's anything simplifying we
>>>>>>>>>>>> can put into HBase that would be good to know as well, I can make that
>>>>>>>>>>>> happen.
>>>>>>>>>>>>
>>>>>>>>>>>> -- Lars
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Feb 1, 2014 at 7:43 PM, James Taylor <
>>>>>>>>>>>> jtaylor@salesforce.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Marcel,
>>>>>>>>>>>>> Wow, this is great, and good timing as I spoke with Lars
>>>>>>>>>>>>> in-depth about this idea, and he liked it. He gave some good tips the APIs
>>>>>>>>>>>>> we can use to ensure that the spilled data stays local (*Lars
>>>>>>>>>>>>> - can you confirm I got this right?*).
>>>>>>>>>>>>>
>>>>>>>>>>>>> First, the trick to get a scan to show all versions, is to use
>>>>>>>>>>>>> the scan.setRaw(true) before getting the Scanner.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Second, would you mind coding against the mirror on Github of
>>>>>>>>>>>>> our Apache git here (as we've switch to Apache now):
>>>>>>>>>>>>> https://github.com/apache/incubator-phoenix? You can send me
>>>>>>>>>>>>> a pull request against this repo.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The tips that Lars gave me:
>>>>>>>>>>>>> - create your own HRegion which ensures that things stay local
>>>>>>>>>>>>> by using the static method: HRegion.openHRegion(Path tableDir,
>>>>>>>>>>>>> final HRegionInfo info, final HTableDescriptor htd, finalHLog wal,
>>>>>>>>>>>>> final Configuration conf)
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are a bunch of static constructors, so if the above
>>>>>>>>>>>>> doesn't work, we can ask Lars which one is the "right" one. The other one
>>>>>>>>>>>>> that looked promising to me was this one:
>>>>>>>>>>>>>
>>>>>>>>>>>>> public static HRegion createHRegion(final HRegionInfo info,
>>>>>>>>>>>>> final Path rootDir,
>>>>>>>>>>>>> final Configuration conf,
>>>>>>>>>>>>> final HTableDescriptor
>>>>>>>>>>>>> hTableDescriptor,
>>>>>>>>>>>>> final HLog hlog,
>>>>>>>>>>>>> final boolean
>>>>>>>>>>>>> initialize, final boolean ignoreHLog)
>>>>>>>>>>>>>
>>>>>>>>>>>>> with ignoreHLog=true, hlog=null, not sure about initialize,
>>>>>>>>>>>>> but probably true
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can create an HRegionInfo like this:
>>>>>>>>>>>>> new HRegionInfo(spillableTableNameAsBytes, null, null,
>>>>>>>>>>>>> false);
>>>>>>>>>>>>> For HLog, just use null
>>>>>>>>>>>>> For HTableDescriptor, use
>>>>>>>>>>>>> HTableDescriptor(spillableTableNameAsBytes) and then do an
>>>>>>>>>>>>> hTableDesc.addFamily(new
>>>>>>>>>>>>> HColumnDescriptor(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES))
>>>>>>>>>>>>> It doesn't really matter what you call the family, but you
>>>>>>>>>>>>> should call
>>>>>>>>>>>>> familyDesc.setMaxVersions(Integer.MAX_VALUE); // as you want
>>>>>>>>>>>>> to keep all versions
>>>>>>>>>>>>> Also do a
>>>>>>>>>>>>> tableDesc.setMaxFileSize(Long.MAX_VALUE); // as we never want
>>>>>>>>>>>>> to trigger a split
>>>>>>>>>>>>> And finally, an important one that controls how much each
>>>>>>>>>>>>> spillable group will hold in memory is:
>>>>>>>>>>>>>
>>>>>>>>>>>>> tableDesc.setMemStoreFlushSize(long memstoreFlushSize)
>>>>>>>>>>>>>
>>>>>>>>>>>>> We should make that configurable from Phoenix.
>>>>>>>>>>>>> - then use the hregion.put(put, false) to put the data. I
>>>>>>>>>>>>> don't think there's any advantage to batching here, since it's all local.
>>>>>>>>>>>>>
>>>>>>>>>>>>> One design decision/trade-off that we can worry about later is
>>>>>>>>>>>>> whether we want to spin up a single HRegion which would be shared across
>>>>>>>>>>>>> all regions spilling to a given region server, versus spinning up a new one
>>>>>>>>>>>>> each time. The problem with spinning up a new one each time is that each
>>>>>>>>>>>>> one will have a limited amount of memory versus a single region-server-wide
>>>>>>>>>>>>> global one that could have more memory allocated to it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The other issue is whether we want to always go through these
>>>>>>>>>>>>> APIs, or use the in-memory GroupByCache until we run into a memory issue
>>>>>>>>>>>>> and falling back to these. These HRegion APIs will use more memory, as
>>>>>>>>>>>>> we'll keep all the duplicates until the end and only then condense them
>>>>>>>>>>>>> down.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me know if you have questions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> James
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Feb 1, 2014 at 5:44 PM, Marcel Kutsch <
>>>>>>>>>>>>> kutschm@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi James,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I hacked up your proposed solution below.
>>>>>>>>>>>>>> When you get a chance, could you please take a quick look at
>>>>>>>>>>>>>> the attached file. I must be missing something,the scanner only returns the
>>>>>>>>>>>>>> latest version of the row and not all of them. Maybe you see
>>>>>>>>>>>>>> something obvious.
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Marcel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Marcel,
>>>>>>>>>>>>>> I think it's worth trying the HBase-based spill approach.
>>>>>>>>>>>>>> Should be pretty easy. Here's what you need to do:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) Pre-create a new Phoenix table from
>>>>>>>>>>>>>> ConnectionQueryServicesImpl.init(String url, Properties props). This
>>>>>>>>>>>>>> ensures that the spill table will always be there. Just add another
>>>>>>>>>>>>>> try/catch block after the one for QueryConstants.CREATE_SEQUENCE_METADATA
>>>>>>>>>>>>>> that issues the following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "CREATE TABLE SYSTEM.SPILLABLE_CACHE(k VARBINARY PRIMARY
>>>>>>>>>>>>>> KEY) " +
>>>>>>>>>>>>>> HConstants.VERSIONS + "=" + Integer.MAX_VALUE +","
>>>>>>>>>>>>>> HColumnDescriptor.DATA_BLOCK_ENCODING + "=" + "NONE"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We might also want to play around with the
>>>>>>>>>>>>>> HTableDescriptor.MEMSTORE_FLUSHSIZE (the size before the in-memory cache
>>>>>>>>>>>>>> gets flushed to disk) and HTableDescriptor.MAX_FILESIZE (how big a file can
>>>>>>>>>>>>>> get before it's split).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) From your co-processor, you'd get your hands on an HTable
>>>>>>>>>>>>>> RegionCoprocessorEnvironment env = c.getEnvironment();
>>>>>>>>>>>>>> HTableInterface htable =
>>>>>>>>>>>>>> env.getTable(Bytes.toBytes("SYSTEM.SPILLABLE_CACHE"));
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3) Just use the regular HBase API to spill the cache:
>>>>>>>>>>>>>> long ts = 1;
>>>>>>>>>>>>>> List<Mutation> puts =
>>>>>>>>>>>>>> Lists.newArrayListWithExpectedSize(BATCH_SIZE);
>>>>>>>>>>>>>> // Create a Put and add the key and value that you
>>>>>>>>>>>>>> used in your SpillableGroupByCache
>>>>>>>>>>>>>> Put put = new Put(key, ts++); // Increment timestamp
>>>>>>>>>>>>>> for every Put
>>>>>>>>>>>>>> put.setWriteToWAL(false);
>>>>>>>>>>>>>> put.add(QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES,
>>>>>>>>>>>>>> QueryConstants.EMPTY_COLUMN_BYTES, value);
>>>>>>>>>>>>>> puts.add(put);
>>>>>>>>>>>>>> ....
>>>>>>>>>>>>>> // Once there are enough in the batch, call
>>>>>>>>>>>>>> htable.batch
>>>>>>>>>>>>>> // Not sure how to prevent these puts from going out
>>>>>>>>>>>>>> over the wire
>>>>>>>>>>>>>> htable.batch(puts);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4) To access, just implement your own RegionScanner that
>>>>>>>>>>>>>> wraps a ResultScanner and use this to implement the
>>>>>>>>>>>>>> GroupByCache.getScanner() method.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Scan scan = new Scan();
>>>>>>>>>>>>>> scan.setRaw(true);
>>>>>>>>>>>>>> // We may need to "preallocate" a chunk of timestamps
>>>>>>>>>>>>>> from an Atomic long
>>>>>>>>>>>>>> // so that multiple, simultaneous spillages don't
>>>>>>>>>>>>>> tromp on each other. We can
>>>>>>>>>>>>>> // do this after perf testing, though.
>>>>>>>>>>>>>> scan.setTimeRange(new TimeRange(minTs, maxTs);
>>>>>>>>>>>>>> // If we can keep the data local, this scanner will
>>>>>>>>>>>>>> bypass the RPC and deserialization
>>>>>>>>>>>>>> // hit we'd otherwise take.
>>>>>>>>>>>>>> ResultScanner scanner = htable.getScanner(scan);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Take a look at
>>>>>>>>>>>>>> GroupedAggregateRegionObserver.scanOrdered(), as this is essentially how
>>>>>>>>>>>>>> you want to implement your RegionScanner. Think of this scanner above as
>>>>>>>>>>>>>> the RegionScanner argument to the scanOrdered() method. You basically want
>>>>>>>>>>>>>> to aggregate together all rows with the same row key and add this to the
>>>>>>>>>>>>>> List<KeyValue> results argument for the next method of your RegionScanner
>>>>>>>>>>>>>> impl:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @Override
>>>>>>>>>>>>>> public boolean next(List<KeyValue> results)
>>>>>>>>>>>>>> throws IOException
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me know what you think. I'll ask Lars in the meantime if
>>>>>>>>>>>>>> there's a way to ensure keeping the data local for a table. One way we
>>>>>>>>>>>>>> could do it ourselves is to pre-split our SYSTEM.SPILLABLE_CACHE table and
>>>>>>>>>>>>>> then send over from the client a byte[] startKey that we know lives on the
>>>>>>>>>>>>>> particular region server on which we'd be doing the spilling (or perhaps we
>>>>>>>>>>>>>> can dynamically calculate before spilling). Then, we'd just prepend this
>>>>>>>>>>>>>> startKey to our key and the data would essentially be pegged to the region
>>>>>>>>>>>>>> server on which we're working. We can make it fixed length, and then just
>>>>>>>>>>>>>> skip those bytes with a fixed offset for the KeyValue we send back in (4).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> Spool GROUP BY to disk using HBase itself
> -----------------------------------------
>
> Key: PHOENIX-1217
> URL: https://issues.apache.org/jira/browse/PHOENIX-1217
> Project: Phoenix
> Issue Type: Bug
> Reporter: James Taylor
>
> We have an implementation in place now to spill a GROUP BY to disk on the server side if it gets too big to fit into memory. An alternate, simpler implementation would be to use HBase itself during this spooling instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)