You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by highpointe <hi...@gmail.com> on 2011/08/30 08:29:42 UTC
Real time dynamic data and hbase
We are attempting to build what is akin to a CRM (but not).
Our backend is an interface in which clients can control the variables of their assets offering; the variables of the template they use act as the over call on the Db.
Within the UI, they have the ability to set thresholds for variables (once reached the variable is ejected and in essence a completely new template is created).
The variables can change rapidly throughout the day so speed on updates and being able to always have the latest template data sets on the front end (as simple search engine) is a priority.
I have gone through numerous schema in my head but cannot seem to figure our the automating of the process of expiring data and introducing new variables on the fly without a complete reindex of the table.
Any help with architecture ideas would be greatly greatly appreciated.
Cheers.
H-p
Sent from my iPhone
Re: Real time dynamic data and hbase
Posted by high pointe <hi...@gmail.com>.
Very interesting.
Have you deployed such a setup? How is the performance? It seems it would
require quite a bit more than commodity hardware.
H-p
On Tue, Aug 30, 2011 at 12:37 AM, lohit <lo...@gmail.com> wrote:
> 2011/8/29 highpointe <hi...@gmail.com>
>
> > We are attempting to build what is akin to a CRM (but not).
> >
> > Our backend is an interface in which clients can control the variables of
> > their assets offering; the variables of the template they use act as the
> > over call on the Db.
> >
> > Within the UI, they have the ability to set thresholds for variables
> (once
> > reached the variable is ejected and in essence a completely new template
> is
> > created).
> >
> > The variables can change rapidly throughout the day so speed on updates
> and
> > being able to always have the latest template data sets on the front end
> (as
> > simple search engine) is a priority.
> >
> > I have gone through numerous schema in my head but cannot seem to figure
> > our the automating of the process of expiring data and introducing new
> > variables on the fly without a complete reindex of the table.
> >
> > Any help with architecture ideas would be greatly greatly appreciated.
> >
>
> Could coprocessors help in this case?
> http://hbaseblog.com/2010/11/30/hbase-coprocessors/
>
>
> >
> > Cheers.
> >
> > H-p
> >
> >
> >
> >
> > Sent from my iPhone
> >
>
>
>
> --
> Have a Nice Day!
> Lohit
>
Re: Real time dynamic data and hbase
Posted by lohit <lo...@gmail.com>.
2011/8/29 highpointe <hi...@gmail.com>
> We are attempting to build what is akin to a CRM (but not).
>
> Our backend is an interface in which clients can control the variables of
> their assets offering; the variables of the template they use act as the
> over call on the Db.
>
> Within the UI, they have the ability to set thresholds for variables (once
> reached the variable is ejected and in essence a completely new template is
> created).
>
> The variables can change rapidly throughout the day so speed on updates and
> being able to always have the latest template data sets on the front end (as
> simple search engine) is a priority.
>
> I have gone through numerous schema in my head but cannot seem to figure
> our the automating of the process of expiring data and introducing new
> variables on the fly without a complete reindex of the table.
>
> Any help with architecture ideas would be greatly greatly appreciated.
>
Could coprocessors help in this case?
http://hbaseblog.com/2010/11/30/hbase-coprocessors/
>
> Cheers.
>
> H-p
>
>
>
>
> Sent from my iPhone
>
--
Have a Nice Day!
Lohit
RE: Real time dynamic data and hbase
Posted by Michael Segel <mi...@hotmail.com>.
I don't understand why you're having trouble with this.
You have a simple geo location search based on zip and then a product and inventory count.
I mean its not really geo-spatial because you're searching based on zip code. So you don't need to worry about any sort of geospatial or geodetic components.
Note: This really isn't a problem that is solved by HBase but better solved by a relational model (RDBMs)
This sounds more like a homework problem than a real problem....
But here's a free clue.
You don't index on inventory count.
You filter the vendors on the client side or just add a filter to your query.
Sorry but as you describe it, its really a trivial problem.
(Which is why I think its a homework assignment.)
-Mike
> Date: Tue, 30 Aug 2011 00:57:37 -0600
> Subject: Re: Real time dynamic data and hbase
> From: highpointe3i@gmail.com
> To: user@hbase.apache.org
>
> Thanks for the response Sonal. Here is an example.
>
> The client (backend) is online grocers. The front end client is shoppers /
> consumers.
>
> The grocers have an interface that they can log into and enter their
> inventory. Example: Apple = 9
>
> The front end is simple; two values the users enter:
>
> Product: Apples
> Delivers to zip code: 90210
>
> Once the query is submitted, the search results will display all the grocers
> that deliver apples to the users area code.
>
> Whenever a grocer's supply (for any product) decreases to zero, they need to
> be dropped our of the results. We do not want grocers showing quantity
> zero. We only want grocers with a supply on hand.
>
> Now... Should the grocer that was dropped out because of a 0 value get a
> shipment and now have product on hand, once they enter the amount of their
> product, the non zero value should trigger the Db / search to include the
> grocer in the search again.
>
> How would I go about doing this without have multiple tables for the same
> grocer or constantly re-indexing?
>
> Am I going at this the wrong way completely?
>
> Thanks again for the response and for and suggestions.
>
> H-p
>
>
>
> As orders come through for apples and the number decreases, one it hits 0,
> the clients
>
>
>
>
> On Tue, Aug 30, 2011 at 12:35 AM, Sonal Goyal <so...@gmail.com> wrote:
>
> > Hi,
> >
> > Can you please give an example or explain in more detail what you are
> > trying
> > to achieve.
> >
> > Best Regards,
> > Sonal
> > Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> > Nube Technologies <http://www.nubetech.co>
> >
> > <http://in.linkedin.com/in/sonalgoyal>
> >
> >
> >
> >
> >
> > On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com>
> > wrote:
> >
> > > We are attempting to build what is akin to a CRM (but not).
> > >
> > > Our backend is an interface in which clients can control the variables of
> > > their assets offering; the variables of the template they use act as the
> > > over call on the Db.
> > >
> > > Within the UI, they have the ability to set thresholds for variables
> > (once
> > > reached the variable is ejected and in essence a completely new template
> > is
> > > created).
> > >
> > > The variables can change rapidly throughout the day so speed on updates
> > and
> > > being able to always have the latest template data sets on the front end
> > (as
> > > simple search engine) is a priority.
> > >
> > > I have gone through numerous schema in my head but cannot seem to figure
> > > our the automating of the process of expiring data and introducing new
> > > variables on the fly without a complete reindex of the table.
> > >
> > > Any help with architecture ideas would be greatly greatly appreciated.
> > >
> > > Cheers.
> > >
> > > H-p
> > >
> > >
> > >
> > >
> > > Sent from my iPhone
> > >
> >
RE: Real time dynamic data and hbase
Posted by Michael Segel <mi...@hotmail.com>.
You still need to organize your vendors by delivery zip.
Which gets very ugly when you try <product code> <grocerCode>.
Even doing something like <zip><product><vendor> as your key gets you a lot of rows.
This will work, where you have columns for price, qty on hand , sku, etc...
The problem gets worse when you consider some vendors have multiple warehouses and ship to multiple zips from a warehouse so you have a bit of a problem when you
want to place an order and update your inventory count.
NoSQL databases like HBase don't have transactions.
You want to try it and set it like
<product><vendor> and then have a column for each zip that it delivers to, and/or a column for each warehouse that contains a map of the zip codes and a column for each warehouse that contains the inventory count?
You can do this but again, you lack transactional concepts which make this a non-optimal solution.
Also this works if you're selecting vendors based on a product. Like I want Apples delivered to 44124 (Pepper Pike/Beachwood OH).
Not well for find me all of the grocers who ship to 44124 and get a list of their products.
HTH
-Mike
PS. This is why its a good relational problem and not a good big data. You can do it as a big data, but you end up with an ugly solution.
> Date: Tue, 30 Aug 2011 13:45:56 +0530
> Subject: Re: Real time dynamic data and hbase
> From: sonalgoyal4@gmail.com
> To: user@hbase.apache.org
>
> How about a table which has <product code><grocerCode> as the key, and
> column family items with qualifier quantity. You can have other column
> families like grocer(columns name etc) Then you can scan for productCode and
> return those grocers whose items:quantiy > 0.
>
> Best Regards,
> Sonal
> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Tue, Aug 30, 2011 at 12:27 PM, high pointe <hi...@gmail.com>wrote:
>
> > Thanks for the response Sonal. Here is an example.
> >
> > The client (backend) is online grocers. The front end client is shoppers /
> > consumers.
> >
> > The grocers have an interface that they can log into and enter their
> > inventory. Example: Apple = 9
> >
> > The front end is simple; two values the users enter:
> >
> > Product: Apples
> > Delivers to zip code: 90210
> >
> > Once the query is submitted, the search results will display all the
> > grocers
> > that deliver apples to the users area code.
> >
> > Whenever a grocer's supply (for any product) decreases to zero, they need
> > to
> > be dropped our of the results. We do not want grocers showing quantity
> > zero. We only want grocers with a supply on hand.
> >
> > Now... Should the grocer that was dropped out because of a 0 value get a
> > shipment and now have product on hand, once they enter the amount of their
> > product, the non zero value should trigger the Db / search to include the
> > grocer in the search again.
> >
> > How would I go about doing this without have multiple tables for the same
> > grocer or constantly re-indexing?
> >
> > Am I going at this the wrong way completely?
> >
> > Thanks again for the response and for and suggestions.
> >
> > H-p
> >
> >
> >
> > As orders come through for apples and the number decreases, one it hits 0,
> > the clients
> >
> >
> >
> >
> > On Tue, Aug 30, 2011 at 12:35 AM, Sonal Goyal <so...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Can you please give an example or explain in more detail what you are
> > > trying
> > > to achieve.
> > >
> > > Best Regards,
> > > Sonal
> > > Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> > > Nube Technologies <http://www.nubetech.co>
> > >
> > > <http://in.linkedin.com/in/sonalgoyal>
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com>
> > > wrote:
> > >
> > > > We are attempting to build what is akin to a CRM (but not).
> > > >
> > > > Our backend is an interface in which clients can control the variables
> > of
> > > > their assets offering; the variables of the template they use act as
> > the
> > > > over call on the Db.
> > > >
> > > > Within the UI, they have the ability to set thresholds for variables
> > > (once
> > > > reached the variable is ejected and in essence a completely new
> > template
> > > is
> > > > created).
> > > >
> > > > The variables can change rapidly throughout the day so speed on updates
> > > and
> > > > being able to always have the latest template data sets on the front
> > end
> > > (as
> > > > simple search engine) is a priority.
> > > >
> > > > I have gone through numerous schema in my head but cannot seem to
> > figure
> > > > our the automating of the process of expiring data and introducing new
> > > > variables on the fly without a complete reindex of the table.
> > > >
> > > > Any help with architecture ideas would be greatly greatly appreciated.
> > > >
> > > > Cheers.
> > > >
> > > > H-p
> > > >
> > > >
> > > >
> > > >
> > > > Sent from my iPhone
> > > >
> > >
> >
Re: Real time dynamic data and hbase
Posted by highpointe <hi...@gmail.com>.
Thank you all for your input. I think I am closer to a solution that will work.
Cheers.
H-p
Sent from my iPhone
On Aug 30, 2011, at 4:57 AM, Sonal Goyal <so...@gmail.com> wrote:
> I was talking about concatenating the product code and grocer code. Use that
> as the rowkey, and put quantity as a column. You can then scan for a
> particular product code, and return only those rows where quantity is
> greater than zero.
>
> Best Regards,
> Sonal
> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Tue, Aug 30, 2011 at 2:06 PM, highpointe <hi...@gmail.com> wrote:
>
>> Inline.
>>
>> Sent from my iPhone
>>
>> On Aug 30, 2011, at 2:15 AM, Sonal Goyal <so...@gmail.com> wrote:
>>
>>
>> This part I understand. And in essence the would be a key for each product
>> a grocer sells, yes?
>>
>>> How about a table which has <product code><grocerCode> as the key, and
>>> column family items with qualifier quantity.
>>
>>
>> I'm not sure what you mean here.
>>
>>> You can have other column
>>> families like grocer(columns name etc) Then you can scan for productCode
>> and
>>> return those grocers whose items:quantiy > 0.
>>>
>>
>> You mean use the productCode as the key and the value would depend on the
>> integer of the column and those matching >0 are returned as valid (the
>> value) and delivered to the UI?
>>
>>
>>> Best Regards,
>>> Sonal
>>> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
>>> Nube Technologies <http://www.nubetech.co>
>>>
>>> <http://in.linkedin.com/in/sonalgoyal>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Aug 30, 2011 at 12:27 PM, high pointe <highpointe3i@gmail.com
>>> wrote:
>>>
>>>> Thanks for the response Sonal. Here is an example.
>>>>
>>>> The client (backend) is online grocers. The front end client is
>> shoppers /
>>>> consumers.
>>>>
>>>> The grocers have an interface that they can log into and enter their
>>>> inventory. Example: Apple = 9
>>>>
>>>> The front end is simple; two values the users enter:
>>>>
>>>> Product: Apples
>>>> Delivers to zip code: 90210
>>>>
>>>> Once the query is submitted, the search results will display all the
>>>> grocers
>>>> that deliver apples to the users area code.
>>>>
>>>> Whenever a grocer's supply (for any product) decreases to zero, they
>> need
>>>> to
>>>> be dropped our of the results. We do not want grocers showing quantity
>>>> zero. We only want grocers with a supply on hand.
>>>>
>>>> Now... Should the grocer that was dropped out because of a 0 value get
>> a
>>>> shipment and now have product on hand, once they enter the amount of
>> their
>>>> product, the non zero value should trigger the Db / search to include
>> the
>>>> grocer in the search again.
>>>>
>>>> How would I go about doing this without have multiple tables for the
>> same
>>>> grocer or constantly re-indexing?
>>>>
>>>> Am I going at this the wrong way completely?
>>>>
>>>> Thanks again for the response and for and suggestions.
>>>>
>>>> H-p
>>>>
>>>>
>>>>
>>>> As orders come through for apples and the number decreases, one it hits
>> 0,
>>>> the clients
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 30, 2011 at 12:35 AM, Sonal Goyal <so...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can you please give an example or explain in more detail what you are
>>>>> trying
>>>>> to achieve.
>>>>>
>>>>> Best Regards,
>>>>> Sonal
>>>>> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
>>>>> Nube Technologies <http://www.nubetech.co>
>>>>>
>>>>> <http://in.linkedin.com/in/sonalgoyal>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We are attempting to build what is akin to a CRM (but not).
>>>>>>
>>>>>> Our backend is an interface in which clients can control the variables
>>>> of
>>>>>> their assets offering; the variables of the template they use act as
>>>> the
>>>>>> over call on the Db.
>>>>>>
>>>>>> Within the UI, they have the ability to set thresholds for variables
>>>>> (once
>>>>>> reached the variable is ejected and in essence a completely new
>>>> template
>>>>> is
>>>>>> created).
>>>>>>
>>>>>> The variables can change rapidly throughout the day so speed on
>> updates
>>>>> and
>>>>>> being able to always have the latest template data sets on the front
>>>> end
>>>>> (as
>>>>>> simple search engine) is a priority.
>>>>>>
>>>>>> I have gone through numerous schema in my head but cannot seem to
>>>> figure
>>>>>> our the automating of the process of expiring data and introducing new
>>>>>> variables on the fly without a complete reindex of the table.
>>>>>>
>>>>>> Any help with architecture ideas would be greatly greatly appreciated.
>>>>>>
>>>>>> Cheers.
>>>>>>
>>>>>> H-p
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>
>>>>
>>
Re: Real time dynamic data and hbase
Posted by Sonal Goyal <so...@gmail.com>.
I was talking about concatenating the product code and grocer code. Use that
as the rowkey, and put quantity as a column. You can then scan for a
particular product code, and return only those rows where quantity is
greater than zero.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Tue, Aug 30, 2011 at 2:06 PM, highpointe <hi...@gmail.com> wrote:
> Inline.
>
> Sent from my iPhone
>
> On Aug 30, 2011, at 2:15 AM, Sonal Goyal <so...@gmail.com> wrote:
>
>
> This part I understand. And in essence the would be a key for each product
> a grocer sells, yes?
>
> > How about a table which has <product code><grocerCode> as the key, and
> > column family items with qualifier quantity.
>
>
> I'm not sure what you mean here.
>
> > You can have other column
> > families like grocer(columns name etc) Then you can scan for productCode
> and
> > return those grocers whose items:quantiy > 0.
> >
>
> You mean use the productCode as the key and the value would depend on the
> integer of the column and those matching >0 are returned as valid (the
> value) and delivered to the UI?
>
>
> > Best Regards,
> > Sonal
> > Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> > Nube Technologies <http://www.nubetech.co>
> >
> > <http://in.linkedin.com/in/sonalgoyal>
> >
> >
> >
> >
> >
> > On Tue, Aug 30, 2011 at 12:27 PM, high pointe <highpointe3i@gmail.com
> >wrote:
> >
> >> Thanks for the response Sonal. Here is an example.
> >>
> >> The client (backend) is online grocers. The front end client is
> shoppers /
> >> consumers.
> >>
> >> The grocers have an interface that they can log into and enter their
> >> inventory. Example: Apple = 9
> >>
> >> The front end is simple; two values the users enter:
> >>
> >> Product: Apples
> >> Delivers to zip code: 90210
> >>
> >> Once the query is submitted, the search results will display all the
> >> grocers
> >> that deliver apples to the users area code.
> >>
> >> Whenever a grocer's supply (for any product) decreases to zero, they
> need
> >> to
> >> be dropped our of the results. We do not want grocers showing quantity
> >> zero. We only want grocers with a supply on hand.
> >>
> >> Now... Should the grocer that was dropped out because of a 0 value get
> a
> >> shipment and now have product on hand, once they enter the amount of
> their
> >> product, the non zero value should trigger the Db / search to include
> the
> >> grocer in the search again.
> >>
> >> How would I go about doing this without have multiple tables for the
> same
> >> grocer or constantly re-indexing?
> >>
> >> Am I going at this the wrong way completely?
> >>
> >> Thanks again for the response and for and suggestions.
> >>
> >> H-p
> >>
> >>
> >>
> >> As orders come through for apples and the number decreases, one it hits
> 0,
> >> the clients
> >>
> >>
> >>
> >>
> >> On Tue, Aug 30, 2011 at 12:35 AM, Sonal Goyal <so...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Can you please give an example or explain in more detail what you are
> >>> trying
> >>> to achieve.
> >>>
> >>> Best Regards,
> >>> Sonal
> >>> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> >>> Nube Technologies <http://www.nubetech.co>
> >>>
> >>> <http://in.linkedin.com/in/sonalgoyal>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com>
> >>> wrote:
> >>>
> >>>> We are attempting to build what is akin to a CRM (but not).
> >>>>
> >>>> Our backend is an interface in which clients can control the variables
> >> of
> >>>> their assets offering; the variables of the template they use act as
> >> the
> >>>> over call on the Db.
> >>>>
> >>>> Within the UI, they have the ability to set thresholds for variables
> >>> (once
> >>>> reached the variable is ejected and in essence a completely new
> >> template
> >>> is
> >>>> created).
> >>>>
> >>>> The variables can change rapidly throughout the day so speed on
> updates
> >>> and
> >>>> being able to always have the latest template data sets on the front
> >> end
> >>> (as
> >>>> simple search engine) is a priority.
> >>>>
> >>>> I have gone through numerous schema in my head but cannot seem to
> >> figure
> >>>> our the automating of the process of expiring data and introducing new
> >>>> variables on the fly without a complete reindex of the table.
> >>>>
> >>>> Any help with architecture ideas would be greatly greatly appreciated.
> >>>>
> >>>> Cheers.
> >>>>
> >>>> H-p
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>
> >>
>
Re: Real time dynamic data and hbase
Posted by highpointe <hi...@gmail.com>.
Inline.
Sent from my iPhone
On Aug 30, 2011, at 2:15 AM, Sonal Goyal <so...@gmail.com> wrote:
This part I understand. And in essence the would be a key for each product a grocer sells, yes?
> How about a table which has <product code><grocerCode> as the key, and
> column family items with qualifier quantity.
I'm not sure what you mean here.
> You can have other column
> families like grocer(columns name etc) Then you can scan for productCode and
> return those grocers whose items:quantiy > 0.
>
You mean use the productCode as the key and the value would depend on the integer of the column and those matching >0 are returned as valid (the value) and delivered to the UI?
> Best Regards,
> Sonal
> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Tue, Aug 30, 2011 at 12:27 PM, high pointe <hi...@gmail.com>wrote:
>
>> Thanks for the response Sonal. Here is an example.
>>
>> The client (backend) is online grocers. The front end client is shoppers /
>> consumers.
>>
>> The grocers have an interface that they can log into and enter their
>> inventory. Example: Apple = 9
>>
>> The front end is simple; two values the users enter:
>>
>> Product: Apples
>> Delivers to zip code: 90210
>>
>> Once the query is submitted, the search results will display all the
>> grocers
>> that deliver apples to the users area code.
>>
>> Whenever a grocer's supply (for any product) decreases to zero, they need
>> to
>> be dropped our of the results. We do not want grocers showing quantity
>> zero. We only want grocers with a supply on hand.
>>
>> Now... Should the grocer that was dropped out because of a 0 value get a
>> shipment and now have product on hand, once they enter the amount of their
>> product, the non zero value should trigger the Db / search to include the
>> grocer in the search again.
>>
>> How would I go about doing this without have multiple tables for the same
>> grocer or constantly re-indexing?
>>
>> Am I going at this the wrong way completely?
>>
>> Thanks again for the response and for and suggestions.
>>
>> H-p
>>
>>
>>
>> As orders come through for apples and the number decreases, one it hits 0,
>> the clients
>>
>>
>>
>>
>> On Tue, Aug 30, 2011 at 12:35 AM, Sonal Goyal <so...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Can you please give an example or explain in more detail what you are
>>> trying
>>> to achieve.
>>>
>>> Best Regards,
>>> Sonal
>>> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
>>> Nube Technologies <http://www.nubetech.co>
>>>
>>> <http://in.linkedin.com/in/sonalgoyal>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com>
>>> wrote:
>>>
>>>> We are attempting to build what is akin to a CRM (but not).
>>>>
>>>> Our backend is an interface in which clients can control the variables
>> of
>>>> their assets offering; the variables of the template they use act as
>> the
>>>> over call on the Db.
>>>>
>>>> Within the UI, they have the ability to set thresholds for variables
>>> (once
>>>> reached the variable is ejected and in essence a completely new
>> template
>>> is
>>>> created).
>>>>
>>>> The variables can change rapidly throughout the day so speed on updates
>>> and
>>>> being able to always have the latest template data sets on the front
>> end
>>> (as
>>>> simple search engine) is a priority.
>>>>
>>>> I have gone through numerous schema in my head but cannot seem to
>> figure
>>>> our the automating of the process of expiring data and introducing new
>>>> variables on the fly without a complete reindex of the table.
>>>>
>>>> Any help with architecture ideas would be greatly greatly appreciated.
>>>>
>>>> Cheers.
>>>>
>>>> H-p
>>>>
>>>>
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>
>>
Re: Real time dynamic data and hbase
Posted by Sonal Goyal <so...@gmail.com>.
How about a table which has <product code><grocerCode> as the key, and
column family items with qualifier quantity. You can have other column
families like grocer(columns name etc) Then you can scan for productCode and
return those grocers whose items:quantiy > 0.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Tue, Aug 30, 2011 at 12:27 PM, high pointe <hi...@gmail.com>wrote:
> Thanks for the response Sonal. Here is an example.
>
> The client (backend) is online grocers. The front end client is shoppers /
> consumers.
>
> The grocers have an interface that they can log into and enter their
> inventory. Example: Apple = 9
>
> The front end is simple; two values the users enter:
>
> Product: Apples
> Delivers to zip code: 90210
>
> Once the query is submitted, the search results will display all the
> grocers
> that deliver apples to the users area code.
>
> Whenever a grocer's supply (for any product) decreases to zero, they need
> to
> be dropped our of the results. We do not want grocers showing quantity
> zero. We only want grocers with a supply on hand.
>
> Now... Should the grocer that was dropped out because of a 0 value get a
> shipment and now have product on hand, once they enter the amount of their
> product, the non zero value should trigger the Db / search to include the
> grocer in the search again.
>
> How would I go about doing this without have multiple tables for the same
> grocer or constantly re-indexing?
>
> Am I going at this the wrong way completely?
>
> Thanks again for the response and for and suggestions.
>
> H-p
>
>
>
> As orders come through for apples and the number decreases, one it hits 0,
> the clients
>
>
>
>
> On Tue, Aug 30, 2011 at 12:35 AM, Sonal Goyal <so...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Can you please give an example or explain in more detail what you are
> > trying
> > to achieve.
> >
> > Best Regards,
> > Sonal
> > Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> > Nube Technologies <http://www.nubetech.co>
> >
> > <http://in.linkedin.com/in/sonalgoyal>
> >
> >
> >
> >
> >
> > On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com>
> > wrote:
> >
> > > We are attempting to build what is akin to a CRM (but not).
> > >
> > > Our backend is an interface in which clients can control the variables
> of
> > > their assets offering; the variables of the template they use act as
> the
> > > over call on the Db.
> > >
> > > Within the UI, they have the ability to set thresholds for variables
> > (once
> > > reached the variable is ejected and in essence a completely new
> template
> > is
> > > created).
> > >
> > > The variables can change rapidly throughout the day so speed on updates
> > and
> > > being able to always have the latest template data sets on the front
> end
> > (as
> > > simple search engine) is a priority.
> > >
> > > I have gone through numerous schema in my head but cannot seem to
> figure
> > > our the automating of the process of expiring data and introducing new
> > > variables on the fly without a complete reindex of the table.
> > >
> > > Any help with architecture ideas would be greatly greatly appreciated.
> > >
> > > Cheers.
> > >
> > > H-p
> > >
> > >
> > >
> > >
> > > Sent from my iPhone
> > >
> >
>
Re: Real time dynamic data and hbase
Posted by high pointe <hi...@gmail.com>.
Thanks for the response Sonal. Here is an example.
The client (backend) is online grocers. The front end client is shoppers /
consumers.
The grocers have an interface that they can log into and enter their
inventory. Example: Apple = 9
The front end is simple; two values the users enter:
Product: Apples
Delivers to zip code: 90210
Once the query is submitted, the search results will display all the grocers
that deliver apples to the users area code.
Whenever a grocer's supply (for any product) decreases to zero, they need to
be dropped our of the results. We do not want grocers showing quantity
zero. We only want grocers with a supply on hand.
Now... Should the grocer that was dropped out because of a 0 value get a
shipment and now have product on hand, once they enter the amount of their
product, the non zero value should trigger the Db / search to include the
grocer in the search again.
How would I go about doing this without have multiple tables for the same
grocer or constantly re-indexing?
Am I going at this the wrong way completely?
Thanks again for the response and for and suggestions.
H-p
As orders come through for apples and the number decreases, one it hits 0,
the clients
On Tue, Aug 30, 2011 at 12:35 AM, Sonal Goyal <so...@gmail.com> wrote:
> Hi,
>
> Can you please give an example or explain in more detail what you are
> trying
> to achieve.
>
> Best Regards,
> Sonal
> Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com>
> wrote:
>
> > We are attempting to build what is akin to a CRM (but not).
> >
> > Our backend is an interface in which clients can control the variables of
> > their assets offering; the variables of the template they use act as the
> > over call on the Db.
> >
> > Within the UI, they have the ability to set thresholds for variables
> (once
> > reached the variable is ejected and in essence a completely new template
> is
> > created).
> >
> > The variables can change rapidly throughout the day so speed on updates
> and
> > being able to always have the latest template data sets on the front end
> (as
> > simple search engine) is a priority.
> >
> > I have gone through numerous schema in my head but cannot seem to figure
> > our the automating of the process of expiring data and introducing new
> > variables on the fly without a complete reindex of the table.
> >
> > Any help with architecture ideas would be greatly greatly appreciated.
> >
> > Cheers.
> >
> > H-p
> >
> >
> >
> >
> > Sent from my iPhone
> >
>
Re: Real time dynamic data and hbase
Posted by Sonal Goyal <so...@gmail.com>.
Hi,
Can you please give an example or explain in more detail what you are trying
to achieve.
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Tue, Aug 30, 2011 at 11:59 AM, highpointe <hi...@gmail.com> wrote:
> We are attempting to build what is akin to a CRM (but not).
>
> Our backend is an interface in which clients can control the variables of
> their assets offering; the variables of the template they use act as the
> over call on the Db.
>
> Within the UI, they have the ability to set thresholds for variables (once
> reached the variable is ejected and in essence a completely new template is
> created).
>
> The variables can change rapidly throughout the day so speed on updates and
> being able to always have the latest template data sets on the front end (as
> simple search engine) is a priority.
>
> I have gone through numerous schema in my head but cannot seem to figure
> our the automating of the process of expiring data and introducing new
> variables on the fly without a complete reindex of the table.
>
> Any help with architecture ideas would be greatly greatly appreciated.
>
> Cheers.
>
> H-p
>
>
>
>
> Sent from my iPhone
>