You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/10/22 08:51:09 UTC
Index in Phoenix view on Hbase is not updated
Hi,
I have a Hbase table that is populated via
org.apache.hadoop.hbase.mapreduce.ImportTsv
through bulk load ever 15 minutes. This works fine.
In Phoenix I created a view on this table
jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
"marketDataHbase" ("price_info"."ticker", "price_info"."price",
"price_info"."timecreated");
This also does what is supposed to do and shows correct count.
I then created an index in Phoenix as below
create index index_dx1 on "marketDataHbase"
("price_info"."timecreated","price_info"."ticker", "price_info"."price");
that showed the records OK at that time. I verified this using explain
0: jdbc:phoenix:rhes564:2181> explain select count(1) from
"marketDataHbase";
+---------------------------------------------------------+
| PLAN |
+---------------------------------------------------------+
| CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
| SERVER FILTER BY FIRST KEY ONLY |
| SERVER AGGREGATE INTO SINGLE ROW |
+---------------------------------------------------------+
Now the issue is that the above does not show new data since build in Hbase
table unless I do the following:
0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
rebuild;
Which is not what an index should do (The covered index should be
maintained automatically).
The simple issue is how to overcome this problem?
As I understand the index in Phoenix ia another file independent of the
original phoenix view so I assume that this index file is not updated for
one reason or other?
Thanks
Re: Index in Phoenix view on Hbase is not updated
Posted by anil gupta <an...@gmail.com>.
Hi MIch,
I would recommend you to use Phoenix API/Tools to write data to a Phoenix
table so that it can handle secondary index seamlessly. Your approach of
**Rebuilding** index after every bulkload will run into scalability
problems as your primary table keeps growing.
~Anil
On Sat, Oct 22, 2016 at 11:22 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> wrote:
> A workout I deployed was to rebuild the index immediately after bulk load
> of data into Hbase table
>
> ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 22 October 2016 at 18:25, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Sorry Ted,
>>
>> This is the syntax for view
>>
>> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
>> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
>> "price_info"."price" VARCHAR);
>>
>> Thanks James for clarification.
>>
>> My understanding is that when one creates an index on a Phoenix view on
>> an Hbase table, a local index file is created in Phoenix with data as is
>> and that Phoenix table has no way of knowing if new data is added to Hbase
>> table.
>>
>> From an operational point of view unless index in Phoenix is immediately
>> updated after any append to Hbase, then it is pretty useless.
>>
>> In my case data is inserted to Hbase table. I am just using Phoenix for
>> data queries (DQ) as opposed to inserts.
>>
>> Regards,
>>
>>
>>
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 22 October 2016 at 17:59, James Taylor <ja...@apache.org> wrote:
>>
>>> Hi Mich,
>>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>>> data.
>>> Thanks,
>>> James
>>>
>>>
>>> On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The first statement creates index, not view.
>>>>
>>>> Can you check ?
>>>>
>>>> Cheers
>>>>
>>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I have a Hbase table that is populated via
>>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>>> > through bulk load ever 15 minutes. This works fine.
>>>> >
>>>> > In Phoenix I created a view on this table
>>>> >
>>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>>> > "price_info"."timecreated");
>>>> >
>>>> > This also does what is supposed to do and shows correct count.
>>>> >
>>>> > I then created an index in Phoenix as below
>>>> >
>>>> > create index index_dx1 on "marketDataHbase"
>>>> > ("price_info"."timecreated","price_info"."ticker",
>>>> "price_info"."price");
>>>> >
>>>> > that showed the records OK at that time. I verified this using explain
>>>> >
>>>> >
>>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>>> > "marketDataHbase";
>>>> > +---------------------------------------------------------+
>>>> > | PLAN |
>>>> > +---------------------------------------------------------+
>>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
>>>> > | SERVER FILTER BY FIRST KEY ONLY |
>>>> > | SERVER AGGREGATE INTO SINGLE ROW |
>>>> > +---------------------------------------------------------+
>>>> >
>>>> > Now the issue is that the above does not show new data since build in
>>>> Hbase
>>>> > table unless I do the following:
>>>> >
>>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>>> "marketDataHbase"
>>>> > rebuild;
>>>> >
>>>> >
>>>> > Which is not what an index should do (The covered index should be
>>>> > maintained automatically).
>>>> > The simple issue is how to overcome this problem?
>>>> >
>>>> > As I understand the index in Phoenix ia another file independent of
>>>> the
>>>> > original phoenix view so I assume that this index file is not updated
>>>> for
>>>> > one reason or other?
>>>> >
>>>> > Thanks
>>>>
>>>
>>
>
--
Thanks & Regards,
Anil Gupta
Re: Index in Phoenix view on Hbase is not updated
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Ankit,
I created a dummy table in Hbase as below
create 'dummy', 'price_info'
Then in Phoenix I created a table on Hbase table
create table "dummy" (PK VARCHAR PRIMARY KEY, "price_info"."ticker"
VARCHAR,"price_info"."timecreated" VARCHAR, "price_info"."price" VARCHAR);
And then used the following comman to load the csv file
HADOOP_CLASSPATH=/home/hduser/jars/hbase-protocol-1.2.3.jar:/usr/lib/hbase/conf
hadoop jar phoenix-4.8.1-HBase-1.2-client.jar
org.apache.phoenix.mapreduce.CsvBulkLoadTool
--table dummy --input /data/prices/2016-10-23/prices.1477228923115
However, it does not seem to find the table dummy !
2016-10-23 14:38:39,442 INFO [main] metrics.Metrics: Initializing metrics
system: phoenix
2016-10-23 14:38:39,479 INFO [main] impl.MetricsConfig: loaded properties
from hadoop-metrics2.properties
2016-10-23 14:38:39,529 INFO [main] impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2016-10-23 14:38:39,529 INFO [main] impl.MetricsSystemImpl: phoenix
metrics system started
Exception in thread "main" java.lang.IllegalArgumentException: Table DUMMY
not found
at org.apache.phoenix.util.SchemaUtil.generateColumnInfo(
SchemaUtil.java:873)
at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.
buildImportColumns(AbstractBulkLoadTool.java:377)
at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(
AbstractBulkLoadTool.java:214)
at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(
AbstractBulkLoadTool.java:183)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(
CsvBulkLoadTool.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
I tried putting it in "" etc but no joy I am afraid!
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 23 October 2016 at 14:12, Ankit Singhal <an...@gmail.com> wrote:
> bq. Will bulk load from Phoenix update the underlying Hbase table?
> Yes. instead of using importTSV try to use CSV bulkload only.
>
> bq. Do I need to replace Phoenix view on Hbase as with CREATE TABLE?
> You can still keep VIEW.
>
> Regards,
> Ankit Singhal
>
> On Sun, Oct 23, 2016 at 6:37 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> Thanks Ankit,
>>
>> couple of questions.
>>
>>
>> 1. Will bulk load from Phoenix update the underlying Hbase table?
>> 2. Do I need to replace Phoenix view on Hbase as with CREATE TABLE?
>>
>> regards
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 23 October 2016 at 13:52, Ankit Singhal <an...@gmail.com>
>> wrote:
>>
>>> Rebuild is currently a costly operation as it will rebuild the complete
>>> index again and should be used when you think your index is corrupted and
>>> you are not aware about the timestamp when it got out of sync.
>>>
>>> Why can't you use Bulk loading tool[1] provided by Phoenix instead of
>>> using importTSV, as this tool will build the data for index table along
>>> with data table?
>>> https://phoenix.apache.org/bulk_dataload.html
>>>
>>>
>>> On Sat, Oct 22, 2016 at 11:52 PM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> A workout I deployed was to rebuild the index immediately after bulk
>>>> load of data into Hbase table
>>>>
>>>> ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;
>>>>
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 22 October 2016 at 18:25, Mich Talebzadeh <mich.talebzadeh@gmail.com
>>>> > wrote:
>>>>
>>>>> Sorry Ted,
>>>>>
>>>>> This is the syntax for view
>>>>>
>>>>> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
>>>>> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
>>>>> "price_info"."price" VARCHAR);
>>>>>
>>>>> Thanks James for clarification.
>>>>>
>>>>> My understanding is that when one creates an index on a Phoenix view
>>>>> on an Hbase table, a local index file is created in Phoenix with data as
>>>>> is and that Phoenix table has no way of knowing if new data is added to
>>>>> Hbase table.
>>>>>
>>>>> From an operational point of view unless index in Phoenix is
>>>>> immediately updated after any append to Hbase, then it is pretty useless.
>>>>>
>>>>> In my case data is inserted to Hbase table. I am just using Phoenix
>>>>> for data queries (DQ) as opposed to inserts.
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>> On 22 October 2016 at 17:59, James Taylor <ja...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Mich,
>>>>>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>>>>>> data.
>>>>>> Thanks,
>>>>>> James
>>>>>>
>>>>>>
>>>>>> On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> The first statement creates index, not view.
>>>>>>>
>>>>>>> Can you check ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>>> >
>>>>>>> > Hi,
>>>>>>> >
>>>>>>> > I have a Hbase table that is populated via
>>>>>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>>>>>> > through bulk load ever 15 minutes. This works fine.
>>>>>>> >
>>>>>>> > In Phoenix I created a view on this table
>>>>>>> >
>>>>>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>>>>>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>>>>>> > "price_info"."timecreated");
>>>>>>> >
>>>>>>> > This also does what is supposed to do and shows correct count.
>>>>>>> >
>>>>>>> > I then created an index in Phoenix as below
>>>>>>> >
>>>>>>> > create index index_dx1 on "marketDataHbase"
>>>>>>> > ("price_info"."timecreated","price_info"."ticker",
>>>>>>> "price_info"."price");
>>>>>>> >
>>>>>>> > that showed the records OK at that time. I verified this using
>>>>>>> explain
>>>>>>> >
>>>>>>> >
>>>>>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>>>>>> > "marketDataHbase";
>>>>>>> > +---------------------------------------------------------+
>>>>>>> > | PLAN |
>>>>>>> > +---------------------------------------------------------+
>>>>>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
>>>>>>> > | SERVER FILTER BY FIRST KEY ONLY |
>>>>>>> > | SERVER AGGREGATE INTO SINGLE ROW |
>>>>>>> > +---------------------------------------------------------+
>>>>>>> >
>>>>>>> > Now the issue is that the above does not show new data since build
>>>>>>> in Hbase
>>>>>>> > table unless I do the following:
>>>>>>> >
>>>>>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>>>>>> "marketDataHbase"
>>>>>>> > rebuild;
>>>>>>> >
>>>>>>> >
>>>>>>> > Which is not what an index should do (The covered index should be
>>>>>>> > maintained automatically).
>>>>>>> > The simple issue is how to overcome this problem?
>>>>>>> >
>>>>>>> > As I understand the index in Phoenix ia another file independent
>>>>>>> of the
>>>>>>> > original phoenix view so I assume that this index file is not
>>>>>>> updated for
>>>>>>> > one reason or other?
>>>>>>> >
>>>>>>> > Thanks
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Index in Phoenix view on Hbase is not updated
Posted by Ankit Singhal <an...@gmail.com>.
bq. Will bulk load from Phoenix update the underlying Hbase table?
Yes. instead of using importTSV try to use CSV bulkload only.
bq. Do I need to replace Phoenix view on Hbase as with CREATE TABLE?
You can still keep VIEW.
Regards,
Ankit Singhal
On Sun, Oct 23, 2016 at 6:37 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Thanks Ankit,
>
> couple of questions.
>
>
> 1. Will bulk load from Phoenix update the underlying Hbase table?
> 2. Do I need to replace Phoenix view on Hbase as with CREATE TABLE?
>
> regards
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 23 October 2016 at 13:52, Ankit Singhal <an...@gmail.com>
> wrote:
>
>> Rebuild is currently a costly operation as it will rebuild the complete
>> index again and should be used when you think your index is corrupted and
>> you are not aware about the timestamp when it got out of sync.
>>
>> Why can't you use Bulk loading tool[1] provided by Phoenix instead of
>> using importTSV, as this tool will build the data for index table along
>> with data table?
>> https://phoenix.apache.org/bulk_dataload.html
>>
>>
>> On Sat, Oct 22, 2016 at 11:52 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> A workout I deployed was to rebuild the index immediately after bulk
>>> load of data into Hbase table
>>>
>>> ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 22 October 2016 at 18:25, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>>> Sorry Ted,
>>>>
>>>> This is the syntax for view
>>>>
>>>> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
>>>> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
>>>> "price_info"."price" VARCHAR);
>>>>
>>>> Thanks James for clarification.
>>>>
>>>> My understanding is that when one creates an index on a Phoenix view on
>>>> an Hbase table, a local index file is created in Phoenix with data as is
>>>> and that Phoenix table has no way of knowing if new data is added to Hbase
>>>> table.
>>>>
>>>> From an operational point of view unless index in Phoenix is
>>>> immediately updated after any append to Hbase, then it is pretty useless.
>>>>
>>>> In my case data is inserted to Hbase table. I am just using Phoenix for
>>>> data queries (DQ) as opposed to inserts.
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 22 October 2016 at 17:59, James Taylor <ja...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Mich,
>>>>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>>>>> data.
>>>>> Thanks,
>>>>> James
>>>>>
>>>>>
>>>>> On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> The first statement creates index, not view.
>>>>>>
>>>>>> Can you check ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>> >
>>>>>> > Hi,
>>>>>> >
>>>>>> > I have a Hbase table that is populated via
>>>>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>>>>> > through bulk load ever 15 minutes. This works fine.
>>>>>> >
>>>>>> > In Phoenix I created a view on this table
>>>>>> >
>>>>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>>>>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>>>>> > "price_info"."timecreated");
>>>>>> >
>>>>>> > This also does what is supposed to do and shows correct count.
>>>>>> >
>>>>>> > I then created an index in Phoenix as below
>>>>>> >
>>>>>> > create index index_dx1 on "marketDataHbase"
>>>>>> > ("price_info"."timecreated","price_info"."ticker",
>>>>>> "price_info"."price");
>>>>>> >
>>>>>> > that showed the records OK at that time. I verified this using
>>>>>> explain
>>>>>> >
>>>>>> >
>>>>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>>>>> > "marketDataHbase";
>>>>>> > +---------------------------------------------------------+
>>>>>> > | PLAN |
>>>>>> > +---------------------------------------------------------+
>>>>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
>>>>>> > | SERVER FILTER BY FIRST KEY ONLY |
>>>>>> > | SERVER AGGREGATE INTO SINGLE ROW |
>>>>>> > +---------------------------------------------------------+
>>>>>> >
>>>>>> > Now the issue is that the above does not show new data since build
>>>>>> in Hbase
>>>>>> > table unless I do the following:
>>>>>> >
>>>>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>>>>> "marketDataHbase"
>>>>>> > rebuild;
>>>>>> >
>>>>>> >
>>>>>> > Which is not what an index should do (The covered index should be
>>>>>> > maintained automatically).
>>>>>> > The simple issue is how to overcome this problem?
>>>>>> >
>>>>>> > As I understand the index in Phoenix ia another file independent of
>>>>>> the
>>>>>> > original phoenix view so I assume that this index file is not
>>>>>> updated for
>>>>>> > one reason or other?
>>>>>> >
>>>>>> > Thanks
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Index in Phoenix view on Hbase is not updated
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Ankit,
couple of questions.
1. Will bulk load from Phoenix update the underlying Hbase table?
2. Do I need to replace Phoenix view on Hbase as with CREATE TABLE?
regards
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 23 October 2016 at 13:52, Ankit Singhal <an...@gmail.com> wrote:
> Rebuild is currently a costly operation as it will rebuild the complete
> index again and should be used when you think your index is corrupted and
> you are not aware about the timestamp when it got out of sync.
>
> Why can't you use Bulk loading tool[1] provided by Phoenix instead of
> using importTSV, as this tool will build the data for index table along
> with data table?
> https://phoenix.apache.org/bulk_dataload.html
>
>
> On Sat, Oct 22, 2016 at 11:52 PM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> A workout I deployed was to rebuild the index immediately after bulk load
>> of data into Hbase table
>>
>> ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 22 October 2016 at 18:25, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> Sorry Ted,
>>>
>>> This is the syntax for view
>>>
>>> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
>>> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
>>> "price_info"."price" VARCHAR);
>>>
>>> Thanks James for clarification.
>>>
>>> My understanding is that when one creates an index on a Phoenix view on
>>> an Hbase table, a local index file is created in Phoenix with data as is
>>> and that Phoenix table has no way of knowing if new data is added to Hbase
>>> table.
>>>
>>> From an operational point of view unless index in Phoenix is immediately
>>> updated after any append to Hbase, then it is pretty useless.
>>>
>>> In my case data is inserted to Hbase table. I am just using Phoenix for
>>> data queries (DQ) as opposed to inserts.
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 22 October 2016 at 17:59, James Taylor <ja...@apache.org>
>>> wrote:
>>>
>>>> Hi Mich,
>>>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>>>> data.
>>>> Thanks,
>>>> James
>>>>
>>>>
>>>> On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The first statement creates index, not view.
>>>>>
>>>>> Can you check ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I have a Hbase table that is populated via
>>>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>>>> > through bulk load ever 15 minutes. This works fine.
>>>>> >
>>>>> > In Phoenix I created a view on this table
>>>>> >
>>>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>>>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>>>> > "price_info"."timecreated");
>>>>> >
>>>>> > This also does what is supposed to do and shows correct count.
>>>>> >
>>>>> > I then created an index in Phoenix as below
>>>>> >
>>>>> > create index index_dx1 on "marketDataHbase"
>>>>> > ("price_info"."timecreated","price_info"."ticker",
>>>>> "price_info"."price");
>>>>> >
>>>>> > that showed the records OK at that time. I verified this using
>>>>> explain
>>>>> >
>>>>> >
>>>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>>>> > "marketDataHbase";
>>>>> > +---------------------------------------------------------+
>>>>> > | PLAN |
>>>>> > +---------------------------------------------------------+
>>>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
>>>>> > | SERVER FILTER BY FIRST KEY ONLY |
>>>>> > | SERVER AGGREGATE INTO SINGLE ROW |
>>>>> > +---------------------------------------------------------+
>>>>> >
>>>>> > Now the issue is that the above does not show new data since build
>>>>> in Hbase
>>>>> > table unless I do the following:
>>>>> >
>>>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>>>> "marketDataHbase"
>>>>> > rebuild;
>>>>> >
>>>>> >
>>>>> > Which is not what an index should do (The covered index should be
>>>>> > maintained automatically).
>>>>> > The simple issue is how to overcome this problem?
>>>>> >
>>>>> > As I understand the index in Phoenix ia another file independent of
>>>>> the
>>>>> > original phoenix view so I assume that this index file is not
>>>>> updated for
>>>>> > one reason or other?
>>>>> >
>>>>> > Thanks
>>>>>
>>>>
>>>
>>
>
Re: Index in Phoenix view on Hbase is not updated
Posted by Ankit Singhal <an...@gmail.com>.
Rebuild is currently a costly operation as it will rebuild the complete
index again and should be used when you think your index is corrupted and
you are not aware about the timestamp when it got out of sync.
Why can't you use Bulk loading tool[1] provided by Phoenix instead of using
importTSV, as this tool will build the data for index table along with data
table?
https://phoenix.apache.org/bulk_dataload.html
On Sat, Oct 22, 2016 at 11:52 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> wrote:
> A workout I deployed was to rebuild the index immediately after bulk load
> of data into Hbase table
>
> ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 22 October 2016 at 18:25, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Sorry Ted,
>>
>> This is the syntax for view
>>
>> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
>> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
>> "price_info"."price" VARCHAR);
>>
>> Thanks James for clarification.
>>
>> My understanding is that when one creates an index on a Phoenix view on
>> an Hbase table, a local index file is created in Phoenix with data as is
>> and that Phoenix table has no way of knowing if new data is added to Hbase
>> table.
>>
>> From an operational point of view unless index in Phoenix is immediately
>> updated after any append to Hbase, then it is pretty useless.
>>
>> In my case data is inserted to Hbase table. I am just using Phoenix for
>> data queries (DQ) as opposed to inserts.
>>
>> Regards,
>>
>>
>>
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 22 October 2016 at 17:59, James Taylor <ja...@apache.org> wrote:
>>
>>> Hi Mich,
>>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>>> data.
>>> Thanks,
>>> James
>>>
>>>
>>> On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The first statement creates index, not view.
>>>>
>>>> Can you check ?
>>>>
>>>> Cheers
>>>>
>>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I have a Hbase table that is populated via
>>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>>> > through bulk load ever 15 minutes. This works fine.
>>>> >
>>>> > In Phoenix I created a view on this table
>>>> >
>>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>>> > "price_info"."timecreated");
>>>> >
>>>> > This also does what is supposed to do and shows correct count.
>>>> >
>>>> > I then created an index in Phoenix as below
>>>> >
>>>> > create index index_dx1 on "marketDataHbase"
>>>> > ("price_info"."timecreated","price_info"."ticker",
>>>> "price_info"."price");
>>>> >
>>>> > that showed the records OK at that time. I verified this using explain
>>>> >
>>>> >
>>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>>> > "marketDataHbase";
>>>> > +---------------------------------------------------------+
>>>> > | PLAN |
>>>> > +---------------------------------------------------------+
>>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
>>>> > | SERVER FILTER BY FIRST KEY ONLY |
>>>> > | SERVER AGGREGATE INTO SINGLE ROW |
>>>> > +---------------------------------------------------------+
>>>> >
>>>> > Now the issue is that the above does not show new data since build in
>>>> Hbase
>>>> > table unless I do the following:
>>>> >
>>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>>> "marketDataHbase"
>>>> > rebuild;
>>>> >
>>>> >
>>>> > Which is not what an index should do (The covered index should be
>>>> > maintained automatically).
>>>> > The simple issue is how to overcome this problem?
>>>> >
>>>> > As I understand the index in Phoenix ia another file independent of
>>>> the
>>>> > original phoenix view so I assume that this index file is not updated
>>>> for
>>>> > one reason or other?
>>>> >
>>>> > Thanks
>>>>
>>>
>>
>
Re: Index in Phoenix view on Hbase is not updated
Posted by Mich Talebzadeh <mi...@gmail.com>.
A workout I deployed was to rebuild the index immediately after bulk load
of data into Hbase table
ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 22 October 2016 at 18:25, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Sorry Ted,
>
> This is the syntax for view
>
> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
> "price_info"."price" VARCHAR);
>
> Thanks James for clarification.
>
> My understanding is that when one creates an index on a Phoenix view on an
> Hbase table, a local index file is created in Phoenix with data as is and
> that Phoenix table has no way of knowing if new data is added to Hbase
> table.
>
> From an operational point of view unless index in Phoenix is immediately
> updated after any append to Hbase, then it is pretty useless.
>
> In my case data is inserted to Hbase table. I am just using Phoenix for
> data queries (DQ) as opposed to inserts.
>
> Regards,
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 22 October 2016 at 17:59, James Taylor <ja...@apache.org> wrote:
>
>> Hi Mich,
>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>> data.
>> Thanks,
>> James
>>
>>
>> On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The first statement creates index, not view.
>>>
>>> Can you check ?
>>>
>>> Cheers
>>>
>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I have a Hbase table that is populated via
>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>> > through bulk load ever 15 minutes. This works fine.
>>> >
>>> > In Phoenix I created a view on this table
>>> >
>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>> > "price_info"."timecreated");
>>> >
>>> > This also does what is supposed to do and shows correct count.
>>> >
>>> > I then created an index in Phoenix as below
>>> >
>>> > create index index_dx1 on "marketDataHbase"
>>> > ("price_info"."timecreated","price_info"."ticker",
>>> "price_info"."price");
>>> >
>>> > that showed the records OK at that time. I verified this using explain
>>> >
>>> >
>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>> > "marketDataHbase";
>>> > +---------------------------------------------------------+
>>> > | PLAN |
>>> > +---------------------------------------------------------+
>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
>>> > | SERVER FILTER BY FIRST KEY ONLY |
>>> > | SERVER AGGREGATE INTO SINGLE ROW |
>>> > +---------------------------------------------------------+
>>> >
>>> > Now the issue is that the above does not show new data since build in
>>> Hbase
>>> > table unless I do the following:
>>> >
>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>> "marketDataHbase"
>>> > rebuild;
>>> >
>>> >
>>> > Which is not what an index should do (The covered index should be
>>> > maintained automatically).
>>> > The simple issue is how to overcome this problem?
>>> >
>>> > As I understand the index in Phoenix ia another file independent of the
>>> > original phoenix view so I assume that this index file is not updated
>>> for
>>> > one reason or other?
>>> >
>>> > Thanks
>>>
>>
>
Re: Index in Phoenix view on Hbase is not updated
Posted by Mich Talebzadeh <mi...@gmail.com>.
Sorry Ted,
This is the syntax for view
create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
“price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
"price_info"."price" VARCHAR);
Thanks James for clarification.
My understanding is that when one creates an index on a Phoenix view on an
Hbase table, a local index file is created in Phoenix with data as is and
that Phoenix table has no way of knowing if new data is added to Hbase
table.
From an operational point of view unless index in Phoenix is immediately
updated after any append to Hbase, then it is pretty useless.
In my case data is inserted to Hbase table. I am just using Phoenix for
data queries (DQ) as opposed to inserts.
Regards,
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 22 October 2016 at 17:59, James Taylor <ja...@apache.org> wrote:
> Hi Mich,
> Phoenix indexes are only updated if you use Phoenix APIs to input the data.
> Thanks,
> James
>
>
> On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
>
>> The first statement creates index, not view.
>>
>> Can you check ?
>>
>> Cheers
>>
>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I have a Hbase table that is populated via
>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>> > through bulk load ever 15 minutes. This works fine.
>> >
>> > In Phoenix I created a view on this table
>> >
>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>> > "price_info"."timecreated");
>> >
>> > This also does what is supposed to do and shows correct count.
>> >
>> > I then created an index in Phoenix as below
>> >
>> > create index index_dx1 on "marketDataHbase"
>> > ("price_info"."timecreated","price_info"."ticker",
>> "price_info"."price");
>> >
>> > that showed the records OK at that time. I verified this using explain
>> >
>> >
>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>> > "marketDataHbase";
>> > +---------------------------------------------------------+
>> > | PLAN |
>> > +---------------------------------------------------------+
>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
>> > | SERVER FILTER BY FIRST KEY ONLY |
>> > | SERVER AGGREGATE INTO SINGLE ROW |
>> > +---------------------------------------------------------+
>> >
>> > Now the issue is that the above does not show new data since build in
>> Hbase
>> > table unless I do the following:
>> >
>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
>> > rebuild;
>> >
>> >
>> > Which is not what an index should do (The covered index should be
>> > maintained automatically).
>> > The simple issue is how to overcome this problem?
>> >
>> > As I understand the index in Phoenix ia another file independent of the
>> > original phoenix view so I assume that this index file is not updated
>> for
>> > one reason or other?
>> >
>> > Thanks
>>
>
Re: Index in Phoenix view on Hbase is not updated
Posted by James Taylor <ja...@apache.org>.
Hi Mich,
Phoenix indexes are only updated if you use Phoenix APIs to input the data.
Thanks,
James
On Saturday, October 22, 2016, Ted Yu <yu...@gmail.com> wrote:
> The first statement creates index, not view.
>
> Can you check ?
>
> Cheers
>
> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> <javascript:;>> wrote:
> >
> > Hi,
> >
> > I have a Hbase table that is populated via
> > org.apache.hadoop.hbase.mapreduce.ImportTsv
> > through bulk load ever 15 minutes. This works fine.
> >
> > In Phoenix I created a view on this table
> >
> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
> > "price_info"."timecreated");
> >
> > This also does what is supposed to do and shows correct count.
> >
> > I then created an index in Phoenix as below
> >
> > create index index_dx1 on "marketDataHbase"
> > ("price_info"."timecreated","price_info"."ticker",
> "price_info"."price");
> >
> > that showed the records OK at that time. I verified this using explain
> >
> >
> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
> > "marketDataHbase";
> > +---------------------------------------------------------+
> > | PLAN |
> > +---------------------------------------------------------+
> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
> > | SERVER FILTER BY FIRST KEY ONLY |
> > | SERVER AGGREGATE INTO SINGLE ROW |
> > +---------------------------------------------------------+
> >
> > Now the issue is that the above does not show new data since build in
> Hbase
> > table unless I do the following:
> >
> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
> > rebuild;
> >
> >
> > Which is not what an index should do (The covered index should be
> > maintained automatically).
> > The simple issue is how to overcome this problem?
> >
> > As I understand the index in Phoenix ia another file independent of the
> > original phoenix view so I assume that this index file is not updated for
> > one reason or other?
> >
> > Thanks
>
Re: Index in Phoenix view on Hbase is not updated
Posted by Ted Yu <yu...@gmail.com>.
The first statement creates index, not view.
Can you check ?
Cheers
> On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Hi,
>
> I have a Hbase table that is populated via
> org.apache.hadoop.hbase.mapreduce.ImportTsv
> through bulk load ever 15 minutes. This works fine.
>
> In Phoenix I created a view on this table
>
> jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx on
> "marketDataHbase" ("price_info"."ticker", "price_info"."price",
> "price_info"."timecreated");
>
> This also does what is supposed to do and shows correct count.
>
> I then created an index in Phoenix as below
>
> create index index_dx1 on "marketDataHbase"
> ("price_info"."timecreated","price_info"."ticker", "price_info"."price");
>
> that showed the records OK at that time. I verified this using explain
>
>
> 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
> "marketDataHbase";
> +---------------------------------------------------------+
> | PLAN |
> +---------------------------------------------------------+
> | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1 |
> | SERVER FILTER BY FIRST KEY ONLY |
> | SERVER AGGREGATE INTO SINGLE ROW |
> +---------------------------------------------------------+
>
> Now the issue is that the above does not show new data since build in Hbase
> table unless I do the following:
>
> 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on "marketDataHbase"
> rebuild;
>
>
> Which is not what an index should do (The covered index should be
> maintained automatically).
> The simple issue is how to overcome this problem?
>
> As I understand the index in Phoenix ia another file independent of the
> original phoenix view so I assume that this index file is not updated for
> one reason or other?
>
> Thanks