You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Xi Yang <al...@gmail.com> on 2017/03/07 19:42:39 UTC
What if I want to write another table in MasterObserver.preModifyColumnFamily
?
All the articles I've ever seen are talking about add increment or
change put/get status or pinrt out logs. what if I want to write some data
to another table in Observer? For
example, MasterObserver.preModifyColumnFamily()? Seems Observer is runing
at server side, so use connection is unneccessary and might raise some
problem.
I know this might be a stupid question, so if you can just give some
links let me to learn without explain in email, I will be grateful to you
for your help
Thanks,
Alex
Re: What if I want to write another table in MasterObserver.preModifyColumnFamily
?
Posted by Ted Yu <yu...@gmail.com>.
Zookeeper is supposed to store transient data.
Plus you need to secure the audit log in zookeeper so that unauthorized
user cannot temper the audit.
On Thu, Mar 9, 2017 at 9:24 AM, Xi Yang <al...@gmail.com> wrote:
> Thank you Anoop. After thinking, I give up the idea about write records to
> RS, I will using other ways like zk, or just a text file. I appreciate your
> help!
>
> 2017-03-07 23:36 GMT-08:00 Anoop John <an...@gmail.com>:
>
> > It will be cross server call.. The pre/postModifyColumn is in master
> > side. (MasterObserver) You want to write the info to a table means
> > this will be in some RS. So an RPC request will be needed. But still
> > it is not a case like one RS to another RS cross server call where
> > there is a remote chance of all handlers getting stuck and possible
> > deadlock. Need to carefully done though!
> >
> > -Anoop-
> >
> > On Wed, Mar 8, 2017 at 8:26 AM, Xi Yang <al...@gmail.com> wrote:
> > > Got it. I appreciate your help!
> > >
> > > 2017-03-07 17:05 GMT-08:00 Ted Yu <yu...@gmail.com>:
> > >
> > >> It seems the following hook is better for your use case:
> > >>
> > >> default void postModifyColumn(final
> > >> ObserverContext<MasterCoprocessorEnvironment> ctx,
> > >> TableName tableName, HColumnDescriptor columnFamily) throws
> > >> IOException {}
> > >>
> > >> since there is no guarantee that column family is modified at
> > >> time preModifyColumnFamily() is called.
> > >>
> > >> Cheers
> > >>
> > >> On Tue, Mar 7, 2017 at 4:43 PM, Xi Yang <al...@gmail.com>
> wrote:
> > >>
> > >> > Requirement:
> > >> >
> > >> > I want to record every change of modify columnFamily by using
> > >> > preModifyColumnFamily().
> > >> > Now I have a table "my_ddl_log" which used to record the change of
> > >> > columnFamily. For example:
> > >> >
> > >> > If jack change the TTL of columnFamily "primary" in table
> "employee".
> > >> Then
> > >> > we should add a put to "my_ddl_log" like this record:
> > >> > log:name= 'jack'
> > >> > log:updateTime= '2017-03-07 12:12 GMT-08:00'
> > >> > log:change= ''Change TTL of Table: employee ColumnFamily: primary'
> > >> >
> > >> > I try to use preModifyColumnFamily to do this stuff.
> > >> >
> > >> > Thanks,
> > >> > Alex
> > >> >
> > >> >
> > >> > 2017-03-07 12:12 GMT-08:00 Ted Yu <yu...@gmail.com>:
> > >> >
> > >> > > Describing your use case would allow people to give better answer.
> > >> > >
> > >> > > What kind of data do you write to other table in
> > >> preModifyColumnFamily()
> > >> > ?
> > >> > >
> > >> > > Cross server call within observer is not good idea.
> > >> > >
> > >> > > Take a look at ConnectionUtils.createShortCircuitConnection().
> > >> > >
> > >> > > Cheers
> > >> > >
> > >> > > On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang <al...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > > All the articles I've ever seen are talking about add
> > increment
> > >> or
> > >> > > > change put/get status or pinrt out logs. what if I want to write
> > some
> > >> > > data
> > >> > > > to another table in Observer? For
> > >> > > > example, MasterObserver.preModifyColumnFamily()? Seems Observer
> > is
> > >> > > runing
> > >> > > > at server side, so use connection is unneccessary and might
> raise
> > >> some
> > >> > > > problem.
> > >> > > > I know this might be a stupid question, so if you can just
> give
> > >> some
> > >> > > > links let me to learn without explain in email, I will be
> > grateful to
> > >> > you
> > >> > > > for your help
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Alex
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
Re: What if I want to write another table in MasterObserver.preModifyColumnFamily
?
Posted by Xi Yang <al...@gmail.com>.
Thank you Anoop. After thinking, I give up the idea about write records to
RS, I will using other ways like zk, or just a text file. I appreciate your
help!
2017-03-07 23:36 GMT-08:00 Anoop John <an...@gmail.com>:
> It will be cross server call.. The pre/postModifyColumn is in master
> side. (MasterObserver) You want to write the info to a table means
> this will be in some RS. So an RPC request will be needed. But still
> it is not a case like one RS to another RS cross server call where
> there is a remote chance of all handlers getting stuck and possible
> deadlock. Need to carefully done though!
>
> -Anoop-
>
> On Wed, Mar 8, 2017 at 8:26 AM, Xi Yang <al...@gmail.com> wrote:
> > Got it. I appreciate your help!
> >
> > 2017-03-07 17:05 GMT-08:00 Ted Yu <yu...@gmail.com>:
> >
> >> It seems the following hook is better for your use case:
> >>
> >> default void postModifyColumn(final
> >> ObserverContext<MasterCoprocessorEnvironment> ctx,
> >> TableName tableName, HColumnDescriptor columnFamily) throws
> >> IOException {}
> >>
> >> since there is no guarantee that column family is modified at
> >> time preModifyColumnFamily() is called.
> >>
> >> Cheers
> >>
> >> On Tue, Mar 7, 2017 at 4:43 PM, Xi Yang <al...@gmail.com> wrote:
> >>
> >> > Requirement:
> >> >
> >> > I want to record every change of modify columnFamily by using
> >> > preModifyColumnFamily().
> >> > Now I have a table "my_ddl_log" which used to record the change of
> >> > columnFamily. For example:
> >> >
> >> > If jack change the TTL of columnFamily "primary" in table "employee".
> >> Then
> >> > we should add a put to "my_ddl_log" like this record:
> >> > log:name= 'jack'
> >> > log:updateTime= '2017-03-07 12:12 GMT-08:00'
> >> > log:change= ''Change TTL of Table: employee ColumnFamily: primary'
> >> >
> >> > I try to use preModifyColumnFamily to do this stuff.
> >> >
> >> > Thanks,
> >> > Alex
> >> >
> >> >
> >> > 2017-03-07 12:12 GMT-08:00 Ted Yu <yu...@gmail.com>:
> >> >
> >> > > Describing your use case would allow people to give better answer.
> >> > >
> >> > > What kind of data do you write to other table in
> >> preModifyColumnFamily()
> >> > ?
> >> > >
> >> > > Cross server call within observer is not good idea.
> >> > >
> >> > > Take a look at ConnectionUtils.createShortCircuitConnection().
> >> > >
> >> > > Cheers
> >> > >
> >> > > On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang <al...@gmail.com>
> >> wrote:
> >> > >
> >> > > > All the articles I've ever seen are talking about add
> increment
> >> or
> >> > > > change put/get status or pinrt out logs. what if I want to write
> some
> >> > > data
> >> > > > to another table in Observer? For
> >> > > > example, MasterObserver.preModifyColumnFamily()? Seems Observer
> is
> >> > > runing
> >> > > > at server side, so use connection is unneccessary and might raise
> >> some
> >> > > > problem.
> >> > > > I know this might be a stupid question, so if you can just give
> >> some
> >> > > > links let me to learn without explain in email, I will be
> grateful to
> >> > you
> >> > > > for your help
> >> > > >
> >> > > > Thanks,
> >> > > > Alex
> >> > > >
> >> > >
> >> >
> >>
>
Re: What if I want to write another table in MasterObserver.preModifyColumnFamily
?
Posted by Anoop John <an...@gmail.com>.
It will be cross server call.. The pre/postModifyColumn is in master
side. (MasterObserver) You want to write the info to a table means
this will be in some RS. So an RPC request will be needed. But still
it is not a case like one RS to another RS cross server call where
there is a remote chance of all handlers getting stuck and possible
deadlock. Need to carefully done though!
-Anoop-
On Wed, Mar 8, 2017 at 8:26 AM, Xi Yang <al...@gmail.com> wrote:
> Got it. I appreciate your help!
>
> 2017-03-07 17:05 GMT-08:00 Ted Yu <yu...@gmail.com>:
>
>> It seems the following hook is better for your use case:
>>
>> default void postModifyColumn(final
>> ObserverContext<MasterCoprocessorEnvironment> ctx,
>> TableName tableName, HColumnDescriptor columnFamily) throws
>> IOException {}
>>
>> since there is no guarantee that column family is modified at
>> time preModifyColumnFamily() is called.
>>
>> Cheers
>>
>> On Tue, Mar 7, 2017 at 4:43 PM, Xi Yang <al...@gmail.com> wrote:
>>
>> > Requirement:
>> >
>> > I want to record every change of modify columnFamily by using
>> > preModifyColumnFamily().
>> > Now I have a table "my_ddl_log" which used to record the change of
>> > columnFamily. For example:
>> >
>> > If jack change the TTL of columnFamily "primary" in table "employee".
>> Then
>> > we should add a put to "my_ddl_log" like this record:
>> > log:name= 'jack'
>> > log:updateTime= '2017-03-07 12:12 GMT-08:00'
>> > log:change= ''Change TTL of Table: employee ColumnFamily: primary'
>> >
>> > I try to use preModifyColumnFamily to do this stuff.
>> >
>> > Thanks,
>> > Alex
>> >
>> >
>> > 2017-03-07 12:12 GMT-08:00 Ted Yu <yu...@gmail.com>:
>> >
>> > > Describing your use case would allow people to give better answer.
>> > >
>> > > What kind of data do you write to other table in
>> preModifyColumnFamily()
>> > ?
>> > >
>> > > Cross server call within observer is not good idea.
>> > >
>> > > Take a look at ConnectionUtils.createShortCircuitConnection().
>> > >
>> > > Cheers
>> > >
>> > > On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang <al...@gmail.com>
>> wrote:
>> > >
>> > > > All the articles I've ever seen are talking about add increment
>> or
>> > > > change put/get status or pinrt out logs. what if I want to write some
>> > > data
>> > > > to another table in Observer? For
>> > > > example, MasterObserver.preModifyColumnFamily()? Seems Observer is
>> > > runing
>> > > > at server side, so use connection is unneccessary and might raise
>> some
>> > > > problem.
>> > > > I know this might be a stupid question, so if you can just give
>> some
>> > > > links let me to learn without explain in email, I will be grateful to
>> > you
>> > > > for your help
>> > > >
>> > > > Thanks,
>> > > > Alex
>> > > >
>> > >
>> >
>>
Re: What if I want to write another table in MasterObserver.preModifyColumnFamily
?
Posted by Xi Yang <al...@gmail.com>.
Got it. I appreciate your help!
2017-03-07 17:05 GMT-08:00 Ted Yu <yu...@gmail.com>:
> It seems the following hook is better for your use case:
>
> default void postModifyColumn(final
> ObserverContext<MasterCoprocessorEnvironment> ctx,
> TableName tableName, HColumnDescriptor columnFamily) throws
> IOException {}
>
> since there is no guarantee that column family is modified at
> time preModifyColumnFamily() is called.
>
> Cheers
>
> On Tue, Mar 7, 2017 at 4:43 PM, Xi Yang <al...@gmail.com> wrote:
>
> > Requirement:
> >
> > I want to record every change of modify columnFamily by using
> > preModifyColumnFamily().
> > Now I have a table "my_ddl_log" which used to record the change of
> > columnFamily. For example:
> >
> > If jack change the TTL of columnFamily "primary" in table "employee".
> Then
> > we should add a put to "my_ddl_log" like this record:
> > log:name= 'jack'
> > log:updateTime= '2017-03-07 12:12 GMT-08:00'
> > log:change= ''Change TTL of Table: employee ColumnFamily: primary'
> >
> > I try to use preModifyColumnFamily to do this stuff.
> >
> > Thanks,
> > Alex
> >
> >
> > 2017-03-07 12:12 GMT-08:00 Ted Yu <yu...@gmail.com>:
> >
> > > Describing your use case would allow people to give better answer.
> > >
> > > What kind of data do you write to other table in
> preModifyColumnFamily()
> > ?
> > >
> > > Cross server call within observer is not good idea.
> > >
> > > Take a look at ConnectionUtils.createShortCircuitConnection().
> > >
> > > Cheers
> > >
> > > On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang <al...@gmail.com>
> wrote:
> > >
> > > > All the articles I've ever seen are talking about add increment
> or
> > > > change put/get status or pinrt out logs. what if I want to write some
> > > data
> > > > to another table in Observer? For
> > > > example, MasterObserver.preModifyColumnFamily()? Seems Observer is
> > > runing
> > > > at server side, so use connection is unneccessary and might raise
> some
> > > > problem.
> > > > I know this might be a stupid question, so if you can just give
> some
> > > > links let me to learn without explain in email, I will be grateful to
> > you
> > > > for your help
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > >
> >
>
Re: What if I want to write another table in MasterObserver.preModifyColumnFamily
?
Posted by Ted Yu <yu...@gmail.com>.
It seems the following hook is better for your use case:
default void postModifyColumn(final
ObserverContext<MasterCoprocessorEnvironment> ctx,
TableName tableName, HColumnDescriptor columnFamily) throws
IOException {}
since there is no guarantee that column family is modified at
time preModifyColumnFamily() is called.
Cheers
On Tue, Mar 7, 2017 at 4:43 PM, Xi Yang <al...@gmail.com> wrote:
> Requirement:
>
> I want to record every change of modify columnFamily by using
> preModifyColumnFamily().
> Now I have a table "my_ddl_log" which used to record the change of
> columnFamily. For example:
>
> If jack change the TTL of columnFamily "primary" in table "employee". Then
> we should add a put to "my_ddl_log" like this record:
> log:name= 'jack'
> log:updateTime= '2017-03-07 12:12 GMT-08:00'
> log:change= ''Change TTL of Table: employee ColumnFamily: primary'
>
> I try to use preModifyColumnFamily to do this stuff.
>
> Thanks,
> Alex
>
>
> 2017-03-07 12:12 GMT-08:00 Ted Yu <yu...@gmail.com>:
>
> > Describing your use case would allow people to give better answer.
> >
> > What kind of data do you write to other table in preModifyColumnFamily()
> ?
> >
> > Cross server call within observer is not good idea.
> >
> > Take a look at ConnectionUtils.createShortCircuitConnection().
> >
> > Cheers
> >
> > On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang <al...@gmail.com> wrote:
> >
> > > All the articles I've ever seen are talking about add increment or
> > > change put/get status or pinrt out logs. what if I want to write some
> > data
> > > to another table in Observer? For
> > > example, MasterObserver.preModifyColumnFamily()? Seems Observer is
> > runing
> > > at server side, so use connection is unneccessary and might raise some
> > > problem.
> > > I know this might be a stupid question, so if you can just give some
> > > links let me to learn without explain in email, I will be grateful to
> you
> > > for your help
> > >
> > > Thanks,
> > > Alex
> > >
> >
>
Re: What if I want to write another table in MasterObserver.preModifyColumnFamily
?
Posted by Xi Yang <al...@gmail.com>.
Requirement:
I want to record every change of modify columnFamily by using
preModifyColumnFamily().
Now I have a table "my_ddl_log" which used to record the change of
columnFamily. For example:
If jack change the TTL of columnFamily "primary" in table "employee". Then
we should add a put to "my_ddl_log" like this record:
log:name= 'jack'
log:updateTime= '2017-03-07 12:12 GMT-08:00'
log:change= ''Change TTL of Table: employee ColumnFamily: primary'
I try to use preModifyColumnFamily to do this stuff.
Thanks,
Alex
2017-03-07 12:12 GMT-08:00 Ted Yu <yu...@gmail.com>:
> Describing your use case would allow people to give better answer.
>
> What kind of data do you write to other table in preModifyColumnFamily() ?
>
> Cross server call within observer is not good idea.
>
> Take a look at ConnectionUtils.createShortCircuitConnection().
>
> Cheers
>
> On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang <al...@gmail.com> wrote:
>
> > All the articles I've ever seen are talking about add increment or
> > change put/get status or pinrt out logs. what if I want to write some
> data
> > to another table in Observer? For
> > example, MasterObserver.preModifyColumnFamily()? Seems Observer is
> runing
> > at server side, so use connection is unneccessary and might raise some
> > problem.
> > I know this might be a stupid question, so if you can just give some
> > links let me to learn without explain in email, I will be grateful to you
> > for your help
> >
> > Thanks,
> > Alex
> >
>
Re: What if I want to write another table in MasterObserver.preModifyColumnFamily
?
Posted by Ted Yu <yu...@gmail.com>.
Describing your use case would allow people to give better answer.
What kind of data do you write to other table in preModifyColumnFamily() ?
Cross server call within observer is not good idea.
Take a look at ConnectionUtils.createShortCircuitConnection().
Cheers
On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang <al...@gmail.com> wrote:
> All the articles I've ever seen are talking about add increment or
> change put/get status or pinrt out logs. what if I want to write some data
> to another table in Observer? For
> example, MasterObserver.preModifyColumnFamily()? Seems Observer is runing
> at server side, so use connection is unneccessary and might raise some
> problem.
> I know this might be a stupid question, so if you can just give some
> links let me to learn without explain in email, I will be grateful to you
> for your help
>
> Thanks,
> Alex
>