RocketMQ support message filtering based on SQL92

[liuxue <li...@gmail.com>]

Hi everyone,

I'm glad that this feature has been merged into develop branch. Please
forgive me to take you some time to introduce it.

*Background*

So far, RocketMQ only support message filtering feature by TAG, but one
message only can own one tag, this is too limited to meet complex business
requirements.

So, we want to define and implement a reasonable filter language based on a
subset of the SQL 92 expression syntax to support customized message
filtering.

*Subset of SQL92*

 Let RocketMQ has the ability of message filtering is the purpose of this
issue, and as we know, SQL92 is used widely and most persons are familiar
with it.It's resonable to select it as RocketMQ's grammar.

As I know, ActiveMQ already impllement this functionality based on JavaCC,
it's simple and exntensible.So I just extract it and integrate into
RocketMQ, only some grammars:

   1. Numeric comparison, like >, >=, <, <=, BETWEEN, =;
   2. Character comparison, like =, <>, IN;
   3. IS NULL or IS NOT NULL;
   4. Logical AND, logical OR, logical NOT;

*Design*

The main logic is:

   1. Broker collects the expression of consumer through heartbeat request,
   and saved in ConsumerFilterManager.
   2. When consumer pulls messages, broker will construct a
   MessageFilter(an interface) with compiled expression and subscription data
   to select matched messages in CommitLog.

But it's poor performance to do filter when pull message,

   1. Off-heap to heap, once every consumer subscribed same topic pull
   message.
   2. Decode message properties, once every consumer subscribed same topic
   pull message.

BloomFilter and pre-calculation are adopted to optimize the situation, it's
suitable for high subscription ratio or large properties.

More details could be found at here
<https://rocketmq.incubator.apache.org/rocketmq/filter-messages-by-sql92-in-rocketmq/>,
and it's only a draft edition, everyone is welcome to join in and improve
it.

Thanks,
Eric.

Re: RocketMQ support message filtering based on SQL92

[dongeforever <10...@qq.com>]

Cool. Thank liuxue's work on documentation, this could be a good example for new features.




------------------ Original ------------------
From:  "liuxue";<li...@gmail.com>;
Date:  Wed, May 3, 2017 08:55 PM
To:  "dev"<de...@rocketmq.incubator.apache.org>; 

Subject:  RocketMQ support message filtering based on SQL92



Hi everyone,

I'm glad that this feature has been merged into develop branch. Please
forgive me to take you some time to introduce it.

*Background*

So far, RocketMQ only support message filtering feature by TAG, but one
message only can own one tag, this is too limited to meet complex business
requirements.

So, we want to define and implement a reasonable filter language based on a
subset of the SQL 92 expression syntax to support customized message
filtering.

*Subset of SQL92*

 Let RocketMQ has the ability of message filtering is the purpose of this
issue, and as we know, SQL92 is used widely and most persons are familiar
with it.It's resonable to select it as RocketMQ's grammar.

As I know, ActiveMQ already impllement this functionality based on JavaCC,
it's simple and exntensible.So I just extract it and integrate into
RocketMQ, only some grammars:

   1. Numeric comparison, like >, >=, <, <=, BETWEEN, =;
   2. Character comparison, like =, <>, IN;
   3. IS NULL or IS NOT NULL;
   4. Logical AND, logical OR, logical NOT;

*Design*

The main logic is:

   1. Broker collects the expression of consumer through heartbeat request,
   and saved in ConsumerFilterManager.
   2. When consumer pulls messages, broker will construct a
   MessageFilter(an interface) with compiled expression and subscription data
   to select matched messages in CommitLog.

But it's poor performance to do filter when pull message,

   1. Off-heap to heap, once every consumer subscribed same topic pull
   message.
   2. Decode message properties, once every consumer subscribed same topic
   pull message.

BloomFilter and pre-calculation are adopted to optimize the situation, it's
suitable for high subscription ratio or large properties.

More details could be found at here
<https://rocketmq.incubator.apache.org/rocketmq/filter-messages-by-sql92-in-rocketmq/>,
and it's only a draft edition, everyone is welcome to join in and improve
it.

Thanks,
Eric.