You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mamaco <ma...@163.com> on 2018/03/24 17:59:07 UTC
how to organize groups in cluster?
assume we a ignite cluster of nodes A,B,C, is it possible to create partion
cache only on node A and node B, and use topic message to replicate data to
node C for real-time query? if not, is it possible to create 2 independant
clusters, cluster1 for node A+node B, cluster2 for node C, and let them
communicate through topic messaging? it looks like a solution for separated
data groups, sometimes I want to isolated write and read processes.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: how to organize groups in cluster?
Posted by mamaco <ma...@163.com>.
Hi Val,
Here's the background:
I'm thinking about creating message pipelines right in ignite cluster.
it's a nightmare to save data in a centralized cache and run everything on
it,
so my whole idea is to guarantee *none-blocking process* and scalable
consumers.
1. (producer) put real-time stream into internal message pipeline
2. (consumer) top N algorithm process
3. (consumer) cross-cache replication
<http://apache-ignite-users.70518.x6.nabble.com/file/t1585/topitem.jpg>
//Topic based message receiver
IgniteMessaging msg = ignite.message(ignite.cluster().forRemotes());
msg.localListen("Item", (nodeId, message) -> {
this.topitems.Process((AdobeItem) message);
return true;
});
//algorithm class
public class AdobeTopViewedItems {
private static Ignite ignite;
private static IgniteCache<String,AdobeItem> cacheItems;
private Map<String, AdobeItem> ListTopN = new LinkedHashMap<String,
AdobeItem>();
private AdobeItem smallest=null;
private Integer TopN=Integer.parseInt(System.getProperty("TopN"));
private static WebsocketUtil socket;
public AdobeTopViewedItems(String accountid) {
ignite=Ignition.ignite();
cacheItems = ignite.cache("Items");
Initialize(accountid);
socket = new WebsocketUtil("s7biapp10",9001,"ItemChannel");
}
public Map<String, AdobeItem> Get(){
return ListTopN;
}
public void Initialize(String accountid){
this.ListTopN=load(accountid);
}
public Map<String, AdobeItem> load(String accountid){
Map<String, AdobeItem> output =new LinkedHashMap<String, AdobeItem>();
String script="from AdobeItem \n"
+ "where \n"
+ "datekey>='"+DateUtil.DateTime2DateString(DateUtil.DateTime2Date(new
Date()))+"' \n"
+ "and accountid='"+accountid+"' \n"
+ "order by visits desc \n"
+ "limit "+Integer.toString(TopN)+" \n";
SqlQuery<String, AdobeItem> sql=new SqlQuery<String,
AdobeItem>(AdobeItem.class, script);
QueryCursor<javax.cache.Cache.Entry<String, AdobeItem>>
cursor=cacheItems.query(sql);
for (javax.cache.Cache.Entry<String, AdobeItem> row : cursor){
AdobeItem entry=row.getValue();
output.put(entry.key(), entry);
}
return output;
}
public void Process(AdobeItem item){
if(this.smallest==null){
this.smallest=item;
this.ListTopN.put(item.key(), item);
this.ListTopN=sort(ListTopN,false);
}else{
if(item.visits>this.smallest.visits){
this.smallest=item;
this.ListTopN.put(item.key(), item);
this.ListTopN=sort(ListTopN,false);
*send(item);*
}
}
}
private void send(AdobeItem item){
String message=new Gson().toJson(item);
socket.send(message, MessageRoute.server2client.toString());
}
private Map<String, AdobeItem> sort(Map<String, AdobeItem> map,Boolean
descending) {
final Boolean is_descending=descending;
List<Entry<String, AdobeItem>> list = new
LinkedList<Entry<String, AdobeItem>>(map.entrySet());
Collections.sort(list, new Comparator<Entry<String, AdobeItem>>() {
public int compare(Entry<String, AdobeItem> o1, Entry<String,
AdobeItem> o2) {
if (is_descending) {
return o2.getValue().visits.compareTo(o1.getValue().visits);
} else {
return
o2.getValue().visits.compareTo(o1.getValue().visits);
}
}
});
Map<String, AdobeItem> sortedMap = new LinkedHashMap<String,
AdobeItem>();
Integer counter=0;
for (Entry<String, AdobeItem> entry : list) {
sortedMap.put(entry.getKey(), entry.getValue());
counter+=1;
smallest=entry.getValue();
if(counter==TopN) break;
}
list.clear();
list=null;
return sortedMap;
}
}
vkulichenko wrote
> Sorry, I'm still failing to understand what you're trying to achieve. What
> is
> the reason to manually maintain a tree structure which is basically an
> index? Why not use Ignite indexes that are provided out of the box?
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: how to organize groups in cluster?
Posted by mamaco <ma...@163.com>.
Hi Andrew,
Thank you for the response.
1. yes, I will try this way to build sub caches
2. yes, make sense.
3. I tested EvictionPolicy, it seemed to be not available in ignite 2.0+,
please refer to my earlier discuss (with denis)
<http://apache-ignite-users.70518.x6.nabble.com/SortedEvictionPolicy-doesn-t-work-as-expected-td20249.html#a20269>
, is there sth else that I need to know?
I'm fighting against none-blocking real-time processes, I hope I could have
better understanding of ignite documents
Andrew Mashenkov wrote
> Hi,
>
> 1. NodeFilter can be used [1] to bound cache to certain data nodes.
> 2. You can subscribe to cache updates via ContinuousQuery [2].
> But keep in mind, ContinuousQuery listener is called from sensitive code
> and it is usually bad idea to make blocking operations in it.
> 3. To keep only TOP N entries in cache, a EvictionPolicy [3] can be used.
> EvictionPolicy doesn't support persistence.
>
> Hope, this helps.
>
> [1]
> https://stackoverflow.com/questions/39612201/apache-ignite-how-to-deploy-cache-to-some-certain-cluster-group
> [2] https://apacheignite.readme.io/docs/continuous-queries
> [3]
> https://apacheignite.readme.io/docs/evictions#section-first-in-first-out-fifo-
>
> On Thu, Mar 29, 2018 at 2:42 AM, vkulichenko <
> valentin.kulichenko@
> >
> wrote:
>
>> Sorry, I'm still failing to understand what you're trying to achieve.
>> What
>> is
>> the reason to manually maintain a tree structure which is basically an
>> index? Why not use Ignite indexes that are provided out of the box?
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: how to organize groups in cluster?
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
1. NodeFilter can be used [1] to bound cache to certain data nodes.
2. You can subscribe to cache updates via ContinuousQuery [2].
But keep in mind, ContinuousQuery listener is called from sensitive code
and it is usually bad idea to make blocking operations in it.
3. To keep only TOP N entries in cache, a EvictionPolicy [3] can be used.
EvictionPolicy doesn't support persistence.
Hope, this helps.
[1]
https://stackoverflow.com/questions/39612201/apache-ignite-how-to-deploy-cache-to-some-certain-cluster-group
[2] https://apacheignite.readme.io/docs/continuous-queries
[3]
https://apacheignite.readme.io/docs/evictions#section-first-in-first-out-fifo-
On Thu, Mar 29, 2018 at 2:42 AM, vkulichenko <va...@gmail.com>
wrote:
> Sorry, I'm still failing to understand what you're trying to achieve. What
> is
> the reason to manually maintain a tree structure which is basically an
> index? Why not use Ignite indexes that are provided out of the box?
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: how to organize groups in cluster?
Posted by vkulichenko <va...@gmail.com>.
Sorry, I'm still failing to understand what you're trying to achieve. What is
the reason to manually maintain a tree structure which is basically an
index? Why not use Ignite indexes that are provided out of the box?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: how to organize groups in cluster?
Posted by mamaco <ma...@163.com>.
Hi Val,
1M per day in memory sales data, 100 parallel financial aggregations in each
batch plus additional PUB/SUB requests for both instant query and server
push. It has been confirmed to play group count distinct is way too slow, so
we decided to play some of TOP N query in separate domain, for example:
Capture a cache event and export to messaging-->client node subscribe to the
TOPIC-->a IN-MEMORY tree set with particular comparator for filtering and
sorting -->publish result once TOP N map is changed.-->update user
visualization
I believe this way won't block cache.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: how to organize groups in cluster?
Posted by vkulichenko <va...@gmail.com>.
Can you clarify what you mean by "real-time query" in this case? Why not just
start node C as a client and run a query from it?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/