You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by neerajbhatt <ne...@gmail.com> on 2017/07/14 08:04:46 UTC
sql query in case of cluster group
Hello All
We have a requirement to set time out for each node so that we can get a
partial result , in case of a time out from a particular node. To solve this
we have created a cluster group for each node and run compute
ClusterGroup cg = cluster.forAttribute("ROLE", group);
computes.add(ignite.compute(cg).withAsync().withTimeout(5000));
with qry.setLocal(true);
Since the caches are partitioned on a id we should get different result from
each cluster group but we are getting same records from different cluster
groups
So does cluster group works in case of sql query like it works in job
execution, service deployment, messaging, events (as given in documentation)
?
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Neeraj,
Looks like queryParallelizm has buggy in ignite-2.0 that already fixed in
master.
Fix will be available in 2.1 version that it coming. Seems, it should be
released in few days.
On Mon, Jul 17, 2017 at 3:32 PM, neerajbhatt <ne...@gmail.com>
wrote:
> Hi Andrey,
>
> we are not using query timeout, because we want to get partial results, in
> case 1 node is slow, but other nodes are faster.
> So, we are using cluster groups.
>
> Yes, just checked. If we don't use queryParallelism, then the query works
> well.. :)
>
> Kindly create a jira if you have same observation
>
> Thanks
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p15000.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: sql query in case of cluster group
Posted by neerajbhatt <ne...@gmail.com>.
Hi Andrey,
we are not using query timeout, because we want to get partial results, in
case 1 node is slow, but other nodes are faster.
So, we are using cluster groups.
Yes, just checked. If we don't use queryParallelism, then the query works
well.. :)
Kindly create a jira if you have same observation
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p15000.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Posted by Andrey Mashenkov <an...@gmail.com>.
You can use SqlFieldsQuery.setTimeout() for this.
Would you please check if it is workable without queryParalellism?
Looks weird, I need a time to make a test and check if local queries with
queryParalellism>1 is broken.
On Mon, Jul 17, 2017 at 1:44 PM, neerajbhatt <ne...@gmail.com>
wrote:
> We are using cluster groups to run query on each nodes of the cluster so
> that
> we can set the timeout on each query.
> We are running local queries and in TestIgniteCallable
> we have given List<List<?>> res =
> cache.query(qry.setArgs().setLocal(true)).getAll();
>
> What are we missing ?
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14998.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: sql query in case of cluster group
Posted by neerajbhatt <ne...@gmail.com>.
We are using cluster groups to run query on each nodes of the cluster so that
we can set the timeout on each query.
We are running local queries and in TestIgniteCallable
we have given List<List<?>> res =
cache.query(qry.setArgs().setLocal(true)).getAll();
What are we missing ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14998.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Posted by Andrey Mashenkov <an...@gmail.com>.
1. LIMIT make no sense without ORDER for non-local queries and when
sqlQueryParallelizm > 1.
As in that case, data from different index segments will be merged out of
order and you can get different results with same query from run to run .
2. You use ClusterGroup to make your jobs to be run on certain nodes, but
it doesn't affect operations inside a job.
So, non-local sql query will be run on all data nodes.
On Mon, Jul 17, 2017 at 12:44 PM, neerajbhatt <ne...@gmail.com>
wrote:
> Hi Andrew
>
> It seems to be an order by issue. We have created a test project in github
> https://github.com/neerajbhatt/testIgnite
>
> If we use PutItems to put some records in a partitioned cache(TESTCACHE)
> args[0]=number of records, and then Get the items by
> GetItemsWithClusterGroup args[0] = query type (0 or 1), args[1]= number of
> queries and args [3] as last digit of ip.
>
> for first query ("SELECT T.id FROM " + "TESTCache.TestItem as T LIMIT 3")
> different ids are given for different cluster groups( each cluster group
> having one node)
>
> for second query ("SELECT T.id FROM " + "TESTCache.TestItem as T order by
> field1 desc LIMIT 3") we are getting same ids for each cluster group. It
> seems in case of order by instead of cluster group data is being fetched
> from whole cluster.
>
> Please suggest
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14994.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: sql query in case of cluster group
Posted by neerajbhatt <ne...@gmail.com>.
Hi Andrew
It seems to be an order by issue. We have created a test project in github
https://github.com/neerajbhatt/testIgnite
If we use PutItems to put some records in a partitioned cache(TESTCACHE)
args[0]=number of records, and then Get the items by
GetItemsWithClusterGroup args[0] = query type (0 or 1), args[1]= number of
queries and args [3] as last digit of ip.
for first query ("SELECT T.id FROM " + "TESTCache.TestItem as T LIMIT 3")
different ids are given for different cluster groups( each cluster group
having one node)
for second query ("SELECT T.id FROM " + "TESTCache.TestItem as T order by
field1 desc LIMIT 3") we are getting same ids for each cluster group. It
seems in case of order by instead of cluster group data is being fetched
from whole cluster.
Please suggest
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14994.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Posted by Andrey Mashenkov <an...@gmail.com>.
I don't think it is ORDER BY issue.
Are you sure you run queries locally and on different nodes?
On Fri, Jul 14, 2017 at 3:52 PM, neerajbhatt <ne...@gmail.com>
wrote:
> Hi Andrew
>
> Thanks for your inputs now we are not getting duplicate records as we are
> now using different cache and ignite objects for each compute.
>
> But for queries which have order by (sort by) clause we are getting exactly
> same result from all cluster groups(nodes). It looks in case of order by
> the
> query is getting executed in all nodes
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14906.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: sql query in case of cluster group
Posted by neerajbhatt <ne...@gmail.com>.
Hi Andrew
Thanks for your inputs now we are not getting duplicate records as we are
now using different cache and ignite objects for each compute.
But for queries which have order by (sort by) clause we are getting exactly
same result from all cluster groups(nodes). It looks in case of order by the
query is getting executed in all nodes
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14906.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Posted by Andrey Mashenkov <an...@gmail.com>.
Here you use same cache object from all computes.
>> cache.query(qry.setArgs(args.toArray()).setLocal(true)).getAll();
So, actually, you run query on same cache node from threads belongs to
other nodes.
This will not work in multi JVM env.
Just keep in mind that your task\job object can be transfered to another
node (JVM or machine).
You should get cache instance inside task [1].
[1] https://apacheignite.readme.io/docs/resource-injection
On Fri, Jul 14, 2017 at 2:10 PM, neerajbhatt <ne...@gmail.com>
wrote:
> Any update ?, Also we have data as back up as one. In case of back ups can
> we
> get duplicate result as we might be getting result from both master and
> slave
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14903.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov
Re: sql query in case of cluster group
Posted by neerajbhatt <ne...@gmail.com>.
Any update ?, Also we have data as back up as one. In case of back ups can we
get duplicate result as we might be getting result from both master and
slave
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14903.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Posted by neerajbhatt <ne...@gmail.com>.
--code is as below
--all caches are partitioned
--args[0] is type of query, args[1] is number of queries, args[2] is ip of
cluster group (each node is a separate cluster group. group name is
'i-<ipaddress>'
--we are trying with different args[2] (cluster groups) and we are getting
duplicate result. Web console shows that a particular id is in only in one
node (hence data partition and collocation is working fine(
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteCluster;
import org.apache.ignite.IgniteCompute;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.query.SqlFieldsQuery;
import org.apache.ignite.cluster.ClusterGroup;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.lang.IgniteCallable;
import org.apache.ignite.lang.IgniteFuture;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
public class GetItemsWithClusterGroup {
public static void main(String args[]) {
//args[0] is type of query (0,1,2), args[1] is number of queries, args[2]
is ip of cluster group
//we are trying with different args[2] and we are getting duplicate result
int queryType = Integer.parseInt(args[0]);
int maxNumberOfQueries = Integer.parseInt(args[1]);
Ignition.setClientMode(true);
IgniteConfiguration conf = new IgniteConfiguration();
conf.setPeerClassLoadingEnabled(true);
TcpDiscoverySpi discovery = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
String ipaddress = args[2];
ipFinder.setAddresses(Arrays.asList(ipaddress));
discovery.setIpFinder(ipFinder);
conf.setDiscoverySpi(discovery);
Ignite ignite = Ignition.start(conf);
System.out.println("IP=> " + ipaddress);
IgniteCluster cluster = ignite.cluster();
List<String> groups = Arrays.asList("i-"+ipaddress);
List<IgniteCompute> computes = new ArrayList<>();
IgniteCache<Integer, Item> itemCache =
cluster.ignite().getOrCreateCache("ITEMCACHE");
for (String group : groups) {
ClusterGroup cg = cluster.forAttribute("ROLE", group);
System.out.println(group + " " + cg.nodes());
computes.add(ignite.compute(cg).withAsync().withTimeout(5000));
}
int numberOfQueries = 0;
while (numberOfQueries < maxNumberOfQueries) {
numberOfQueries++;
process(ignite, computes, itemCache, queryType);
}
}
private static void process(Ignite ignite, List<IgniteCompute> computes,
IgniteCache<Integer, Item> cache,
int queryType) {
List<IgniteFuture<List<String>>> futures = new ArrayList<>();
String geoId = "144";
StringBuilder buff = new StringBuilder();
final List args = new ArrayList<>();
if (queryType == 0) {
buff = new StringBuilder("SELECT T.uniqueSkuId FROM " + "ITEMCACHE.Item
as T LIMIT 2");
}
if (queryType == 1) {
buff = new StringBuilder(
"SELECT r.id FROM IGPCACHE.Rank as r " + " WHERE r.geoId=? " + "
order by r.rank desc limit 3");
args.add(geoId);
}
if (queryType == 2) {
buff = new StringBuilder("SELECT r.id FROM IGPCACHE.Rank as r " + "
WHERE r.geoId=? " + " limit 3");
args.add(geoId);
}
if (queryType == 3) {
buff = new StringBuilder("SELECT r.id FROM IGPCACHE.Rank as r " + "
limit 3");
}
SqlFieldsQuery qry = new SqlFieldsQuery(buff.toString());
qry.setEnforceJoinOrder(true);
for (IgniteCompute async : computes) {
async.call(new IgniteCallable<List<String>>() {
@Override
public List<String> call() throws Exception {
try {
List<List<?>> res =
cache.query(qry.setArgs(args.toArray()).setLocal(true)).getAll();
List<String> items = convert(res);
return items;
} catch (Exception e) {
System.out.println("exceptionx: " + e.getMessage());
e.printStackTrace();
}
return null;
}
private List<String> convert(List<List<?>> res) {
List<String> items = new ArrayList<>();
if (res != null) {
for (List<?> list : res) {
items.addAll((Collection<? extends String>) list);
}
}
return items;
}
});
IgniteFuture<List<String>> future = async.future();
futures.add(future);
}
List<String> list = new ArrayList<>();
for (IgniteFuture<List<String>> future : futures) {
try {
List<String> returnList = future.get(500, TimeUnit.MILLISECONDS);
// System.out.println(" returning list "+returnList);
if (returnList != null)
list.addAll(returnList);
} catch (Exception e) {
System.out.println("got exception1 " + e.getMessage());
e.printStackTrace();
}
}
//displaying output list
System.out.println(list.size() + "#" + list);
}
}
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884p14891.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: sql query in case of cluster group
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Sound weird. Query qry.setLocal(true) should return local query result.
Would you please share a code?
On Fri, Jul 14, 2017 at 11:04 AM, neerajbhatt <ne...@gmail.com>
wrote:
> Hello All
>
> We have a requirement to set time out for each node so that we can get a
> partial result , in case of a time out from a particular node. To solve
> this
> we have created a cluster group for each node and run compute
>
> ClusterGroup cg = cluster.forAttribute("ROLE", group);
> computes.add(ignite.compute(cg).withAsync().withTimeout(5000));
>
> with qry.setLocal(true);
>
> Since the caches are partitioned on a id we should get different result
> from
> each cluster group but we are getting same records from different cluster
> groups
>
> So does cluster group works in case of sql query like it works in job
> execution, service deployment, messaging, events (as given in
> documentation)
> ?
>
> Thanks
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/sql-query-in-case-of-cluster-group-tp14884.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov