You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by nragon <nu...@wedotechnologies.com> on 2017/03/31 15:29:15 UTC
Client near cache with Apache Flink
I'm currently trying to integrate ignite client within apache flink
taskmanagers(jvm) to fast lookup cache. Although my thoughput without ignite
is about 30k/s when i add ignite it drops to 2k/s.
- Cache configuration was set with defaults expect for off-heap.
- I have 3 different caches with one records each (testing purposes).
- I've defined also near cache for clients.
- Both flink and ignite ara managed by yarn.
Is there any better and fastest way to access data? Am I doing something
wrong with the following configurations?
Thanks
------------------ Data access in client
this.nearCache.get(key);
------------------ Cache configuration
CacheConfiguration<String, Object[]> cacheConfig = new
CacheConfiguration<>();
cacheConfig.setName(table.getName());
cacheConfig.setCopyOnRead(false);
cacheConfig.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED);
cacheConfig.setOffHeapMaxMemory(0L);
------------------ Loading from client
IgniteDataStreamer<String, Object[]> streamer =
ignite.dataStreamer(cacheConfig.getName())
------------------ Near cache configuration
NearCacheConfiguration<String, Object[]> nearCacheConfig = new
NearCacheConfiguration<>();
nearCacheConfig.setNearEvictionPolicy(new LruEvictionPolicy(10000));
IgniteCache<String, Object[]> nearCache =
ignite.getOrCreateNearCache("near-" + cacheConfig.getName(),
nearCacheConfig);
------------------ Ignite configuration
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>xx.xxx.xxx.111:47500..47510</value>
<value>xx.xxx.xxx.112:47500..47510</value>
</list>
</property>
</bean>
</property>
<property name="joinTimeout" value="60000"/>
</bean>
</property>
<property name="communicationSpi">
<bean
class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="sharedMemoryPort" value="-1"/>
</bean>
</property>
</bean>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by vkulichenko <va...@gmail.com>.
Both.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11881.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Just one more question :)
Where is data stored when i call put() with near cache? client near cache or
server cache?
Thanks you
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11857.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Hi,
Sorry, i wrote "99% cases will have matching keys unless it's an updatable
lookup", not misses :).
I understand the behaviour of affinity function. Summing up, the problem I
was having is due missing keys which cases the delay. Having thoses keys in
cache should prevent that.
I'll test some iterations with all value on cache and comment the results :)
Thanks you
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11851.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by vkulichenko <va...@gmail.com>.
Primary node is calculated for a particular key, this is done by affinity
function. This function is stateless and does not depend on weather value
exists in cache or not, so the request will always go to the primary node
for requested key. In case you have near cache, and value for requested key
is already there, there will be no network trip.
However, what you provided is not a description of use case. You previously
mentioned that you have 99% misses which makes near cache useless, so I'm
wondering why you're trying to use it in the first place. Most likely there
is a better approach for you, but I'm having hard time understanding your
goals.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11850.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Ok, let's take a use case were i want to lookup an existing value on ignite.
The second time I would request that same value, will it be querying near
cache or primary node? Having the LRU value on near cache will avoid a
network trip to primary node right?
Now, if a key doesn't exist, ignite client will be querying which primary
node(3 servers containing cache values)?
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11844.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by vkulichenko <va...@gmail.com>.
"Real time streaming should not go through network io all the time." - can
you please clarify what exactly you mean by this? For which operations do
you want to avoid io? What is the actual use case scenario? Please describe
it in more detail.
Ignite always sends request only to primary node for a key, never to all
nodes. It does not depend on whether value exists or not.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11843.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Hi,
I think near cache it's the intended behaviour because LRU values, in my
case, will be on each flink task manager. The thing the constraint which
causes the delay is when doesn't exist values on cache. Real time streaming
should not go through network io all the time. What is the normal behaviour
when a cache doen't exist? Ignite will query every primary node? When exists
I know it will only query the primary node which contains the key.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11842.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by vkulichenko <va...@gmail.com>.
Hi Nuno,
First of all, Ignite does not query all nodes when you do a read with near
cache. It first checks locally, and it if value is not available, goes to
primary node.
Second of all, it still sounds like near cache is not what you need and you
chose incorrect approach in the first place. How near cache is supposed to
help with real-time streaming? Can you elaborate more on the use case?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11840.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Hi,
Yes, that was the behaviour I was getting from missing values on cache. With
real time streaming it's important to have near cache, at least in my case I
can't afford to perform network trips all the time. You're right about the
missing values, 99% cases will have matching keys unless it's an updatable
lookup. But still, it should be possible to check metadata before querying
all nodes for that same key. But now I know that's the intended behaviour
and not my implementation.
Thank you all,
Nuno
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11838.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by vkulichenko <va...@gmail.com>.
This seems to be correct behavior. When you read a value through near cache
for the first time, you actually create a new entry in near cache and also
add a reader on the server node (basically it's a node ID that server node
uses to update near cache when value is changed). So near cache does provide
overhead and I believe that it would be too expensive to do the same for
values which do not exist at all (technically should be possible though).
This can cause a problem only if you have too many cache misses, which most
likely means some kind of design issue in your application. Or at least you
don't need near cache as it is not going to help a lot in this case.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11833.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
I think I found the problem.
IgniteTest.java
<http://apache-ignite-users.70518.x6.nabble.com/file/n11826/IgniteTest.java>
Taking this example. If I try to get data from servers with
this.nearCache.get("1"), which is a key on cache, ignite resolves this
within nano seconds, however if a key doesn't exists,
this.nearCache.get("2"), it takes about milliseconds.
I've executed this example with both cases, when the key exists it takes
less than a minute to run 1M iterations, otherwise it scales to 10 MINUTES.
You think it's a bug regarding metadata checking? Because it seems that
client is consulting all nodes when the key doesn't exists.
Thanks for your help.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11826.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Nikolai,
please see IgniteTest.java where I've tested a loop with cache.get() results
were as expected.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11816.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
This time:
this.cCache = new ConcurrentHashMap<>();
this.cCache.put("TEST123", new Object[]{1, 1, 1});
access with test:
this.matchRow = this.cCache.get("TEST123");
access with ignite:
this.matchRow = this.cache.get(key);
but still the same performance, 273k against 35k in 1min
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11815.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by Nikolai Tikhonov <nt...@apache.org>.
> it sould be local access right?
Yes.
> Just like accessing an hashmap onheap?
Like lookup in ConcurrentHashMap. Try to compare with it and check numbers.
On Fri, Apr 7, 2017 at 2:58 PM, nragon <nu...@wedotechnologies.com>
wrote:
> Yes you're right, not random but Object[]{1, 1, 1}. Althought my question
> remains,
> "I know that the second test doesn't require network trips, but still, with
> near cache wouldn't it be require only one trip?"
> After the first request ignite should pull the record from servers into
> ignite client near cache, and from there it sould be local access right?
> Just like accessing an hashmap onheap?. I'm trying to understand why the
> major drop from 378k to 70k.
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11811.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Yes you're right, not random but Object[]{1, 1, 1}. Althought my question
remains,
"I know that the second test doesn't require network trips, but still, with
near cache wouldn't it be require only one trip?"
After the first request ignite should pull the record from servers into
ignite client near cache, and from there it sould be local access right?
Just like accessing an hashmap onheap?. I'm trying to understand why the
major drop from 378k to 70k.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11811.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by Nikolai Tikhonov <nt...@apache.org>.
Your comparison isn't fair. How I understand you comapre IgniteCache#get vs
ThreadLocalRandom.current().nextInt(). Am I right?
On Fri, Apr 7, 2017 at 1:46 PM, nragon <nu...@wedotechnologies.com>
wrote:
> Same topology as before but now it's using this.cache =
> ignite.getOrCreateCache(this.cacheConfig); and this.cache.get(key)
>
>
> Flight recordings:
> flight_recording_10228245113_withignite.jfr
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n11807/flight_recording_10228245113_withignite.jfr>
> flight_recording_10228245113_withoutignite.jfr
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> n11807/flight_recording_10228245113_withoutignite.jfr>
>
> Job with ignite processed 70k whereas without ignite 378k, in 2minutes.
> I know that the second test doesn't require network trips, but still, with
> near cache wouldn't it be require only one trip?
>
> Thanks for the help
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11807.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Same topology as before but now it's using this.cache =
ignite.getOrCreateCache(this.cacheConfig); and this.cache.get(key)
Flight recordings:
flight_recording_10228245113_withignite.jfr
<http://apache-ignite-users.70518.x6.nabble.com/file/n11807/flight_recording_10228245113_withignite.jfr>
flight_recording_10228245113_withoutignite.jfr
<http://apache-ignite-users.70518.x6.nabble.com/file/n11807/flight_recording_10228245113_withoutignite.jfr>
Job with ignite processed 70k whereas without ignite 378k, in 2minutes.
I know that the second test doesn't require network trips, but still, with
near cache wouldn't it be require only one trip?
Thanks for the help
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11807.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by Nikolai Tikhonov <nt...@apache.org>.
Yes, if you don't use SQL then indexes not needed.
On Fri, Apr 7, 2017 at 1:01 PM, nragon <nu...@wedotechnologies.com>
wrote:
> In my case i don't need indexes right? I'm not doing any sql on it, only
> pure
> get(). I will remove near cache and test it again.
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11803.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
In my case i don't need indexes right? I'm not doing any sql on it, only pure
get(). I will remove near cache and test it again.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11803.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by Nikolai Tikhonov <nt...@apache.org>.
Hi,
Let's try to get rid of near cache in your case.
Also you configured the follwing indexed types:
> cacheConfig.setIndexedTypes(String.class, Object.class);
It's wrong way, you need to provide specific type for value. See more
details there [1].
1. https://apacheignite.readme.io/docs/indexes.
On Thu, Apr 6, 2017 at 12:56 PM, nragon <nuno.goncalves@wedotechnologies.com
> wrote:
> Might be related:
> http://apache-ignite-users.70518.x6.nabble.com/Client-
> got-stucked-on-get-operation-td11313.html
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11771.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Might be related:
http://apache-ignite-users.70518.x6.nabble.com/Client-got-stucked-on-get-operation-td11313.html
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11771.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
I think this kind of what i'm doing, in a abstract way as simple as possible
IgniteTest.java
<http://apache-ignite-users.70518.x6.nabble.com/file/n11761/IgniteTest.java>
Here I get this when executing IgniteTest:
gets: 3462000 - seconds past: 15 - rate: 230800 gets/s
If i run the same logic within flink:
gets: 6000 - seconds past: 26 - rate: 230 gets/s
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11761.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
So, when flink as to process the first record it creates a latch telling the
other task managers(jvm) to wait until the cache is loaded.
//
--------------------------------------------------------------------------------------------
Ignite ignite = IgniteClient.ignite();
IgniteCountDownLatch latch =
ignite.countDownLatch(this.cacheConfig.getName() + "-loadlatch", 1, false,
true);
if (!cacheExists(ignite)) {
ignite.getOrCreateCache(this.cacheConfig, this.nearCacheConfig);
load(ignite);
latch.countDown();
}
latch.await();
this.nearCache =
ignite.getOrCreateNearCache(this.cacheConfig.getName(),
this.nearCacheConfig);
} catch (Exception var3) {
System.out.println(var3);
}
//
--------------------------------------------------------------------------------------------
As testing purposes I'm calling the function for lookup but with fixed
Object[] when the key is TEST123
//
--------------------------------------------------------------------------------------------
if(key.contains("TEST123")){
this.matchRow = new Object[]{1, 1, 1};
}
else{
this.matchRow = this.nearCache.get(key);
}
//
--------------------------------------------------------------------------------------------
If I access ignite with this.nearCache.get(key) it drops my throughtput like
i mentioned before.
Let me know if more information is needed.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11754.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by Andrey Gura <ag...@apache.org>.
What is the source of your data in case *without* Ignite cache? I'm
trying to understand your design without Ignite. What does your code
instead of nearCache.get(key)?
Also the code snippet that creates code still should fail because it
try to create near cache for original cache that doesn't exist yet.
On Mon, Apr 3, 2017 at 7:03 PM, nragon
<nu...@wedotechnologies.com> wrote:
> Hi,
>
> The code is not all there but i think it's the important one.
> Cache access: this.nearCache.get(key); (Mentioned above)
> Cache name comes from table.getName() which, for instance, can be
> "LK_SUBS_UPP_INST".
> Data in ignite cache: String key and Object[] value. For now I only have one
> record for each cache (3 total)
> +======================================================================================================================+
> | Key Class | Key | Value Class |
> Value |
> +======================================================================================================================+
> | java.lang.String | 1 | java.lang.Object[] | size=10, values=[1, 1, null,
> null, null, null, null, null, null, null] |
> +----------------------------------------------------------------------------------------------------------------------+
>
> The cache is loaded with datastreamer from oracle database. Again, loading
> is not the problem.
> I'm trying to understand if this (this.nearCache.get(key)) is the best way
> to access for fast lookups.
>
> Thanks
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11670.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Hi,
The code is not all there but i think it's the important one.
Cache access: this.nearCache.get(key); (Mentioned above)
Cache name comes from table.getName() which, for instance, can be
"LK_SUBS_UPP_INST".
Data in ignite cache: String key and Object[] value. For now I only have one
record for each cache (3 total)
+======================================================================================================================+
| Key Class | Key | Value Class |
Value |
+======================================================================================================================+
| java.lang.String | 1 | java.lang.Object[] | size=10, values=[1, 1, null,
null, null, null, null, null, null, null] |
+----------------------------------------------------------------------------------------------------------------------+
The cache is loaded with datastreamer from oracle database. Again, loading
is not the problem.
I'm trying to understand if this (this.nearCache.get(key)) is the best way
to access for fast lookups.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11670.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by Andrey Gura <ag...@apache.org>.
Hi,
it still doesn't describe your use case in details. How exactly do you
use Ignite cache in your system? What kind of data do you cache and
what source of this data in case without Ignite?
Also it looks strange for me that you try to create near cache with
name that doesn't match with original cache name. This code should
fail at runtime.
On Sat, Apr 1, 2017 at 3:49 AM, nragon
<nu...@wedotechnologies.com> wrote:
> Starting with ignite:
> - 4 nodes managed by yarn
> - Node configurations are as above
> - 3 off heap caches with one record each
>
> Flink:
> - 5 task managers with 2 slots each and 2gb
>
> In my current job I'm reading from kafka, mapping some business rules with
> flink map functions and sink to hbase. This job without ignite cache
> processes 30k/s events. When I add ignite on flink map function to enrich
> data with cache.get(key) the performance drops to 2k/s.
> I'm bypassing data enrichment at first test(30k/s, just assigning a random
> value)
> Near cache should pull the one record from cache and from there it should be
> really quick right?
> Just wondering if the configurations above are the correct oned for fast
> lookup caches.
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11636.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by nragon <nu...@wedotechnologies.com>.
Starting with ignite:
- 4 nodes managed by yarn
- Node configurations are as above
- 3 off heap caches with one record each
Flink:
- 5 task managers with 2 slots each and 2gb
In my current job I'm reading from kafka, mapping some business rules with
flink map functions and sink to hbase. This job without ignite cache
processes 30k/s events. When I add ignite on flink map function to enrich
data with cache.get(key) the performance drops to 2k/s.
I'm bypassing data enrichment at first test(30k/s, just assigning a random
value)
Near cache should pull the one record from cache and from there it should be
really quick right?
Just wondering if the configurations above are the correct oned for fast
lookup caches.
Thanks
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11636.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Client near cache with Apache Flink
Posted by vkulichenko <va...@gmail.com>.
Honestly, it's absolutely not clear what you're doing. Can you better
describe your use case? Throughput for which operations are you measuring?
What are you comparing with (i.e. what did you use before Ignite for data
storage)? Give as much information as possible.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Client-near-cache-with-Apache-Flink-tp11627p11634.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.