You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by KR Kumar <kr...@gmail.com> on 2019/09/15 12:49:18 UTC
DataStreamer addData takes lot of time after 500 million writes
Hi all - Why data streamer take lot of time randomly after 500+ million
writes, it frequently and very consistently takes lot of time to finish the
writes and to the extent of 25 to 45 seconds for write. May be its flushing
the data as i have flush frequency set but why not in the beginning and why
only in the end. And also I see heap size going up after sometime and trend
is consistently upwards.
Here is the streamer configuration:
dataStreamer.autoFlushFrequency(10000);
dataStreamer.perNodeBufferSize(32 * 1024);
dataStreamer.perNodeParallelOperations(32);
Not sure if this is of any use but here is the dataStorageConfiguration
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property
name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property
name="initialSize" value="#{512L * 1024 * 1024}"></property>
<property
name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
<property
name="persistenceEnabled" value="true" />
<property
name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
</bean>
</property>
<property name="pageSize"
value="#{4 * 1024}" />
<property
name="storagePath" value="${grid.data}" />
<property name="walPath"
value="${grid.wal}" />
<property
name="walArchivePath" value="${grid.wal}/archive" />
<property name="walMode"
value="BACKGROUND" />
<property
name="walFlushFrequency" value="5000"></property>
</bean>
</property>
Re: DataStreamer addData takes lot of time after 500 million writes
Posted by Павлухин Иван <vo...@gmail.com>.
Hi,
Is the problem specific to DataStreamer? Will the problem disappear if
insert the same data using IgniteCache.put (or putAll batches)?
вс, 15 сент. 2019 г. в 15:49, KR Kumar <kr...@gmail.com>:
>
> Hi all - Why data streamer take lot of time randomly after 500+ million writes, it frequently and very consistently takes lot of time to finish the writes and to the extent of 25 to 45 seconds for write. May be its flushing the data as i have flush frequency set but why not in the beginning and why only in the end. And also I see heap size going up after sometime and trend is consistently upwards.
>
> Here is the streamer configuration:
>
> dataStreamer.autoFlushFrequency(10000);
> dataStreamer.perNodeBufferSize(32 * 1024);
> dataStreamer.perNodeParallelOperations(32);
>
> Not sure if this is of any use but here is the dataStorageConfiguration
>
> <property name="dataStorageConfiguration">
> <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property name="defaultDataRegionConfiguration">
> <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="initialSize" value="#{512L * 1024 * 1024}"></property>
> <property name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
> <property name="persistenceEnabled" value="true" />
> <property name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
> </bean>
> </property>
> <property name="pageSize" value="#{4 * 1024}" />
> <property name="storagePath" value="${grid.data}" />
> <property name="walPath" value="${grid.wal}" />
> <property name="walArchivePath" value="${grid.wal}/archive" />
> <property name="walMode" value="BACKGROUND" />
> <property name="walFlushFrequency" value="5000"></property>
> </bean>
> </property>
--
Best regards,
Ivan Pavlukhin
Re: DataStreamer addData takes lot of time after 500 million writes
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
This would happen if you have any non-inlined indexes. You need to load a
lot of pages for every write and you are running out of offheap.
4G is a tiny offheap.
Regards,
--
Ilya Kasnacheev
вс, 15 сент. 2019 г. в 15:49, KR Kumar <kr...@gmail.com>:
> Hi all - Why data streamer take lot of time randomly after 500+ million
> writes, it frequently and very consistently takes lot of time to finish the
> writes and to the extent of 25 to 45 seconds for write. May be its flushing
> the data as i have flush frequency set but why not in the beginning and why
> only in the end. And also I see heap size going up after sometime and trend
> is consistently upwards.
>
> Here is the streamer configuration:
>
> dataStreamer.autoFlushFrequency(10000);
> dataStreamer.perNodeBufferSize(32 * 1024);
> dataStreamer.perNodeParallelOperations(32);
>
> Not sure if this is of any use but here is the dataStorageConfiguration
>
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property
> name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property
> name="initialSize" value="#{512L * 1024 * 1024}"></property>
> <property
> name="maxSize" value="#{4L * 1024 * 1024 * 1024}" />
> <property
> name="persistenceEnabled" value="true" />
> <property
> name="checkpointPageBufferSize" value="#{2L *1024 * 1024 * 1024}" />
> </bean>
> </property>
> <property name="pageSize"
> value="#{4 * 1024}" />
> <property
> name="storagePath" value="${grid.data}" />
> <property name="walPath"
> value="${grid.wal}" />
> <property
> name="walArchivePath" value="${grid.wal}/archive" />
> <property name="walMode"
> value="BACKGROUND" />
> <property
> name="walFlushFrequency" value="5000"></property>
> </bean>
> </property>
>