Igfs - questions and optimal configuration settings

[matt <go...@gmail.com>]

Hi,

I've recently started using the Ignite FileSystem (igfs) in our API, to
fully buffer an incoming stream of byte[] values (chunked InputStream). I'm
doing this because that stream then needs to be sent along to another remote
service, and I'd like the ability to retry without telling the sender to
send again. The thinking is that if this all gets "buffered" into Ignite,
then pulling the "file" out again and sending/retrying should be possible
and present no burden on the original sender. After the file has been
successfully sent, it is then deleted from Ignite -- this all seems to work,
however, is there a better way?

If this approach is a good one, I have questions on how to configure. I had
to look around quite a bit to get a working configuration (version 2.3) and
even now, I'm not clear as to what is needed in order to get a good
configuration setup, based on environment/memory/hardware etc.. Is it OK to
just use the default settings?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Igfs - questions and optimal configuration settings

[matt <go...@gmail.com>]

Thanks Denis! That helps a lot. I'll dig into those settings and see if I get
my head around it all. I did notice that the default/max memory settings are
based off of system settings/resources, so I'll try the defaults too and see
what happens.

- Matt



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Igfs - questions and optimal configuration settings

[Denis Mekhanikov <dm...@gmail.com>]

You plan sounds legit.

Basically, Ignite splits your files into chunks of data and puts them into
an ordinary distributed cache.

You can find general recommendation on configuring the Ignite cluster in
the documentation: https://apacheignite.readme.io/docs/

Regarding IGFS: you are going to need to configure data region size for
IGFS cache.
You can either configure size of the default data region, or specify a
cache name in *FileSystemConfiguration#dataCacheName* property and
configure data region for this cache separately.
More on data region configuration:
https://apacheignite.readme.io/docs/memory-configuration

You can also try to tune IGFS-specific parameters, that are available in
the *FileSystemConfiguration
<https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/FileSystemConfiguration.html>
*
class.

Denis

вт, 26 июн. 2018 г. в 17:41, matt <go...@gmail.com>:

> Hi,
>
> I've recently started using the Ignite FileSystem (igfs) in our API, to
> fully buffer an incoming stream of byte[] values (chunked InputStream). I'm
> doing this because that stream then needs to be sent along to another
> remote
> service, and I'd like the ability to retry without telling the sender to
> send again. The thinking is that if this all gets "buffered" into Ignite,
> then pulling the "file" out again and sending/retrying should be possible
> and present no burden on the original sender. After the file has been
> successfully sent, it is then deleted from Ignite -- this all seems to
> work,
> however, is there a better way?
>
> If this approach is a good one, I have questions on how to configure. I had
> to look around quite a bit to get a working configuration (version 2.3) and
> even now, I'm not clear as to what is needed in order to get a good
> configuration setup, based on environment/memory/hardware etc.. Is it OK to
> just use the default settings?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>