C# DataStreamer Best Practices

["William.L" <wi...@gmail.com>]

Hi,

We are hitting a performance wall with using the C# thin client PutAsync for
data upload to Ignite. We are in the process of migrating to C#'s thick
client (using client mode) datastreamer. I would appreciate some advice on
some best practices on how to use the thick client:
* Ignite instance -- should we use a single instance within the server
process. It seems like C# is a wrapper around JVM so there is no point to
use multiple instances?
* DataStreamer - I am aware from some posting that it is thread-safe, my
question is whether there are any benefits in trying to share/reuse
DataStreamer instance (e.g. better batching for colocated data) vs using
more DataStreamer instances (more parallelism/connections)?
* Retries and connection failures - there is no mention about connection
failure scenarios or retry setting. Can I assume the DataStreamer (via JVM)
will take care of all the connection failures and reconnecting? 
* Failures - are these surfaced as exceptions from Task returned by
AddData()? Or do I have to use Flush/Close?

Thanks





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# DataStreamer Best Practices

[Pavel Tupitsyn <pt...@apache.org>]

Hi William,

> should we use a single instance within the server process
Yes, use single instance per process in production.
Multiple instances per process are useful for integration tests.

> any benefits in trying to share/reuse DataStreamer instance
DataStreamer is thread-safe and, in most cases,
using it from multiple threads provides better overall throughput.

Using multiple streamers can also provide an improvement,
but consumes more resources. I'd recommend to start with a single
streamer and multiple writer threads.

> Can I assume the DataStreamer (via JVM)
> will take care of all the connection failures and reconnecting
Yes

> Failures - are these surfaced as exceptions from Task returned by
AddData()
Please ignore the Task returned by AddData(), and don't await it.
Multiple AddData calls return the same Task until the buffer is flushed out.
This is a bit of a weird/legacy behavior.

Exceptions will be thrown by the AddData method itself (e.g. in case of a
timeout).

> do I have to use Flush/Close
No need to Flush manually, unless you need current buffered data to be
saved immediately.
Make sure to Close/Dispose the streamer in the end - typically with a C#
"using" block.

On Sat, Apr 24, 2021 at 7:20 AM William.L <wi...@gmail.com> wrote:

> Hi,
>
> We are hitting a performance wall with using the C# thin client PutAsync
> for
> data upload to Ignite. We are in the process of migrating to C#'s thick
> client (using client mode) datastreamer. I would appreciate some advice on
> some best practices on how to use the thick client:
> * Ignite instance -- should we use a single instance within the server
> process. It seems like C# is a wrapper around JVM so there is no point to
> use multiple instances?
> * DataStreamer - I am aware from some posting that it is thread-safe, my
> question is whether there are any benefits in trying to share/reuse
> DataStreamer instance (e.g. better batching for colocated data) vs using
> more DataStreamer instances (more parallelism/connections)?
> * Retries and connection failures - there is no mention about connection
> failure scenarios or retry setting. Can I assume the DataStreamer (via JVM)
> will take care of all the connection failures and reconnecting?
> * Failures - are these surfaced as exceptions from Task returned by
> AddData()? Or do I have to use Flush/Close?
>
> Thanks
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>