BinaryObject and String.intern

[Shawn Du <sh...@neulion.com.cn>]

Hi experts,

 

I am trying to use String.intern to save memory, below is Pseudo code, will
it work? 

 

public class Example

{

   String[] values;

}

 

Map<String,String> fields = new HashMap<String,String>();

fields.put("example",Example.class.getTypeName());

queryEntity.setFields(fields);

 

 

Public class MyEntryProcessor implements
EntryProcessor<String,BinaryObject,Object>

{

Private List<String> values;

 

         public Object process(MutableEntry<String, BinaryObject> entry,
Object... args)

{

         BinaryObjectBuilder builder = ignite.binary().builder("Example");

         Example example = new Example();

    example.values = new String[this.values.size()];

         for(int i=0;i<values.size();i++)

{

 Example.values[i[ = this.values.get(i).intern(); 

}

builder.setField("example",example);

entry.setValue(builder.build());

}

}

 

Thanks

Shawn

Re: 答复: 回复：BinaryObject and String.intern

[dkarachentsev <dk...@gridgain.com>]

Hi Shawn,

BitSet will be marshaled using BitSet.writeObject() serialization
implementation. In other words, if this approach saves your memory in heap,
it will save it in Ignite as well.

If you save collection of strings, Ignite will not check for duplicates as
JVM does, and as mentioned Alexey, there is still no compression supported.
So only you can do is to write algorithms in your side to handle such cases
like your Map<String, BitSet>.

Thanks!



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/BinaryObject-and-String-intern-tp9826p9891.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: 答复: BinaryObject and String.intern

[dkarachentsev <dk...@gridgain.com>]

Actually no, because Ignite internally will store it as a BinaryObject and
will send to other nodes in a binary format as well, where all string fields
will be unmarshaled without intern(). 



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/BinaryObject-and-String-intern-tp9826p9834.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: 答复: 回复：BinaryObject and String.intern

[Alexey Kuznetsov <ak...@apache.org>]

Hi, Shawn.

Some time ago I created issue "Provide some kind of pluggable compression
SPI support" [1]
But it is not implemented yet. May be in Ignie 2.0 there could be some kind
of such API.
See discussion [2]

[1] https://issues.apache.org/jira/browse/IGNITE-3592
[2]
http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-2-0-td10099.html


On Thu, Jan 5, 2017 at 11:01 AM, Shawn Du <sh...@neulion.com.cn> wrote:

> Hi dkarachentsev <dk...@gridgain.com>,
>
>
>
> Suppose I have a String array and array length is 1000. There are many
> duplicated values and only five distinct values in the array.
>
>
>
> We store them using blow ways, which one save memory most?
>
> #1 List<String>
>
> #2 Map<String,BitSet>
>
> In client side, #2 will save memory greatly, but how does it in ignite
> server?
>
> If Ignite stores BitSet as Integer array, it seems that #2 will not save
> so much memory?
>
>
>
> If there any tips to save memory for above case. Thanks in advance.
>
>
>
> Thanks
>
> Shawn
>
>
>
> *发件人:* shawn.du [mailto:shawn.du@neulion.com.cn]
> *发送时间:* 2017年1月3日 18:37
> *收件人:* user@ignite.apache.org
> *抄送:* user@ignite.apache.org
> *主题:* 回复：BinaryObject and String.intern
>
>
>
>
>
> thanks dkarachentsev <dk...@gridgain.com>.
>
>
>
> 在2017年01月03日 18:25，dkarachentsev <dk...@gridgain.com> 写道:
>
> Actually no, because Ignite internally will store it as a BinaryObject and
> will send to other nodes in a binary format as well, where all string
> fields
> will be unmarshaled without intern().
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/BinaryObject-and-String-intern-tp9826p9834.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>


-- 
Alexey Kuznetsov

答复: 回复：BinaryObject and String.intern

[Shawn Du <sh...@neulion.com.cn>]

Hi  <ma...@gridgain.com> dkarachentsev,
 
Suppose I have a String array and array length is 1000. There are many duplicated values and only five distinct values in the array.
 
We store them using blow ways, which one save memory most?
#1 List<String>
#2 Map<String,BitSet>
In client side, #2 will save memory greatly, but how does it in ignite server?
If Ignite stores BitSet as Integer array, it seems that #2 will not save so much memory?
 
If there any tips to save memory for above case. Thanks in advance.
 
Thanks
Shawn
 
发件人: shawn.du [mailto:shawn.du@neulion.com.cn] 
发送时间: 2017年1月3日 18:37
收件人: user@ignite.apache.org
抄送: user@ignite.apache.org
主题: 回复：BinaryObject and String.intern
 
 
thanks  <ma...@gridgain.com> dkarachentsev.
 
在2017年01月03日 18:25， <ma...@gridgain.com> dkarachentsev 写道:
Actually no, because Ignite internally will store it as a BinaryObject and 
will send to other nodes in a binary format as well, where all string fields 
will be unmarshaled without intern(). 



-- 
View this message in context:  <http://apache-ignite-users.70518.x6.nabble.com/BinaryObject-and-String-intern-tp9826p9834.html> http://apache-ignite-users.70518.x6.nabble.com/BinaryObject-and-String-intern-tp9826p9834.html 
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

答复: BinaryObject and String.intern

[Shawn Du <sh...@neulion.com.cn>]

If I don't use binary object and use POJO and never call keepBinary, it
will?

Thanks
Shawn

-----邮件原件-----
发件人: dkarachentsev [mailto:dkarachentsev@gridgain.com] 
发送时间: 2017年1月3日 15:16
收件人: user@ignite.apache.org
主题: Re: BinaryObject and String.intern

This won't give you any benefit, because Strings will be marshaled and
stored in binary format. In other words you'll get a binary copy of your
string, which is managed just like any other object.



--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/BinaryObject-and-String-inter
n-tp9826p9827.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: BinaryObject and String.intern

[dkarachentsev <dk...@gridgain.com>]

This won't give you any benefit, because Strings will be marshaled and stored
in binary format. In other words you'll get a binary copy of your string,
which is managed just like any other object.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/BinaryObject-and-String-intern-tp9826p9827.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.