You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by joseph gao <ga...@gmail.com> on 2015/12/15 03:02:55 UTC
Re: memory usage problem of Metadata.tokenMap.tokenToHost
That's great. Thanks, Alex!
2015-10-02 0:36 GMT+08:00 Alexandre Dutra <al...@datastax.com>:
> I confirm that the optional schema metadata feature (JAVA-828
> <https://datastax-oss.atlassian.net/browse/JAVA-828>) has been merged
> upstream and is included in driver 3.0.0-alpha3 and will as well be
> included in the forthcoming 2.1.8 version.
>
> On a side note, JAVA-664
> <https://datastax-oss.atlassian.net/browse/JAVA-664> has been open for
> quite a while now and aims to fix the problem you describe. It will
> hopefully make it into 2.1.8 as well, although I cannot guarantee that at
> this point.
>
> Thanks,
>
> Alexandre Dutra
>
> On Mon, Sep 28, 2015 at 7:55 PM Alex Popescu <al...@datastax.com> wrote:
>
>> Besides the others' advice that 2000+ keyspaces might be too much, the
>> latest Java driver (2.0.11) includes an option to disable the Metadata API
>> http://www.datastax.com/dev/blog/datastax-java-driver-2-0-11-released.
>> I'm not sure at this moment if this has been merged into 2.1 already.
>>
>> On Sun, Sep 20, 2015 at 9:22 AM, joseph gao <ga...@gmail.com>
>> wrote:
>>
>>> cassandra: 2.1.7
>>> java driver: datastax java driver 2.1.6
>>>
>>> Here is the problem:
>>> My application uses 2000+ keyspaces, and will dynamically create
>>> keyspaces and tables. And then in java client, the
>>> Metadata.tokenMap.tokenToHost would use about 1g memory. so this will cause
>>> a lot of full gc.
>>> As I see, the key of the tokenToHost is keyspace, and the value is a
>>> tokenId_to_replicateNodes map.
>>>
>>> When I try to solve this problem, I find something not sure: all
>>> keyspaces have same 'tokenId_to_replicateNodes' map.
>>> My replication strategy of all keyspaces is : simpleStrategy and
>>> replicationFactor is 3
>>>
>>> So would it be possible if keyspaces use same strategy, the value of
>>> tokenToHost map use a same map. So it would extremely reduce the memory
>>> usage
>>>
>>> thanks a lot
>>>
>>> --
>>> ------
>>> Joseph Gao
>>> PhoneNum:15210513582
>>> QQ: 409343351
>>>
>>
>>
>>
>> --
>> Bests,
>>
>> Alex Popescu | @al3xandru
>> Sen. Product Manager @ DataStax
>>
>>
>>
>> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
>>
> --
> Alexandre Dutra
> Driver & Tools Engineer @ DataStax
>
--
------
Joseph Gao
PhoneNum:15210513582
QQ: 409343351
Re: memory usage problem of Metadata.tokenMap.tokenToHost
Posted by Alex Popescu <al...@datastax.com>.
There are still a couple of tickets being worked on for 2.1.10 and the
release will come afterwards. You can check the list:
https://datastax-oss.atlassian.net/browse/JAVA-989?jql=project%20%3D%20JAVA%20AND%20fixVersion%20%3D%202.1.10
On Mon, Jan 4, 2016 at 5:57 AM, joseph gao <ga...@gmail.com> wrote:
> By the way, when would be the version 2.1.10 published?
>
> 2015-12-15 10:02 GMT+08:00 joseph gao <ga...@gmail.com>:
>
>> That's great. Thanks, Alex!
>>
>> 2015-10-02 0:36 GMT+08:00 Alexandre Dutra <al...@datastax.com>:
>>
>>> I confirm that the optional schema metadata feature (JAVA-828
>>> <https://datastax-oss.atlassian.net/browse/JAVA-828>) has been merged
>>> upstream and is included in driver 3.0.0-alpha3 and will as well be
>>> included in the forthcoming 2.1.8 version.
>>>
>>> On a side note, JAVA-664
>>> <https://datastax-oss.atlassian.net/browse/JAVA-664> has been open for
>>> quite a while now and aims to fix the problem you describe. It will
>>> hopefully make it into 2.1.8 as well, although I cannot guarantee that at
>>> this point.
>>>
>>> Thanks,
>>>
>>> Alexandre Dutra
>>>
>>> On Mon, Sep 28, 2015 at 7:55 PM Alex Popescu <al...@datastax.com> wrote:
>>>
>>>> Besides the others' advice that 2000+ keyspaces might be too much, the
>>>> latest Java driver (2.0.11) includes an option to disable the Metadata API
>>>> http://www.datastax.com/dev/blog/datastax-java-driver-2-0-11-released.
>>>> I'm not sure at this moment if this has been merged into 2.1 already.
>>>>
>>>> On Sun, Sep 20, 2015 at 9:22 AM, joseph gao <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> cassandra: 2.1.7
>>>>> java driver: datastax java driver 2.1.6
>>>>>
>>>>> Here is the problem:
>>>>> My application uses 2000+ keyspaces, and will dynamically create
>>>>> keyspaces and tables. And then in java client, the
>>>>> Metadata.tokenMap.tokenToHost would use about 1g memory. so this will cause
>>>>> a lot of full gc.
>>>>> As I see, the key of the tokenToHost is keyspace, and the value is
>>>>> a tokenId_to_replicateNodes map.
>>>>>
>>>>> When I try to solve this problem, I find something not sure: all
>>>>> keyspaces have same 'tokenId_to_replicateNodes' map.
>>>>> My replication strategy of all keyspaces is : simpleStrategy and
>>>>> replicationFactor is 3
>>>>>
>>>>> So would it be possible if keyspaces use same strategy, the value
>>>>> of tokenToHost map use a same map. So it would extremely reduce the memory
>>>>> usage
>>>>>
>>>>> thanks a lot
>>>>>
>>>>> --
>>>>> ------
>>>>> Joseph Gao
>>>>> PhoneNum:15210513582
>>>>> QQ: 409343351
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Bests,
>>>>
>>>> Alex Popescu | @al3xandru
>>>> Sen. Product Manager @ DataStax
>>>>
>>>>
>>>>
>>>> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
>>>>
>>> --
>>> Alexandre Dutra
>>> Driver & Tools Engineer @ DataStax
>>>
>>
>>
>>
>> --
>> ------
>> Joseph Gao
>> PhoneNum:15210513582
>> QQ: 409343351
>>
>
>
>
> --
> ------
> Joseph Gao
> PhoneNum:15210513582
> QQ: 409343351
>
--
Bests,
Alex Popescu | @al3xandru
Sen. Product Manager @ DataStax
Re: memory usage problem of Metadata.tokenMap.tokenToHost
Posted by joseph gao <ga...@gmail.com>.
By the way, when would be the version 2.1.10 published?
2015-12-15 10:02 GMT+08:00 joseph gao <ga...@gmail.com>:
> That's great. Thanks, Alex!
>
> 2015-10-02 0:36 GMT+08:00 Alexandre Dutra <al...@datastax.com>:
>
>> I confirm that the optional schema metadata feature (JAVA-828
>> <https://datastax-oss.atlassian.net/browse/JAVA-828>) has been merged
>> upstream and is included in driver 3.0.0-alpha3 and will as well be
>> included in the forthcoming 2.1.8 version.
>>
>> On a side note, JAVA-664
>> <https://datastax-oss.atlassian.net/browse/JAVA-664> has been open for
>> quite a while now and aims to fix the problem you describe. It will
>> hopefully make it into 2.1.8 as well, although I cannot guarantee that at
>> this point.
>>
>> Thanks,
>>
>> Alexandre Dutra
>>
>> On Mon, Sep 28, 2015 at 7:55 PM Alex Popescu <al...@datastax.com> wrote:
>>
>>> Besides the others' advice that 2000+ keyspaces might be too much, the
>>> latest Java driver (2.0.11) includes an option to disable the Metadata API
>>> http://www.datastax.com/dev/blog/datastax-java-driver-2-0-11-released.
>>> I'm not sure at this moment if this has been merged into 2.1 already.
>>>
>>> On Sun, Sep 20, 2015 at 9:22 AM, joseph gao <ga...@gmail.com>
>>> wrote:
>>>
>>>> cassandra: 2.1.7
>>>> java driver: datastax java driver 2.1.6
>>>>
>>>> Here is the problem:
>>>> My application uses 2000+ keyspaces, and will dynamically create
>>>> keyspaces and tables. And then in java client, the
>>>> Metadata.tokenMap.tokenToHost would use about 1g memory. so this will cause
>>>> a lot of full gc.
>>>> As I see, the key of the tokenToHost is keyspace, and the value is a
>>>> tokenId_to_replicateNodes map.
>>>>
>>>> When I try to solve this problem, I find something not sure: all
>>>> keyspaces have same 'tokenId_to_replicateNodes' map.
>>>> My replication strategy of all keyspaces is : simpleStrategy and
>>>> replicationFactor is 3
>>>>
>>>> So would it be possible if keyspaces use same strategy, the value
>>>> of tokenToHost map use a same map. So it would extremely reduce the memory
>>>> usage
>>>>
>>>> thanks a lot
>>>>
>>>> --
>>>> ------
>>>> Joseph Gao
>>>> PhoneNum:15210513582
>>>> QQ: 409343351
>>>>
>>>
>>>
>>>
>>> --
>>> Bests,
>>>
>>> Alex Popescu | @al3xandru
>>> Sen. Product Manager @ DataStax
>>>
>>>
>>>
>>> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
>>>
>> --
>> Alexandre Dutra
>> Driver & Tools Engineer @ DataStax
>>
>
>
>
> --
> ------
> Joseph Gao
> PhoneNum:15210513582
> QQ: 409343351
>
--
------
Joseph Gao
PhoneNum:15210513582
QQ: 409343351