You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Айсина Роза Мунеровна <ro...@sbermarket.ru> on 2022/12/26 08:30:49 UTC

Method get_all does not work in Python

Hola!

I can’t use method get_all for caches from PyIgnite client.
The problem is that keys are some binary strings that I do not know how to parse.

Here what I do:


class ProductFeaturesKey(
    metaclass=GenericObjectMeta,
    schema={
        'product_sku': IntObject
    },
    type_name='io.sbmt.ProductFeaturesKey'
):
    pass

…

keys = [ProductFeaturesKey(product_sku=15052004),
        ProductFeaturesKey(product_sku=15058148)]

res = cache.get_all(keys)



>> res
{
(b'g\x01\x0b\x00\xa3\xb8r&%&\xd1\x01"\x00\x00\x00>8_Q\x1d\x00\x00\x00\x03\xe4\xac\xe5\x00\xcd\xa9\x1c\xa7\x18', 0): IoSbmtProductFeaturesValue(sku_mean_order_weight=10531.3330078125, sku_mean_order_quantity=21.0, sku_mean_order_item_count=15.333333015441895, total_cnt_orders_with_sku=12, total_cnt_retailers_with_sku=1, total_cnt_stores_with_sku=3, version=1),

(b'g\x01\x0b\x00\xa3\xb8r&=\x80\xd1\x01"\x00\x00\x00>8_Q\x1d\x00\x00\x00\x03\xe4\xc4\xe5\x00\xcd\xa9\x1c\xa7\x18', 0): IoSbmtProductFeaturesValue(sku_mean_order_weight=9669.0, sku_mean_order_quantity=26.33333396911621, sku_mean_order_item_count=20.33333396911621, total_cnt_orders_with_sku=3, total_cnt_retailers_with_sku=2, total_cnt_stores_with_sku=3, version=1)
}



>> res[keys[0]]
...
TypeError: unhashable type: 'ProductFeaturesKey'



In the resulting dict the keys are binary strings which is odd.
And as I see possible reason can be that original key instances are not hashable
(see GenericObjectMeta definition: https://github.com/apache/ignite-python-thin-client/blob/master/pyignite/binary.py#L220 ).

Is there any way to extract several keys and be able to parse resulting keys to object?
Single cache.get(key) method works too slow (about ~10ms on 400RPS) in the key loop, we want to extract all keys by one call.

Thanks! Please help.

Ignite version: 2.13.0
Pyignite version: 0.5.2

--

Роза Айсина

Старший разработчик ПО

СберМаркет | Доставка из любимых магазинов



Email: roza.aysina@sbermarket.ru<ma...@sbermarket.ru>

Mob:

Web: sbermarket.ru<https://sbermarket.ru/>

App: iOS<https://apps.apple.com/ru/app/%D1%81%D0%B1%D0%B5%D1%80%D0%BC%D0%B0%D1%80%D0%BA%D0%B5%D1%82-%D0%B4%D0%BE%D1%81%D1%82%D0%B0%D0%B2%D0%BA%D0%B0-%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D0%BE%D0%B2/id1166642457> и Android<https://play.google.com/store/apps/details?id=ru.instamart&hl=en&gl=ru>



УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: это электронное сообщение и любые документы, приложенные к нему, содержат конфиденциальную информацию. Настоящим уведомляем Вас о том, что, если это сообщение не предназначено Вам, использование, копирование, распространение информации, содержащейся в настоящем сообщении, а также осуществление любых действий на основе этой информации, строго запрещено. Если Вы получили это сообщение по ошибке, пожалуйста, сообщите об этом отправителю по электронной почте и удалите это сообщение.
CONFIDENTIALITY NOTICE: This email and any files attached to it are confidential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email.

Re: Method get_all does not work in Python

Posted by Айсина Роза Мунеровна <ro...@sbermarket.ru>.
Hi Ivan!

Thank you very much for your reply!
And explanations :)

Function you wrote works like a charm.

On 26 Dec 2022, at 6:59 PM, Ivan Daschinsky <iv...@gmail.com> wrote:

Внимание: Внешний отправитель!
Если вы не знаете отправителя - не открывайте вложения, не переходите по ссылкам, не пересылайте письмо!

Also, please use pyignite 0.6.0 if possible.

пн, 26 дек. 2022 г. в 18:58, Ivan Daschinsky <iv...@gmail.com>>:
I perfectly understand that the API is awful, but there is no way to change it without breaking backward compatibility.

It is impossible to make classes with the metaclass GeneralObjectMeta hashable. Firstly, these classes are mutable.
Secondly, their hashcode can be calculated properly only with meta registry and possibly with network requests.

Possible solution is to return not a dictionary, but a list of kv tuples. But it will also break backward compatibility.
I don't understand why this variant was not chosen.

пн, 26 дек. 2022 г. в 18:52, Ivan Daschinsky <iv...@gmail.com>>:
Nope, unfotunately there is not any built in method. But it seems that this is not necessary at all to do this?


def wrap_binary(client, value) :
    with BinaryStream(client) as stream:
        BinaryObject.from_python(stream, value)
        return (stream.getvalue(), 0)

for k in data.keys():
    key = client.unwrap_binary(k)

    print(key)
    print(data[wrap_binary(client, key)])


--
Sincerely yours, Ivan Daschinskiy


--
Sincerely yours, Ivan Daschinskiy


--

Роза Айсина

Старший разработчик ПО

СберМаркет | Доставка из любимых магазинов



Email: roza.aysina@sbermarket.ru<ma...@sbermarket.ru>

Mob:

Web: sbermarket.ru<https://sbermarket.ru/>

App: iOS<https://apps.apple.com/ru/app/%D1%81%D0%B1%D0%B5%D1%80%D0%BC%D0%B0%D1%80%D0%BA%D0%B5%D1%82-%D0%B4%D0%BE%D1%81%D1%82%D0%B0%D0%B2%D0%BA%D0%B0-%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D0%BE%D0%B2/id1166642457> и Android<https://play.google.com/store/apps/details?id=ru.instamart&hl=en&gl=ru>



УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: это электронное сообщение и любые документы, приложенные к нему, содержат конфиденциальную информацию. Настоящим уведомляем Вас о том, что, если это сообщение не предназначено Вам, использование, копирование, распространение информации, содержащейся в настоящем сообщении, а также осуществление любых действий на основе этой информации, строго запрещено. Если Вы получили это сообщение по ошибке, пожалуйста, сообщите об этом отправителю по электронной почте и удалите это сообщение.
CONFIDENTIALITY NOTICE: This email and any files attached to it are confidential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email.

Re: Method get_all does not work in Python

Posted by Ivan Daschinsky <iv...@gmail.com>.
Also, please use pyignite 0.6.0 if possible.

пн, 26 дек. 2022 г. в 18:58, Ivan Daschinsky <iv...@gmail.com>:

> I perfectly understand that the API is awful, but there is no way to
> change it without breaking backward compatibility.
>
> It is impossible to make classes with the metaclass GeneralObjectMeta
> hashable. Firstly, these classes are mutable.
> Secondly, their hashcode can be calculated properly only with meta
> registry and possibly with network requests.
>
> Possible solution is to return not a dictionary, but a list of kv tuples.
> But it will also break backward compatibility.
> I don't understand why this variant was not chosen.
>
> пн, 26 дек. 2022 г. в 18:52, Ivan Daschinsky <iv...@gmail.com>:
>
>> Nope, unfotunately there is not any built in method. But it seems that
>> this is not necessary at all to do this?
>>
>> def wrap_binary(client, value) :
>>     with BinaryStream(client) as stream:
>>         BinaryObject.from_python(stream, value)
>>         return (stream.getvalue(), 0)
>>
>> for k in data.keys():
>>     key = client.unwrap_binary(k)
>>
>>     print(key)
>>     print(data[wrap_binary(client, key)])
>>
>>
>
> --
> Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Method get_all does not work in Python

Posted by Ivan Daschinsky <iv...@gmail.com>.
I perfectly understand that the API is awful, but there is no way to change
it without breaking backward compatibility.

It is impossible to make classes with the metaclass GeneralObjectMeta
hashable. Firstly, these classes are mutable.
Secondly, their hashcode can be calculated properly only with meta registry
and possibly with network requests.

Possible solution is to return not a dictionary, but a list of kv tuples.
But it will also break backward compatibility.
I don't understand why this variant was not chosen.

пн, 26 дек. 2022 г. в 18:52, Ivan Daschinsky <iv...@gmail.com>:

> Nope, unfotunately there is not any built in method. But it seems that
> this is not necessary at all to do this?
>
> def wrap_binary(client, value) :
>     with BinaryStream(client) as stream:
>         BinaryObject.from_python(stream, value)
>         return (stream.getvalue(), 0)
>
> for k in data.keys():
>     key = client.unwrap_binary(k)
>
>     print(key)
>     print(data[wrap_binary(client, key)])
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: Method get_all does not work in Python

Posted by Ivan Daschinsky <iv...@gmail.com>.
Nope, unfotunately there is not any built in method. But it seems that this
is not necessary at all to do this?

def wrap_binary(client, value) :
    with BinaryStream(client) as stream:
        BinaryObject.from_python(stream, value)
        return (stream.getvalue(), 0)

for k in data.keys():
    key = client.unwrap_binary(k)

    print(key)
    print(data[wrap_binary(client, key)])

Re: Method get_all does not work in Python

Posted by Айсина Роза Мунеровна <ro...@sbermarket.ru>.
Hi Ivan!

Thank you!

Is there any way to access this dict by key?

We have 8 caches and there is no way to iterate over them all simultaneously, so I want to iterate over one of them and access others by key.
Is there some kind of .wrap_binary(key_obj) method?

On 26 Dec 2022, at 4:40 PM, Ivan Daschinsky <iv...@gmail.com> wrote:

Внимание: Внешний отправитель!
Если вы не знаете отправителя - не открывайте вложения, не переходите по ссылкам, не пересылайте письмо!

You can use unwrap_binary:


data = cache.get_all([Key(id=10)])
for k,v in data.items():
    print(client.unwrap_binary(k))
    print(v)

пн, 26 дек. 2022 г. в 11:31, Айсина Роза Мунеровна <ro...@sbermarket.ru>>:
Hola!

I can’t use method get_all for caches from PyIgnite client.
The problem is that keys are some binary strings that I do not know how to parse.

Here what I do:


class ProductFeaturesKey(
    metaclass=GenericObjectMeta,
    schema={
        'product_sku': IntObject
    },
    type_name='io.sbmt.ProductFeaturesKey'
):
    pass

…

keys = [ProductFeaturesKey(product_sku=15052004),
        ProductFeaturesKey(product_sku=15058148)]

res = cache.get_all(keys)



>> res
{
(b'g\x01\x0b\x00\xa3\xb8r&%&\xd1\x01"\x00\x00\x00>8_Q\x1d\x00\x00\x00\x03\xe4\xac\xe5\x00\xcd\xa9\x1c\xa7\x18', 0): IoSbmtProductFeaturesValue(sku_mean_order_weight=10531.3330078125, sku_mean_order_quantity=21.0, sku_mean_order_item_count=15.333333015441895, total_cnt_orders_with_sku=12, total_cnt_retailers_with_sku=1, total_cnt_stores_with_sku=3, version=1),

(b'g\x01\x0b\x00\xa3\xb8r&=\x80\xd1\x01"\x00\x00\x00>8_Q\x1d\x00\x00\x00\x03\xe4\xc4\xe5\x00\xcd\xa9\x1c\xa7\x18', 0): IoSbmtProductFeaturesValue(sku_mean_order_weight=9669.0, sku_mean_order_quantity=26.33333396911621, sku_mean_order_item_count=20.33333396911621, total_cnt_orders_with_sku=3, total_cnt_retailers_with_sku=2, total_cnt_stores_with_sku=3, version=1)
}



>> res[keys[0]]
...
TypeError: unhashable type: 'ProductFeaturesKey'



In the resulting dict the keys are binary strings which is odd.
And as I see possible reason can be that original key instances are not hashable
(see GenericObjectMeta definition: https://github.com/apache/ignite-python-thin-client/blob/master/pyignite/binary.py#L220 ).

Is there any way to extract several keys and be able to parse resulting keys to object?
Single cache.get(key) method works too slow (about ~10ms on 400RPS) in the key loop, we want to extract all keys by one call.

Thanks! Please help.

Ignite version: 2.13.0
Pyignite version: 0.5.2
--

Роза Айсина
Старший разработчик ПО
СберМаркет | Доставка из любимых магазинов



Email: roza.aysina@sbermarket.ru<ma...@sbermarket.ru>
Mob:
Web: sbermarket.ru<https://sbermarket.ru/>
App: iOS<https://apps.apple.com/ru/app/%D1%81%D0%B1%D0%B5%D1%80%D0%BC%D0%B0%D1%80%D0%BA%D0%B5%D1%82-%D0%B4%D0%BE%D1%81%D1%82%D0%B0%D0%B2%D0%BA%D0%B0-%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D0%BE%D0%B2/id1166642457> и Android<https://play.google.com/store/apps/details?id=ru.instamart&hl=en&gl=ru>








УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: это электронное сообщение и любые документы, приложенные к нему, содержат конфиденциальную информацию. Настоящим уведомляем Вас о том, что, если это сообщение не предназначено Вам, использование, копирование, распространение информации, содержащейся в настоящем сообщении, а также осуществление любых действий на основе этой информации, строго запрещено. Если Вы получили это сообщение по ошибке, пожалуйста, сообщите об этом отправителю по электронной почте и удалите это сообщение.
CONFIDENTIALITY NOTICE: This email and any files attached to it are confidential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email.


--
Sincerely yours, Ivan Daschinskiy


--

Роза Айсина

Старший разработчик ПО

СберМаркет | Доставка из любимых магазинов



Email: roza.aysina@sbermarket.ru<ma...@sbermarket.ru>

Mob:

Web: sbermarket.ru<https://sbermarket.ru/>

App: iOS<https://apps.apple.com/ru/app/%D1%81%D0%B1%D0%B5%D1%80%D0%BC%D0%B0%D1%80%D0%BA%D0%B5%D1%82-%D0%B4%D0%BE%D1%81%D1%82%D0%B0%D0%B2%D0%BA%D0%B0-%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D0%BE%D0%B2/id1166642457> и Android<https://play.google.com/store/apps/details?id=ru.instamart&hl=en&gl=ru>



УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: это электронное сообщение и любые документы, приложенные к нему, содержат конфиденциальную информацию. Настоящим уведомляем Вас о том, что, если это сообщение не предназначено Вам, использование, копирование, распространение информации, содержащейся в настоящем сообщении, а также осуществление любых действий на основе этой информации, строго запрещено. Если Вы получили это сообщение по ошибке, пожалуйста, сообщите об этом отправителю по электронной почте и удалите это сообщение.
CONFIDENTIALITY NOTICE: This email and any files attached to it are confidential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email.

Re: Method get_all does not work in Python

Posted by Ivan Daschinsky <iv...@gmail.com>.
You can use unwrap_binary:

data = cache.get_all([Key(id=10)])
for k,v in data.items():
    print(client.unwrap_binary(k))
    print(v)


пн, 26 дек. 2022 г. в 11:31, Айсина Роза Мунеровна <
roza.aysina@sbermarket.ru>:

> Hola!
>
> I can’t use method *get_all* for caches from PyIgnite client.
> The problem is that keys are some binary strings that I do not know how to
> parse.
>
> *Here what I do: *
>
>
> class ProductFeaturesKey(
>     metaclass=GenericObjectMeta,
>     schema={
>         'product_sku': IntObject
>     },
>     type_name='io.sbmt.ProductFeaturesKey'
> ):
>     pass
>
> …
>
> keys = [ProductFeaturesKey(product_sku=15052004),
>         ProductFeaturesKey(product_sku=15058148)]
>
> res = cache.get_all(keys)
>
>
>
> *>> res*
> {
> (b'g\x01\x0b\x00\xa3\xb8r&%&\xd1\x01"\x00\x00\x00>8_Q\x1d\x00\x00\x00\x03\xe4\xac\xe5\x00\xcd\xa9\x1c\xa7\x18',
> 0): IoSbmtProductFeaturesValue(sku_mean_order_weight=10531.3330078125,
> sku_mean_order_quantity=21.0, sku_mean_order_item_count=15.333333015441895,
> total_cnt_orders_with_sku=12, total_cnt_retailers_with_sku=1,
> total_cnt_stores_with_sku=3, version=1),
>
> (b'g\x01\x0b\x00\xa3\xb8r&=\x80\xd1\x01"\x00\x00\x00>8_Q\x1d\x00\x00\x00\x03\xe4\xc4\xe5\x00\xcd\xa9\x1c\xa7\x18',
> 0): IoSbmtProductFeaturesValue(sku_mean_order_weight=9669.0,
> sku_mean_order_quantity=26.33333396911621,
> sku_mean_order_item_count=20.33333396911621, total_cnt_orders_with_sku=3,
> total_cnt_retailers_with_sku=2, total_cnt_stores_with_sku=3, version=1)
> }
>
>
>
> *>> res[keys[0]]*
> ...
> TypeError: unhashable type: 'ProductFeaturesKey'
>
>
>
> In the resulting dict the keys are binary strings which is odd.
> And as I see possible reason can be that original key instances are not
> hashable
> (see GenericObjectMeta definition:
> https://github.com/apache/ignite-python-thin-client/blob/master/pyignite/binary.py#L220
>  ).
>
> Is there any way to extract several keys and be able to parse resulting
> keys to object?
> Single cache.get(key) method works too slow (about ~10ms on 400RPS) in
> the key loop, we want to extract all keys by one call.
>
> Thanks! Please help.
>
> Ignite version: 2.13.0
> Pyignite version: 0.5.2
>
> *--*
>
> *Роза Айсина*
>
> Старший разработчик ПО
>
> *СберМаркет* | Доставка из любимых магазинов
>
>
>
> Email: roza.aysina@sbermarket.ru
>
> Mob:
>
> Web: sbermarket.ru
>
> App: iOS
> <https://apps.apple.com/ru/app/%D1%81%D0%B1%D0%B5%D1%80%D0%BC%D0%B0%D1%80%D0%BA%D0%B5%D1%82-%D0%B4%D0%BE%D1%81%D1%82%D0%B0%D0%B2%D0%BA%D0%B0-%D0%BF%D1%80%D0%BE%D0%B4%D1%83%D0%BA%D1%82%D0%BE%D0%B2/id1166642457>
> и Android
> <https://play.google.com/store/apps/details?id=ru.instamart&hl=en&gl=ru>
>
>
>
> *УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ:* это электронное сообщение и любые
> документы, приложенные к нему, содержат конфиденциальную информацию.
> Настоящим уведомляем Вас о том, что, если это сообщение не предназначено
> Вам, использование, копирование, распространение информации, содержащейся в
> настоящем сообщении, а также осуществление любых действий на основе этой
> информации, строго запрещено. Если Вы получили это сообщение по ошибке,
> пожалуйста, сообщите об этом отправителю по электронной почте и удалите это
> сообщение.
> *CONFIDENTIALITY NOTICE:* This email and any files attached to it are
> confidential. If you are not the intended recipient you are notified that
> using, copying, distributing or taking any action in reliance on the
> contents of this information is strictly prohibited. If you have received
> this email in error please notify the sender and delete this email.
>


-- 
Sincerely yours, Ivan Daschinskiy