Client serialization issue: read from Couchbase in Trident function

[Leo <le...@gmail.com>]

Hi All,

Basically, I want to read some additional attributes from Couchbase with
attributes from the tuple, for example, the tuple has input fields "{a, b,
c}", I want to emit a tuple "{a, b, c, d, e}" where "d" and "e" are read
from Couchbase with "a" as the key.

However, Trident BaseFunction is serializable while CouchbaseClient is not,
which leads to "NotSerializable" exception, is there anyway to get around
this? There are two possible solutions I can think of:

1) read the additional attributes outside Trident and send them together to
Trident (I don't want to do this cause I want to put this kind of
processing in Storm)

2) Use state, which is read-only (only implement multi-get, leaving
multi-put blank, not sure whether this will do the trick)

Is there any easy way to do this? This should be dorable in Storm itself
and is pretty common in real world. It also applies to other NoSQL db like
Cassandra.

Thanks,

Leo

Re: Client serialization issue: read from Couchbase in Trident function

[Leo <le...@gmail.com>]

Looks there is prepare() method in BaseFunction, let me try to create there
and see what's going on. Will keep you updated.

On Wed, May 6, 2015 at 8:07 AM, Leo <le...@gmail.com> wrote:

> It's dorable in Storm, but Trident Functions require everything to be
> serializable, or it throws errors. I tried to make CouchbaseClient
> transient, but this won't work either. I'm just curious this should be
> pretty common, or there is another way to get around this?
>
> Thanks,
> Leo
>
> On Wed, May 6, 2015 at 6:18 AM, Nathan Leung <nc...@gmail.com> wrote:
>
>> Why does couchbase client need to be serializable? In core storm you
>> would initialize the client in prepare(), not when the bolt object is
>> constructed. I'm not familiar with trident but I'm guessing there is
>> something similar.
>> On May 6, 2015 1:22 AM, "Leo" <le...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Basically, I want to read some additional attributes from Couchbase with
>>> attributes from the tuple, for example, the tuple has input fields "{a, b,
>>> c}", I want to emit a tuple "{a, b, c, d, e}" where "d" and "e" are read
>>> from Couchbase with "a" as the key.
>>>
>>> However, Trident BaseFunction is serializable while CouchbaseClient is
>>> not, which leads to "NotSerializable" exception, is there anyway to get
>>> around this? There are two possible solutions I can think of:
>>>
>>> 1) read the additional attributes outside Trident and send them together
>>> to Trident (I don't want to do this cause I want to put this kind of
>>> processing in Storm)
>>>
>>> 2) Use state, which is read-only (only implement multi-get, leaving
>>> multi-put blank, not sure whether this will do the trick)
>>>
>>> Is there any easy way to do this? This should be dorable in Storm itself
>>> and is pretty common in real world. It also applies to other NoSQL db like
>>> Cassandra.
>>>
>>> Thanks,
>>>
>>> Leo
>>>
>>
>

Re: Client serialization issue: read from Couchbase in Trident function

[Leo <le...@gmail.com>]

It's dorable in Storm, but Trident Functions require everything to be
serializable, or it throws errors. I tried to make CouchbaseClient
transient, but this won't work either. I'm just curious this should be
pretty common, or there is another way to get around this?

Thanks,
Leo

On Wed, May 6, 2015 at 6:18 AM, Nathan Leung <nc...@gmail.com> wrote:

> Why does couchbase client need to be serializable? In core storm you would
> initialize the client in prepare(), not when the bolt object is
> constructed. I'm not familiar with trident but I'm guessing there is
> something similar.
> On May 6, 2015 1:22 AM, "Leo" <le...@gmail.com> wrote:
>
>> Hi All,
>>
>> Basically, I want to read some additional attributes from Couchbase with
>> attributes from the tuple, for example, the tuple has input fields "{a, b,
>> c}", I want to emit a tuple "{a, b, c, d, e}" where "d" and "e" are read
>> from Couchbase with "a" as the key.
>>
>> However, Trident BaseFunction is serializable while CouchbaseClient is
>> not, which leads to "NotSerializable" exception, is there anyway to get
>> around this? There are two possible solutions I can think of:
>>
>> 1) read the additional attributes outside Trident and send them together
>> to Trident (I don't want to do this cause I want to put this kind of
>> processing in Storm)
>>
>> 2) Use state, which is read-only (only implement multi-get, leaving
>> multi-put blank, not sure whether this will do the trick)
>>
>> Is there any easy way to do this? This should be dorable in Storm itself
>> and is pretty common in real world. It also applies to other NoSQL db like
>> Cassandra.
>>
>> Thanks,
>>
>> Leo
>>
>

Re: Client serialization issue: read from Couchbase in Trident function

[Nathan Leung <nc...@gmail.com>]

Why does couchbase client need to be serializable? In core storm you would
initialize the client in prepare(), not when the bolt object is
constructed. I'm not familiar with trident but I'm guessing there is
something similar.
On May 6, 2015 1:22 AM, "Leo" <le...@gmail.com> wrote:

> Hi All,
>
> Basically, I want to read some additional attributes from Couchbase with
> attributes from the tuple, for example, the tuple has input fields "{a, b,
> c}", I want to emit a tuple "{a, b, c, d, e}" where "d" and "e" are read
> from Couchbase with "a" as the key.
>
> However, Trident BaseFunction is serializable while CouchbaseClient is
> not, which leads to "NotSerializable" exception, is there anyway to get
> around this? There are two possible solutions I can think of:
>
> 1) read the additional attributes outside Trident and send them together
> to Trident (I don't want to do this cause I want to put this kind of
> processing in Storm)
>
> 2) Use state, which is read-only (only implement multi-get, leaving
> multi-put blank, not sure whether this will do the trick)
>
> Is there any easy way to do this? This should be dorable in Storm itself
> and is pretty common in real world. It also applies to other NoSQL db like
> Cassandra.
>
> Thanks,
>
> Leo
>