You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@usergrid.apache.org by Andrew Lane <an...@gmail.com> on 2016/07/18 20:00:54 UTC

how to stream a long list of connections

Let's say I have some collection called someentity that has connections to
other entities via a verb subscribedto. For a particular entity with id
some_id, I'd like to pull all the entities that are connected to this
entity via the subscribedto verb. I can do that via this GET request:

/org/app/someentity/some_id/subscribedto

However, I'm not sure how I stream or page through this data if there are
thousands or more results. I'm not getting back a cursor or anything. Is
having a huge number of connections to a particular entity something that's
just not a smart thing to do with Usergrid?

Andrew Lane

Re: how to stream a long list of connections

Posted by Dave <sn...@gmail.com>.
Tracking this as USERGRID-1310

https://issues.apache.org/jira/browse/USERGRID-1310



On Mon, Jul 18, 2016 at 4:39 PM Andrew Lane <an...@gmail.com> wrote:

> I should be clear that I DO get a cursor when querying the relationship in
> the opposite direction.  That is, if a single user is subscribed to more
> than 1 event, I can stream those successfully and I get a cursor.  It's the
> inverse direction that's not giving me a cursor.
>
> On Mon, Jul 18, 2016 at 4:36 PM, Andrew Lane <an...@gmail.com>
> wrote:
>
>> I have a collection of events, and users can subscribe to those events
>> via a connection called "subscribedto".  For a particular event
>> (some_event_id), I'm trying to stream all the users that are subscribed to
>> that event.  In my scenario, there are over 1000 subscribers.  I'm issuing
>> something like this (note the "connecting" part of the URL, which I
>> accidentally ommitted previously):
>>
>> curl -X GET "
>> http://myserver:8080/myorg/sandbox/events/some_event_id/connecting/subscribedto?limit=5
>> "
>>
>> It's correctly limiting to 5 users, but I'm not getting any cursor where
>> I can continue streaming.  If I remove the limit clause, I only get the
>> first 1000 results, which I expect.  But again, no cursor to continue to
>> pull data.
>>
>> Andrew Lane
>>
>> On Mon, Jul 18, 2016 at 4:14 PM, Dave <sn...@gmail.com> wrote:
>>
>>> Usergrid is designed to support a huge number of connections, and we
>>> implemented "edge sharding" to ensure that we are not thwarted by
>>> Cassandra's 2-billion column limitation.
>>>
>>> The normal way to page through results is to have a cursor, so the fact
>>> that you did not get a cursor is a problem, and most likely a bug -- can
>>> you share the exact API call you are making as a curl (or HTTPie) command?
>>> Do you see any errors in the logs?
>>>
>>> Dave
>>>
>>>
>>> On Mon, Jul 18, 2016 at 4:01 PM Andrew Lane <an...@gmail.com>
>>> wrote:
>>>
>>>> Let's say I have some collection called someentity that has
>>>> connections to other entities via a verb subscribedto. For a
>>>> particular entity with id some_id, I'd like to pull all the entities
>>>> that are connected to this entity via the subscribedto verb. I can do
>>>> that via this GET request:
>>>>
>>>> /org/app/someentity/some_id/subscribedto
>>>>
>>>> However, I'm not sure how I stream or page through this data if there
>>>> are thousands or more results. I'm not getting back a cursor or anything.
>>>> Is having a huge number of connections to a particular entity something
>>>> that's just not a smart thing to do with Usergrid?
>>>>
>>>> Andrew Lane
>>>>
>>>
>>
>

Re: how to stream a long list of connections

Posted by Andrew Lane <an...@gmail.com>.
I should be clear that I DO get a cursor when querying the relationship in
the opposite direction.  That is, if a single user is subscribed to more
than 1 event, I can stream those successfully and I get a cursor.  It's the
inverse direction that's not giving me a cursor.

On Mon, Jul 18, 2016 at 4:36 PM, Andrew Lane <an...@gmail.com>
wrote:

> I have a collection of events, and users can subscribe to those events via
> a connection called "subscribedto".  For a particular event
> (some_event_id), I'm trying to stream all the users that are subscribed to
> that event.  In my scenario, there are over 1000 subscribers.  I'm issuing
> something like this (note the "connecting" part of the URL, which I
> accidentally ommitted previously):
>
> curl -X GET "
> http://myserver:8080/myorg/sandbox/events/some_event_id/connecting/subscribedto?limit=5
> "
>
> It's correctly limiting to 5 users, but I'm not getting any cursor where I
> can continue streaming.  If I remove the limit clause, I only get the first
> 1000 results, which I expect.  But again, no cursor to continue to pull
> data.
>
> Andrew Lane
>
> On Mon, Jul 18, 2016 at 4:14 PM, Dave <sn...@gmail.com> wrote:
>
>> Usergrid is designed to support a huge number of connections, and we
>> implemented "edge sharding" to ensure that we are not thwarted by
>> Cassandra's 2-billion column limitation.
>>
>> The normal way to page through results is to have a cursor, so the fact
>> that you did not get a cursor is a problem, and most likely a bug -- can
>> you share the exact API call you are making as a curl (or HTTPie) command?
>> Do you see any errors in the logs?
>>
>> Dave
>>
>>
>> On Mon, Jul 18, 2016 at 4:01 PM Andrew Lane <an...@gmail.com>
>> wrote:
>>
>>> Let's say I have some collection called someentity that has connections
>>> to other entities via a verb subscribedto. For a particular entity with
>>> id some_id, I'd like to pull all the entities that are connected to
>>> this entity via the subscribedto verb. I can do that via this GET
>>> request:
>>>
>>> /org/app/someentity/some_id/subscribedto
>>>
>>> However, I'm not sure how I stream or page through this data if there
>>> are thousands or more results. I'm not getting back a cursor or anything.
>>> Is having a huge number of connections to a particular entity something
>>> that's just not a smart thing to do with Usergrid?
>>>
>>> Andrew Lane
>>>
>>
>

Re: how to stream a long list of connections

Posted by Andrew Lane <an...@gmail.com>.
I have a collection of events, and users can subscribe to those events via
a connection called "subscribedto".  For a particular event
(some_event_id), I'm trying to stream all the users that are subscribed to
that event.  In my scenario, there are over 1000 subscribers.  I'm issuing
something like this (note the "connecting" part of the URL, which I
accidentally ommitted previously):

curl -X GET "
http://myserver:8080/myorg/sandbox/events/some_event_id/connecting/subscribedto?limit=5
"

It's correctly limiting to 5 users, but I'm not getting any cursor where I
can continue streaming.  If I remove the limit clause, I only get the first
1000 results, which I expect.  But again, no cursor to continue to pull
data.

Andrew Lane

On Mon, Jul 18, 2016 at 4:14 PM, Dave <sn...@gmail.com> wrote:

> Usergrid is designed to support a huge number of connections, and we
> implemented "edge sharding" to ensure that we are not thwarted by
> Cassandra's 2-billion column limitation.
>
> The normal way to page through results is to have a cursor, so the fact
> that you did not get a cursor is a problem, and most likely a bug -- can
> you share the exact API call you are making as a curl (or HTTPie) command?
> Do you see any errors in the logs?
>
> Dave
>
>
> On Mon, Jul 18, 2016 at 4:01 PM Andrew Lane <an...@gmail.com>
> wrote:
>
>> Let's say I have some collection called someentity that has connections
>> to other entities via a verb subscribedto. For a particular entity with
>> id some_id, I'd like to pull all the entities that are connected to this
>> entity via the subscribedto verb. I can do that via this GET request:
>>
>> /org/app/someentity/some_id/subscribedto
>>
>> However, I'm not sure how I stream or page through this data if there are
>> thousands or more results. I'm not getting back a cursor or anything. Is
>> having a huge number of connections to a particular entity something that's
>> just not a smart thing to do with Usergrid?
>>
>> Andrew Lane
>>
>

Re: how to stream a long list of connections

Posted by Dave <sn...@gmail.com>.
Usergrid is designed to support a huge number of connections, and we
implemented "edge sharding" to ensure that we are not thwarted by
Cassandra's 2-billion column limitation.

The normal way to page through results is to have a cursor, so the fact
that you did not get a cursor is a problem, and most likely a bug -- can
you share the exact API call you are making as a curl (or HTTPie) command?
Do you see any errors in the logs?

Dave


On Mon, Jul 18, 2016 at 4:01 PM Andrew Lane <an...@gmail.com> wrote:

> Let's say I have some collection called someentity that has connections
> to other entities via a verb subscribedto. For a particular entity with
> id some_id, I'd like to pull all the entities that are connected to this
> entity via the subscribedto verb. I can do that via this GET request:
>
> /org/app/someentity/some_id/subscribedto
>
> However, I'm not sure how I stream or page through this data if there are
> thousands or more results. I'm not getting back a cursor or anything. Is
> having a huge number of connections to a particular entity something that's
> just not a smart thing to do with Usergrid?
>
> Andrew Lane
>