The Pool stuff is good to go

[David Pollak <fe...@gmail.com>]

Folks,
I've looked at the pool stuff.  Vassil did a great job with it.  I made a
very minor performance enhancement (using Set instead of List which means
O(log n) rather than O(n) for access checking).

Rock and roll and if it's got the votes (which I think it does), I say roll
it on into trunk.

Thanks,

David

-- 
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

Re: The Pool stuff is good to go

[Richard Hirsch <hi...@gmail.com>]

OK. I'll delete the database and create it again, since via stax I don't
have the ability to have fine-grained database admin.  We will lose all data
/ users but this shouldn't be problem.

D.

On Sat, Jun 20, 2009 at 12:55 PM, Vassil Dichev <vd...@apache.org> wrote:

> > we've been trying it out. Everything looks good except that we haven't
> > found a way to view the messages in a particular pool. Is there a special
> > page to view the messages like in yammer, are they posted to the public
> > timeline or is this a feature that isn't implemented yet?
>
> This shouldn't happen. Unfortunately you might need to delete the
> database so that it's rebuilt, because there are some incompatible
> changes, for instance every message now has a reference to a pool.
> I've had this problem as well.
>
> Note also that if someone does not have permissions for a pool at the
> time a message is posted to this pool, the message will not appear in
> the user's timeline even when the user is consequently granted
> permissions for this pool! The message should still be accessible via
> searching though.
>

Re: The Pool stuff is good to go

[Vassil Dichev <vd...@apache.org>]

> we've been trying it out. Everything looks good except that we haven't
> found a way to view the messages in a particular pool. Is there a special
> page to view the messages like in yammer, are they posted to the public
> timeline or is this a feature that isn't implemented yet?

This shouldn't happen. Unfortunately you might need to delete the
database so that it's rebuilt, because there are some incompatible
changes, for instance every message now has a reference to a pool.
I've had this problem as well.

Note also that if someone does not have permissions for a pool at the
time a message is posted to this pool, the message will not appear in
the user's timeline even when the user is consequently granted
permissions for this pool! The message should still be accessible via
searching though.

Re: The Pool stuff is good to go

[Vassil Dichev <vd...@apache.org>]

> Another question: Have you thought about deletion of pools and deleting a
> user from a pool.

Most definitely ESME will have this. The Web UI is lacking somewhat at
this point, but will be refined later on. There are also some
usability issues, for instance when a pool is added, it doesn't appear
in the drop-down menu, and the table of pools and permissions is
positively ugly, but I managed to get Ajax to work. I decided not to
focus on the UI at this point.

> Daniel had a good idea about creating a quick client to read a directory
> (ldap) and the use this data to call the rest api. Of course, this is only
> necessary until tighter integration with conatiner functionality is present

Interesting, where can I find more details? You can add a pool via the
Web UI and indicate it's generated from LDAP by setting the realm to
e.g. "LDAP" (we might need to make it more sophisticated in the
future). If there is already an external structure of pools and
corresponding privileges, it should be copied to the ESME database
(currently possible via the Web UI) and not evaluated dynamically, for
reasons of performance and robustness.

Re: The Pool stuff is good to go

[Richard Hirsch <hi...@gmail.com>]

It appears that the problem may be restricted to the web UI. I'm receiving
messages via the twitter api.

Another question: Have you thought about deletion of pools and deleting a
user from a pool.

Daniel had a good idea about creating a quick client to read a directory
(ldap) and the use this data to call the rest api. Of course, this is only
necessary until tighter integration with conatiner functionality is present

D,

On Sat, Jun 20, 2009 at 10:23 AM, Richard Hirsch <hi...@gmail.com>wrote:

> Hi,
>
> The stax deployment is finished<http://esmecloudserverapache.dickhirsch.staxapps.net/>and we've been trying it out. Everything looks good except that we haven't
> found a way to view the messages in a particular pool. Is there a special
> page to view the messages like in yammer, are they posted to the public
> timeline or is this a feature that isn't implemented yet?
>
> D.
>
>   On Sat, Jun 20, 2009 at 8:05 AM, Richard Hirsch <hi...@gmail.com>wrote:
>
>> @Vassil: Can you provide a very quick description of the pool
>> functionality that you implemented.
>>
>> I'll do a build today and deploy on to stax
>>
>> D.
>>
>>   On Sat, Jun 20, 2009 at 12:51 AM, Vassil Dichev <vd...@apache.org>wrote:
>>
>>> Access pools are merged in trunk now. Go give it a try!
>>>
>>
>>
>

Re: The Pool stuff is good to go

[Richard Hirsch <hi...@gmail.com>]

Hi,

The stax deployment is
finished<http://esmecloudserverapache.dickhirsch.staxapps.net/>and
we've been trying it out. Everything looks good except that we haven't
found a way to view the messages in a particular pool. Is there a special
page to view the messages like in yammer, are they posted to the public
timeline or is this a feature that isn't implemented yet?

D.

On Sat, Jun 20, 2009 at 8:05 AM, Richard Hirsch <hi...@gmail.com>wrote:

> @Vassil: Can you provide a very quick description of the pool functionality
> that you implemented.
>
> I'll do a build today and deploy on to stax
>
> D.
>
>   On Sat, Jun 20, 2009 at 12:51 AM, Vassil Dichev <vd...@apache.org>wrote:
>
>> Access pools are merged in trunk now. Go give it a try!
>>
>
>

Re: The Pool stuff is good to go

[Vassil Dichev <vd...@apache.org>]

There is some more pool functionality, which works now:

- As those of you trying the stax deployment have noticed, the *pool
name* is now displayed on the user's timeline

- Messages in pools you're not authorized to view will not appear in
the Web UI or RestAPI. This includes:
  * timeline
  * tracking
  * search (it's currently disabled, but should be filtered out if it wasn't)
  * viewing other users' timeline and messages from the "List users" page
This doesn't yet include the TwitterAPI and Tags from restricted
messages will (probably) also be visible in the tag cloud. I haven't
decided if the latter is a feature or a bug yet, but tend to think
tags from pools shouldn't leak to people who are otherwise not able to
see them. What do you think?

I managed to add an extra query parameter to "find*" methods via the
findMapDb, if there's a logged in user. Unfortunately addlQueryParams
will not do, as the param added is not static but depends on the
current user.

What's next:
1. Deleting users from a pool.
2. Deleting a pool (by first removing all users from it).
3. Generating messages you can filter on when you're added to a pool;
messages upon creating a new pool shouldn't be generated (I think).
4. Filtering on pools in actions. One idea which this will allow me to
implement is an action, which executes the message in a Scala
interpreter if it's sent to a certain private pool. Of course, this is
as big a security hole as you can get, so you should be able to
enable/disable on startup. Besides, there are already security
restrictions in place in the sandboxed environment of Stax and any
other cloud.

Interpreting a message as Scala code should have educational value
similar to what lotrepls.appspot.com provides. It would be invaluable
for memory profiling and other sort of debugging. For instance, it
will make it possible to execute e.g. David's memory profiling code as
an action, which scores extra coolness points ;-)

Vassil

Re: The Pool stuff is good to go

[Vassil Dichev <vd...@apache.org>]

> @Vassil: Can you provide a very quick description of the pool functionality
> that you implemented.

Of course, here's what you can do with pools.

On the "Manage Access Pools" page:
* You can create a new pool. By default you are an administrator of
this pool. There is a column which indicates the realm of the access
pool, indicating how the pool was created. Currently it's just a
string and if the Web UI is used it is "Native".
* You can set permissions for a user in a pool (if you have Admin
permissions yourself)- Read, Write or Admin.

On the message page:
* Users who have Write permissions can send a message to this pool via
a menu. By default you can send the message without any pool assigned
to it, i.e. to the so called public pool.
* Users can see in their timeline messages in the public pool and in
pools for which they have permissions. There's currently no page where
you can see messages only from a specific pool, nor does the Web UI
show which pool a message belongs to.
* Messages which belong to any pool don't appear in the public timeline.
* Messages in a pool for which a user doesn't have permissions cannot
be tracked and shouldn't show on the Tracking page.

There's also RESTful API for creating a pool, setting permissions of a
user in a pool and sending a message to a pool (described in a
previous mail).

What *doesn't* work yet:
* Messages are still shown where they shouldn't be visible. For
instance, when you view another user's timeline, you can see all of
their messages regardless of the fact you shouldn't have access to
them. Seeing all messages from the Twitter API shouldn't be allowed
either and is actually a bug! Remember- pools are not meant for just
filtering content based on topic or interests, but for access control.
In this way they are a bit different from Yammer's groups because we
don't have public groups. It's probably more similar to Yammer's
private groups.
* You cannot delete a pool. In ESME different entities, like tracking,
actions, are not deleted from the database, but marked as disabled.
Probably same policy would apply to pools.
* You cannot currently remove a user from a pool, which is one of the
first thing I want to implement next.
* You cannot rename a pool.
* You cannot search/track by pool or use them in actions (and I'd like
ESME to be able to use them).

I have a small outline of what pools can do. When I combine it with
the explanations in several mails I plan to publish a more complete
document on the wiki.

Have fun,
Vassil

Re: The Pool stuff is good to go

[Richard Hirsch <hi...@gmail.com>]

@Vassil: Can you provide a very quick description of the pool functionality
that you implemented.

I'll do a build today and deploy on to stax

D.

On Sat, Jun 20, 2009 at 12:51 AM, Vassil Dichev <vd...@apache.org> wrote:

> Access pools are merged in trunk now. Go give it a try!
>

Re: The Pool stuff is good to go

[Vassil Dichev <vd...@apache.org>]

Access pools are merged in trunk now. Go give it a try!

Re: The Pool stuff is good to go

[David Pollak <fe...@gmail.com>]

It's missing from the lift book because the feature is 3 days old... :-)

On Jun 19, 2009 1:30 PM, "Vassil Dichev" <vd...@apache.org> wrote:

> It's easier than that... all models have an "addlQueryParams" instance >
variable.  As a wrapper, ...
Wow, good thing I asked before wasting too much time on it.

Surprisingly, this is the first thing I find missing from the Lift
book... but I have yet to find a way I cannot extend Lift in a
flexible and elegant manner.

Re: The Pool stuff is good to go

[Vassil Dichev <vd...@apache.org>]

> It's easier than that... all models have an "addlQueryParams" instance
> variable.  As a wrapper, define it to limit access to the current pool.  If
> you've got questions on how to do it, ping me and we can work together on
> it.

Wow, good thing I asked before wasting too much time on it.

Surprisingly, this is the first thing I find missing from the Lift
book... but I have yet to find a way I cannot extend Lift in a
flexible and elegant manner.

Re: The Pool stuff is good to go

[David Pollak <fe...@gmail.com>]

On Fri, Jun 19, 2009 at 12:30 PM, Vassil Dichev <vd...@apache.org> wrote:

> > I've looked at the pool stuff.  Vassil did a great job with it.  I made a
>
> Thank you!
>
> > very minor performance enhancement (using Set instead of List which means
> > O(log n) rather than O(n) for access checking).
>
> Good idea. I have some concerns about making a new query for each pool
> a user can write to just to get its name, but it might be early to
> optimize, and we can certainly do it later on.
>
> > Rock and roll and if it's got the votes (which I think it does), I say
> roll
> > it on into trunk.
>
> Great, I will proceed with the merge right away.
>
> @David, I want your advice on one more issue. There are many
> occurrences in ESME where a find* method is called on Message. These
> calls would retrieve also messages from pools the user is not supposed
> to see. What do you think is a better solution:
>
> -I'm toying with the idea of overriding findMapDb (since all calls to
> findAll and findMap eventually call it). I'm trying to insert an SQL
> fragment, which would restrict results only to the allowed pools. This
> is a single point of control, so should be easier to maintain. OTOH it
> could be less transparent and harder to debug if you don't expect it.


It's easier than that... all models have an "addlQueryParams" instance
variable.  As a wrapper, define it to limit access to the current pool.  If
you've got questions on how to do it, ping me and we can work together on
it.


>
>
> -the standard solution of changing all instances of findAll/findMap is
> more easy to understand, but harder to maintain- one must make sure no
> find* call is forgotten.
>
> Overall, I'm leaning towards overriding the single findMapDb. It's
> similar to aspects: it might be more difficult to debug and reason
> about, but it's cleaner and separates the concern. The find* methods
> are part of the model API and potential new ESME committers could
> forget to restrict the call to only return messages from appropriate
> pools.
>
> Everyone: relax, this will not delay merging, we will resolve the
> issue afterwards.
>
> Cheers,
> Vassil
>



-- 
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

Re: The Pool stuff is good to go

[Vassil Dichev <vd...@apache.org>]

> I've looked at the pool stuff.  Vassil did a great job with it.  I made a

Thank you!

> very minor performance enhancement (using Set instead of List which means
> O(log n) rather than O(n) for access checking).

Good idea. I have some concerns about making a new query for each pool
a user can write to just to get its name, but it might be early to
optimize, and we can certainly do it later on.

> Rock and roll and if it's got the votes (which I think it does), I say roll
> it on into trunk.

Great, I will proceed with the merge right away.

@David, I want your advice on one more issue. There are many
occurrences in ESME where a find* method is called on Message. These
calls would retrieve also messages from pools the user is not supposed
to see. What do you think is a better solution:

-I'm toying with the idea of overriding findMapDb (since all calls to
findAll and findMap eventually call it). I'm trying to insert an SQL
fragment, which would restrict results only to the allowed pools. This
is a single point of control, so should be easier to maintain. OTOH it
could be less transparent and harder to debug if you don't expect it.

-the standard solution of changing all instances of findAll/findMap is
more easy to understand, but harder to maintain- one must make sure no
find* call is forgotten.

Overall, I'm leaning towards overriding the single findMapDb. It's
similar to aspects: it might be more difficult to debug and reason
about, but it's cleaner and separates the concern. The find* methods
are part of the model API and potential new ESME committers could
forget to restrict the call to only return messages from appropriate
pools.

Everyone: relax, this will not delay merging, we will resolve the
issue afterwards.

Cheers,
Vassil