[UI] data model and functionality

[Bill Fernandez <bi...@billfernandez.com>]

Greetings ESME team:

In trying to design a proposed end-user web UI for ESME last week, I 
realized there were too many open issues related to the ESME 
dataspace (as seen from the end user's perspective), and the 
functionality associated therewith.  So I spent most of my time last 
week thinking about those issues.  I have a lot of ideas to suggest, 
and questions to ask, and thought for the team to review and discuss. 
It all seems to long and complicated for email, but the esme-dev 
mailing list seems like the only thing we've got -- so here goes (I 
do wish we had a discussion board...).

I hope this all makes sense that that you don't mind me raising so 
many issues at once.

Regards,
Bill


============================================================
THOUGHTS AND QUESTIONS RE. THE END-USER DATA-MODEL AND RELATED 
FUNCTIONALITY FOR ESME

by Bill Fernandez, 19feb09
============================================================

____________________________________________________________
SERVERS, DIRECTORIES, USERS, PROFILES

SCOPE
o Let's assume that we're limiting our discussion to a single server, 
and that we'll deal with federations of servers in a future 
discussion.

USER DATABASE ASSUMPTIONS
o I'm assuming that for each ESME server there will be one and only 
one user directory server (such as an LDAP server).

o I'm assuming that an ESME server will only recognize users listed 
in the directory server.

o I'm assuming that the directory server is managed by 
organization-level administrators, and that only these administrators 
can add, edit and delete users.

IMPLICATIONS OF THE USER DATABASE ASSUMPTIONS
o WHO can use a given ESME server is thus out of control of the ESME 
server, and in control of the user directory administrator(s).

o Since the ESME server will have to keep track of a number of 
ESME-specific data for each user (e.g. "who do I follow?"), the ESME 
server will have to maintain a database of "preferences" that are 
keyed to each user in the directory server.

o Within the ESME end-user UI, this results in there being some info 
that comes from the directory server and thus that the end user CAN'T 
change (e.g. name, work phone, department, mailstop, etc.) and info 
that only applies within the ESME context and that the end user CAN 
change (e.g. nickname, maybe his/her picture or icon, list of people 
he/she follows, etc.).

QUESTIONS, ISSUES
o Are the above assumptions and implications correct?

o Do we want to allow the administrator of an ESME server to be able 
to create and maintain it's own private database of authorized users 
in addition to those provided by the organization's directory server?

       PROs: Makes it easy to set up a test/evaluation ESME server. 
Makes it possible for small organizations (those without corporate 
LDAP, etc. directories) to use ESME.

       CONs: Introduces the possibility of duplicate user entries.  If 
used extensively makes synchronization with the organizational 
directory impossible.  Provides an end run around policies that the 
organization is counting on the organizational directory to enforce.

o What ESME-specific, end-user-defined, user-profile data do we want 
to support?

       ESME Nickname? -- But what if organizational policy dictates 
the use of only a person's "official" user name?

       Picture/Icon? -- But what if the organizational directory 
server supplies one?  Do we use the "official" image by default, and 
let any user-defined image override it?

o Would our strategy be that the ESME administrator could 
enable/disable any of these user-profile-customization fields as 
required by organizational policy, that we use the 
organizational-server-supplied data by default, but override it with 
the user-supplied values (if any)?



____________________________________________________________
GROUPS, POOLS, USERS, PRIVILEGES, ACCESS CONTROL

David says he hates the idea of groups (though he likes "pools") and 
of sending things to groups (though he likes "posting to pools"). 
Nevertheless I'm going to use the term "groups" because it helps me 
describe certain functionality, and I hope we can decide on 
terminology after the functionality's agreed to.

With that, I'd like to suggest some concepts and see how the team 
feels about them.

GROUPS
o I'd like to propose that each server can support an abitrary number 
of groups.

o A group is a logical construct consisting of a set of data:
     -- A set of users.
     -- A pool of messages.
     -- A set of metadata.
     -- Perhaps a set of business rules.

GROUP MEMBERSHIP
o All users are members of a group called something like "Public", 
which by definition contains the set of all users of the ESME server.

o A user may be a member of as many additional groups as desired.

GROUP CREATION, DEFINITION, DELETION
o Any user can create a group (or would we like the ESME 
administrator to be able to grant a "create group" privilege to only 
certain users?).

o The creator of the group is it's first (and initially only) administrator.

o A group's administrator(s) can add and remove users at will, and 
sets the privileges for each user.

o A group's administrator(s) can edit the group's metadata (e.g. it's 
name), and edit any business rules we support for groups.

USER PRIVILEGES
o Within a group, each user has one of the following privilege levels:
       -- Read.  (Let's you read messages in the group's pool)
       -- Read and Write. (adds the ability to post messages)
       -- Read, Write and Administer (adds all admin privileges)

o The above model establishes three roles:  those of viewer, 
participant and administrator.

o Any user having at least READ privileges against a group has 
PERMISSION to see ALL messages in the group's pool.

ADMINISTRATOR PRIVILEGES

o Anyone with admin privileges is an administrator.  Thus a group can 
have as many admins as desired.

o There's no concept of "ownership", only of "control".  All 
administrators have equal control, and the creator of the group can 
drop off the set of administrators to pass control on to others.

o Over time, organizations will inevitably want finer-grained admin 
privileges.  Do we want to try to build them in from the beginning? 
For example:
       -- Moderate group (lets user delete messages?)
       -- Delete group (lets user destroy the group)
       -- Add/remove read and/or write users
       -- Add/remove, promote/demote, admin users

MESSAGE POOLS
o Before posting a message you must choose a group.  This defines the 
set of users who will have permission to read the message.

o By default, all users are members of the Public group, the Public 
group is the only group that is required for an ESME server, and thus 
in the simplest case all messages are posted to the public pool and 
accessible to everyone.

o Every message posted to a group goes into that group's message 
pool; which is the set of all messages posted to/within that group.

o Which messages a given user sees depends upon what the user has 
chosen to see (who he/she follows, what tags he/she tracks, etc.), 
and/or what he/she searches for.

PRIVACY
o So each message exists in one and only one pool, and only users of 
that pool's group have permission to access that message.

o Therefore by choosing a group before you post a message you are 
choosing the scope of privacy for that message; because (a) ALL 
members of that group will have a way (by searching, tracking, 
following, etc.) to SEE that message, (b) NO users outside that group 
will ever be able to see that message.

JUMPING THE WALLS OF PRIVACY
o Note that since a given message can only exist in one pool over its 
lifetime, if you want multiple groups to recieve the same message you 
must post it to each of those groups separately.  From an intuitive, 
end-user standpoint this appears to be a case of posting the same 
message to multiple groups, but to the system each posting is a 
separate logical entity (each has it's own unique ID, even if the 
text of each is the same).

o So if you see a message in one pool that you want users of another 
pool to see, you have to "copy" it from the one and "paste" it into 
the other.

o This gives us a simple, safe privacy model, yet makes it easy and 
convenient for users to circumvent the rules when necessary.  It 
takes the burden off the system to manage access control of 
cross-pool messaging, and puts the burden of responibility on the 
user.

ACCESS CONTROL
o Thus the fundamental access control model is:  (a) create groups of 
users that need to communicate with each other, (b) post messages to 
the appropriate group, (c) the system will keep your messages private 
to the group.

PRIVATE, INTERPERSONAL MESSAGING
o I think we also need to make it possible to have private 
conversations between any two users without having to formally set up 
a group.  It should be easy to make it so that you choose a user in 
the UI as the "group", post a message, and see all the messages in 
this ad-hoc, private pool.

o But then, of course, the two users are going to want to bring 
others in on some conversations, so maybe it would be better to make 
private, ad-hoc group messaging a property of conversations....  I 
don't yet have a suggestion for how to handle this anticipated 
end-user desire.

____________________________________________________________
STREAMS, TRACKS, SEARCHES, VIEWS

o OK, so I've suggested that: messages be clustered into pools, each 
pool has a wall of privacy around it that ony allows users in it's 
group to "fish" for messages from that pool, and that a given user 
may be a member of multiple groups and thus have access to multiple 
pools of messages.

o So how to we convert the user's permissions against this 
multi-pooled dataspace into useful and usable views?

HOW ARE MESSAGES STORED?
o I'm assuming that messages are basically records in a relational 
database, and therefore that creating end-user views is 
(mechanically) a matter of running a query and formatting the output.

STREAMS
o Most of the time the view a user wants is an ordered list of 
messages meeting certain criteria.  I call these lists "streams".

o In various contexts I have heard the use of terms like "timeline", 
"mailbox", "tracking", "conversations", etc.  In all cases the common 
element is that they all consist of a query against the set of 
messages, a results set that is limted by a user's access controls, a 
sort order (usually reverse-chronological order), and the results 
displayed as a list of messages.  I would call all of these streams.

o Here are some common streams:
       -- The Public timeline.
       -- A conversation.
       -- A track (a timeline defined by your tracking criteria).
       -- The messages directed (via @yourname) to you.
       -- Replies to your messages.
       -- Each group would have its own timeline.
       -- The set of messages sent by users you follow.

SYSTEM-DEFINED STREAMS
o The system should create certain predefined streams for you on 
demand.  You should simply be able to ask to view, and get a stream 
for:

       o The messages in a selected group's pool.
       o The messages sent by a selected user.
       o The messages with a seleted tag.
       o The messages posted by users you follow.
       o All messages in the Public pool.
       o The messages you've sent.
       o etc.

ACCESS CONTROL IN STREAMS
o In all cases, each user only gets to see those messages that their 
access privileges allow them to see.

o For example, if you follow someone who posts to groups of which you 
are not a member, you will never under any circumstances be able to 
see THOSE messages.  On the other hand, any messages posted to the 
Public group (of which you ARE a member), and posted to groups of 
which you are a member; you'll be able to see in the streams you 
request.

DIRECTED MESSAGES
o So what happens if a poster directs a message (via @username) to a 
given user, but does so when posting to a group of which the user is 
not a member.  In that case, by the rule of group privacy, the 
targeted user would never see the message.

FILTERING STREAMS
o We should make it possible (and easy) for users to filter a stream.

o For example, let's say that a given server has only one group 
defined so far, the Public group, making all messages in the server 
available (in theory) to all users.

o By default, on viewing the public stream, we would display only the 
most recent (say) 20 messages, and let the user page back in history 
to see previous messages.

o We should also let the user apply ad-hoc filtering criteria to the 
stream without having to formally do a search.  So the user could ask 
to see only messages with a certain tag, and/or by a certain user, 
and the stream should re-display with most recent (say) 20 messages 
from the now-filtered list.  Underr the covers I expect this to be 
implemented as an ad-hoc query, taking some search criteria from the 
basic stream definition and adding some terms from the user-selected 
criteria.


TRACKING
o Tracking seems to me to be nothing more than stored queries that 
produce streams.  So for example when you say "show me my track of 
the ESME tag", the server runs a query against all the groups to 
which you have access for messages tagged with ESME, sorts them in 
reverse chronological order, and shows you the first (say) 20 results.

SEARCHING
o So far I've described timelines and tracking as ways to generate 
streams on-demand by running stored queries.  The only real 
difference is that at the UI level we are calling them different 
things for the user's benefit.

o I think we should also make it possible for users to explicitly "do 
a search".

o users should be able to create a query by specifying:
	o which group pool(s) to search.
	o what tags to look for.
	o what date range to search.
	o whether files, URLs, etc. are attached.
	o what user sent them.
	o etc.
And to be able to show the results in various ways:
	o list of messages.
	o list of files or URLs sent as attachments.
	o a graph of message volume over time.
	o a chart showing the most/least active users.
	o a cloud of tags, or users, or keywords, etc.

CLOUDS
o I think of a cloud as another kind of view onto the results of a 
query; although a cloud is special in that for performance reasons 
you need to pre-compute much of the results set (e.g. by indexing.).

o So, for example, the cloud of tags used by JohnDoe would be the 
precompiled list of tags, filtered by their association with JohnDow.

o Since cloud data has to be pre-compiled, we have to define what 
kinds of clouds we're interested in.  We currently have tag clouds, 
and that seems like it should stay.  We currently have word-frequency 
clouds, and I'm not sure we should keep this.  If we want to have 
user-participation clouds, etc. we'll need to say so up front.

CHARTS
o System administrators will need to see charts of different kinds to 
monitor system loading, performance, error rates, etc.

o Maybe there are charts that end-users would like to see too.  For 
example, would group administrators want to see charts of activity 
levels over time?

____________________________________________________________
CONVERSATIONS

o My idea for conversations is that if you reply to a message you now 
have a two-message conversation: which illustrates the rule that a 
conversation is a set of messages linked in a tree-shaped structure, 
linked from child to parent by virtue of the child being a reply to 
the parent.

o Within any conversation tree there will thus be parents, children 
and siblings, and users might want to see any/all of those in 
"viewing" a conversation.

o I could see one view of a conversation that is simple a 
forward-chronological listing (a stream) of the messages in the 
conversation.

o But I could also image users wanting to navigate the tree to follow 
the threads of discourse; which means a different UI than just a list 
of messages.  Perhaps, then, we'd have to open conversations into 
their own windows to view them...


____________________________________________________________
MESSAGES, PAYLOADS, METADATA

o So what is a message (and consequently, what can you do with it)?

o To a first approximation, a message is a short text string, posted 
by a single user, to a designated message pool.

SYSTEM META-DATA
o But there is also a lot of metadata that's automatically associated 
with the message:
       -- Sender
       -- Posting timepoint
       -- Group (or pool)
       -- is this a reply?
       -- if so, then to what message?
       -- etc.

  USER-SUPPLIED METADATA
  o There is also metadata that the user can add at his/her discretion:
        -- tags
        -- directed recipients (via @username)
        -- expiration timepoint?

  PAYLOAD
  o I suggest that we also allow users to attach "payloads" to messages.

  o One form of payload is a URL.  A URL is a long text string, that 
if contained in the message body would make the message too long.  So 
we should make it possible to select a run of text in the message to 
use as the "name" of the URL, then to define the text of the URL 
elsewhere.  There could be two UIs for this:  For the novice, let the 
user type the URL in a separate text box.  For the expert, let the 
user use a special syntax to type the URL inline with the message 
body.

o Another type of payload would me an email address...

o Another type of payload could be a file...

o In all cases we might require that each payload have a run of text 
in the message body to act as the name, anchor and access point for 
the payload.  payload names might be styled to look like HTML links; 
and clicking on them might invoke the payload (opening a link in a 
new window or tab, opening an email address in the user's mail 
client, downloading a file to the user's computer, etc.).

o It should thus be possible for a message to have an arbitrary 
number of payloads.

o If we require that every payload have some "name" in the body of 
the message, then it should be easy to copy and paste these names 
(and thus the payloads) between messages.  This would be useful, for 
example, when you want to share a file or a link from a message you 
received with another user; or to include it in a new message you are 
sending out.

FILE PAYLOADS
o My first idea about payloads that are files is that the ESME server 
should act like a file store and forward server.  As long as there is 
any message in any pool that references a given file we should store 
it, and allow it to be retrieved whenever a user asks for it.

o Privacy and security is, to a first approximation, enforced by the 
group pools:  if a file is attached to a message, it can only be 
accessed by users in that pool.  If a user chooses to copy the name 
of (and hence a link to) a file and to paste it into a message posted 
to another pool, then it becomes available to that pool too; at the 
responsibility of the user.

o One might ask how to remove a file from circulation once it's been 
attached to an intial message and uploaded to the ESME server.  Once 
might ask if the file has an owner, who can get a list of his/her 
files and force them to be deleted.  Reasonable questions.  But I'm 
wondering if it makes sense to treat files the same way we do URLs. 
Each URL points to a resource (typically a file of SOME sort) on the 
internet, and once published there's no way to retract the 
distribution of a URL string.  So maybe we could treat files the same 
way....

o Or we might say that when the original message either expires 
(because the sender put an expiration timepoint on it) or is purged 
from the system (because there's a server policy in place that purges 
all messages older than a certain age), that only then does the file 
get deleted from the file store.

TEXT MESSAGES, ETC.
o It's easy for a payload reference (in essence, perhaps just a URL) 
to be embedded in the message string itself if the message string is 
HTML.  If so, then we embed URLs as part of anchor tags and the web 
browser automagically displays just the name of the anchor while 
carrying with it the full URL string.

o On the other hand, what happens when we forward a message to a 
non-HTML-based system; such as cellphone text messaging (SMS)?  In 
these cases we'd have to strip out the payload, and perhaps just 
insert a payload stub -- a special symbol that means "there used to 
be a payload attached to this word", etc.


____________________________________________________________
NAMESPACES, COMMAND LINE MESSAGES

o I'd also like us to nail down the at least the basic set of special 
characters and commands that can be embedded into messages.

o We use @username to make sure a user sees a message (subject to 
access rules and depending on what timeline he/she's viewing).

o We also use #tagname to mark embedded tags.

o It also occurs to me that we could use $servicename to direct a 
message to a service (a machine that will respond to our message)...

o Twitter has a number of letter commands, such as "d" at the 
beginning of a message to make it private. I remember seeing a long 
list of special commands recognized by Twitter, but they seemed: (a) 
fragile, and (b) hard to extend the command set.

o I wonder about a command language with a syntax like "p: @fred @joe 
here's my message" to cause a private message to be sent to only fred 
and joe?  Does anyone have a handle on the set of commands we 
envision for our command-line language?





____________________________________________________________



____________________________________________________________
-- 

======================================================================
Bill Fernandez  *  User Interface Architect  *  Bill Fernandez Design

(505) 346-3080  *  bill@billfernandez.com  *  http://billfernandez.com
======================================================================

Re: [UI] data model and functionality

[Daniel Koller <da...@googlemail.com>]

Hi Bill,

thx for compiling such an overview: my comments follow inline:

On Thu, Feb 19, 2009 at 11:25 PM, Bill Fernandez <bi...@billfernandez.com>wrote:


>
> ============================================================
> THOUGHTS AND QUESTIONS RE. THE END-USER DATA-MODEL AND RELATED
> FUNCTIONALITY FOR ESME
>
> by Bill Fernandez, 19feb09
> ============================================================
>
> ____________________________________________________________
> SERVERS, DIRECTORIES, USERS, PROFILES
>
> SCOPE
> o Let's assume that we're limiting our discussion to a single server, and
> that we'll deal with federations of servers in a future discussion.
>
> USER DATABASE ASSUMPTIONS
> o I'm assuming that for each ESME server there will be one and only one
> user directory server (such as an LDAP server).

In my opinion we should allow more sources as a user database: jdbc, ldap,
active directory also in combination.

>
>
> o I'm assuming that an ESME server will only recognize users listed in the
> directory server.
>
> o I'm assuming that the directory server is managed by organization-level
> administrators, and that only these administrators can add, edit and delete
> users.
>
> IMPLICATIONS OF THE USER DATABASE ASSUMPTIONS
> o WHO can use a given ESME server is thus out of control of the ESME
> server, and in control of the user directory administrator(s).
>
> o Since the ESME server will have to keep track of a number of
> ESME-specific data for each user (e.g. "who do I follow?"), the ESME server
> will have to maintain a database of "preferences" that are keyed to each
> user in the directory server.

The ESME server should use an user identifier (from a directory), but should
keep the preferences in an own data store independent from the directoy.
(--> I am thinking a environments, where ESME can read/use the directory,
but not modify it)

>
>
> o Within the ESME end-user UI, this results in there being some info that
> comes from the directory server and thus that the end user CAN'T change
> (e.g. name, work phone, department, mailstop, etc.) and info that only
> applies within the ESME context and that the end user CAN change (e.g.
> nickname, maybe his/her picture or icon, list of people he/she follows,
> etc.).
>
> QUESTIONS, ISSUES
> o Are the above assumptions and implications correct?
>
> o Do we want to allow the administrator of an ESME server to be able to
> create and maintain it's own private database of authorized users in
> addition to those provided by the organization's directory server?
>
>      PROs: Makes it easy to set up a test/evaluation ESME server. Makes it
> possible for small organizations (those without corporate LDAP, etc.
> directories) to use ESME.
>
>      CONs: Introduces the possibility of duplicate user entries.  If used
> extensively makes synchronization with the organizational directory
> impossible.  Provides an end run around policies that the organization is
> counting on the organizational directory to enforce.
>
> o What ESME-specific, end-user-defined, user-profile data do we want to
> support?
>
>      ESME Nickname? -- But what if organizational policy dictates the use
> of only a person's "official" user name?

We should support an ESME specific nicknames, because username policy might
also mean that I am "Z000EC1G", which is ok for windows domains, but not for
a microblogging tool.

>
>
>      Picture/Icon? -- But what if the organizational directory server
> supplies one?  Do we use the "official" image by default, and let any
> user-defined image override it?

If you let the image_url property be either editable or not, this topic
should be solved.

>
>
> o Would our strategy be that the ESME administrator could enable/disable
> any of these user-profile-customization fields as required by organizational
> policy, that we use the organizational-server-supplied data by default, but
> override it with the user-supplied values (if any)?

Good point.

>
>
>
>
> ____________________________________________________________
> GROUPS, POOLS, USERS, PRIVILEGES, ACCESS CONTROL
>
> David says he hates the idea of groups (though he likes "pools") and of
> sending things to groups (though he likes "posting to pools"). Nevertheless
> I'm going to use the term "groups" because it helps me describe certain
> functionality, and I hope we can decide on terminology after the
> functionality's agreed to.
>
> With that, I'd like to suggest some concepts and see how the team feels
> about them.
>
> GROUPS
> o I'd like to propose that each server can support an abitrary number of
> groups.
>
> o A group is a logical construct consisting of a set of data:
>    -- A set of users.
>    -- A pool of messages.
>    -- A set of metadata.
>    -- Perhaps a set of business rules.
>
> GROUP MEMBERSHIP
> o All users are members of a group called something like "Public", which by
> definition contains the set of all users of the ESME server.
>
> o A user may be a member of as many additional groups as desired.
>
> GROUP CREATION, DEFINITION, DELETION
> o Any user can create a group (or would we like the ESME administrator to
> be able to grant a "create group" privilege to only certain users?).
>
> o The creator of the group is it's first (and initially only)
> administrator.
>
> o A group's administrator(s) can add and remove users at will, and sets the
> privileges for each user.
>
> o A group's administrator(s) can edit the group's metadata (e.g. it's
> name), and edit any business rules we support for groups.


I would love a feature to populate groups automatically based on business
rules such as "create a group with all people with jobtitle=controller in
department="XYZ" (based on regular expressions this could be quite powerful.

>
>
> USER PRIVILEGES
> o Within a group, each user has one of the following privilege levels:
>      -- Read.  (Let's you read messages in the group's pool)
>      -- Read and Write. (adds the ability to post messages)
>      -- Read, Write and Administer (adds all admin privileges)
>
> o The above model establishes three roles:  those of viewer, participant
> and administrator.
>
> o Any user having at least READ privileges against a group has PERMISSION
> to see ALL messages in the group's pool.
>
> ADMINISTRATOR PRIVILEGES
>
> o Anyone with admin privileges is an administrator.  Thus a group can have
> as many admins as desired.
>
> o There's no concept of "ownership", only of "control".  All administrators
> have equal control, and the creator of the group can drop off the set of
> administrators to pass control on to others.
>
> o Over time, organizations will inevitably want finer-grained admin
> privileges.  Do we want to try to build them in from the beginning? For
> example:
>      -- Moderate group (lets user delete messages?)
>      -- Delete group (lets user destroy the group)
>      -- Add/remove read and/or write users
>      -- Add/remove, promote/demote, admin users
>
> MESSAGE POOLS
> o Before posting a message you must choose a group.  This defines the set
> of users who will have permission to read the message.
>
> o By default, all users are members of the Public group, the Public group
> is the only group that is required for an ESME server, and thus in the
> simplest case all messages are posted to the public pool and accessible to
> everyone.
>
> o Every message posted to a group goes into that group's message pool;
> which is the set of all messages posted to/within that group.
>
> o Which messages a given user sees depends upon what the user has chosen to
> see (who he/she follows, what tags he/she tracks, etc.), and/or what he/she
> searches for.
>
> PRIVACY
> o So each message exists in one and only one pool, and only users of that
> pool's group have permission to access that message.
>
> o Therefore by choosing a group before you post a message you are choosing
> the scope of privacy for that message; because (a) ALL members of that group
> will have a way (by searching, tracking, following, etc.) to SEE that
> message, (b) NO users outside that group will ever be able to see that
> message.
>
> JUMPING THE WALLS OF PRIVACY
> o Note that since a given message can only exist in one pool over its
> lifetime, if you want multiple groups to recieve the same message you must
> post it to each of those groups separately.  From an intuitive, end-user
> standpoint this appears to be a case of posting the same message to multiple
> groups, but to the system each posting is a separate logical entity (each
> has it's own unique ID, even if the text of each is the same).
>
> o So if you see a message in one pool that you want users of another pool
> to see, you have to "copy" it from the one and "paste" it into the other.
>
> o This gives us a simple, safe privacy model, yet makes it easy and
> convenient for users to circumvent the rules when necessary.  It takes the
> burden off the system to manage access control of cross-pool messaging, and
> puts the burden of responibility on the user.
>
> ACCESS CONTROL
> o Thus the fundamental access control model is:  (a) create groups of users
> that need to communicate with each other, (b) post messages to the
> appropriate group, (c) the system will keep your messages private to the
> group.
>
> PRIVATE, INTERPERSONAL MESSAGING
> o I think we also need to make it possible to have private conversations
> between any two users without having to formally set up a group.  It should
> be easy to make it so that you choose a user in the UI as the "group", post
> a message, and see all the messages in this ad-hoc, private pool.
>
> o But then, of course, the two users are going to want to bring others in
> on some conversations, so maybe it would be better to make private, ad-hoc
> group messaging a property of conversations....  I don't yet have a
> suggestion for how to handle this anticipated end-user desire.
>
> ____________________________________________________________
> STREAMS, TRACKS, SEARCHES, VIEWS
>
> o OK, so I've suggested that: messages be clustered into pools, each pool
> has a wall of privacy around it that ony allows users in it's group to
> "fish" for messages from that pool, and that a given user may be a member of
> multiple groups and thus have access to multiple pools of messages.
>
> o So how to we convert the user's permissions against this multi-pooled
> dataspace into useful and usable views?
>
> HOW ARE MESSAGES STORED?
> o I'm assuming that messages are basically records in a relational
> database, and therefore that creating end-user views is (mechanically) a
> matter of running a query and formatting the output.
>
> STREAMS
> o Most of the time the view a user wants is an ordered list of messages
> meeting certain criteria.  I call these lists "streams".
>
> o In various contexts I have heard the use of terms like "timeline",
> "mailbox", "tracking", "conversations", etc.  In all cases the common
> element is that they all consist of a query against the set of messages, a
> results set that is limted by a user's access controls, a sort order
> (usually reverse-chronological order), and the results displayed as a list
> of messages.  I would call all of these streams.
>
> o Here are some common streams:
>      -- The Public timeline.
>      -- A conversation.
>      -- A track (a timeline defined by your tracking criteria).
>      -- The messages directed (via @yourname) to you.
>      -- Replies to your messages.
>      -- Each group would have its own timeline.
>      -- The set of messages sent by users you follow.
>
> SYSTEM-DEFINED STREAMS
> o The system should create certain predefined streams for you on demand.
>  You should simply be able to ask to view, and get a stream for:
>
>      o The messages in a selected group's pool.
>      o The messages sent by a selected user.
>      o The messages with a seleted tag.
>      o The messages posted by users you follow.
>      o All messages in the Public pool.
>      o The messages you've sent.
>      o etc.
>
All messages containing a defined expression?

>
> ACCESS CONTROL IN STREAMS
> o In all cases, each user only gets to see those messages that their access
> privileges allow them to see.
>
> o For example, if you follow someone who posts to groups of which you are
> not a member, you will never under any circumstances be able to see THOSE
> messages.  On the other hand, any messages posted to the Public group (of
> which you ARE a member), and posted to groups of which you are a member;
> you'll be able to see in the streams you request.
>
> DIRECTED MESSAGES
> o So what happens if a poster directs a message (via @username) to a given
> user, but does so when posting to a group of which the user is not a member.
>  In that case, by the rule of group privacy, the targeted user would never
> see the message.

The sneder should get a feedback on that.

>
>
> FILTERING STREAMS
> o We should make it possible (and easy) for users to filter a stream.
>
> o For example, let's say that a given server has only one group defined so
> far, the Public group, making all messages in the server available (in
> theory) to all users.
>
> o By default, on viewing the public stream, we would display only the most
> recent (say) 20 messages, and let the user page back in history to see
> previous messages.
>
> o We should also let the user apply ad-hoc filtering criteria to the stream
> without having to formally do a search.  So the user could ask to see only
> messages with a certain tag, and/or by a certain user, and the stream should
> re-display with most recent (say) 20 messages from the now-filtered list.
>  Underr the covers I expect this to be implemented as an ad-hoc query,
> taking some search criteria from the basic stream definition and adding some
> terms from the user-selected criteria.
>
>
> TRACKING
> o Tracking seems to me to be nothing more than stored queries that produce
> streams.  So for example when you say "show me my track of the ESME tag",
> the server runs a query against all the groups to which you have access for
> messages tagged with ESME, sorts them in reverse chronological order, and
> shows you the first (say) 20 results.
>
> SEARCHING
> o So far I've described timelines and tracking as ways to generate streams
> on-demand by running stored queries.  The only real difference is that at
> the UI level we are calling them different things for the user's benefit.
>
> o I think we should also make it possible for users to explicitly "do a
> search".
>
> o users should be able to create a query by specifying:
>        o which group pool(s) to search.
>        o what tags to look for.
>        o what date range to search.
>        o whether files, URLs, etc. are attached.
>        o what user sent them.
>        o etc.
> And to be able to show the results in various ways:
>        o list of messages.
>        o list of files or URLs sent as attachments.
>        o a graph of message volume over time.
>        o a chart showing the most/least active users.
>        o a cloud of tags, or users, or keywords, etc.
>
> CLOUDS
> o I think of a cloud as another kind of view onto the results of a query;
> although a cloud is special in that for performance reasons you need to
> pre-compute much of the results set (e.g. by indexing.).
>
> o So, for example, the cloud of tags used by JohnDoe would be the
> precompiled list of tags, filtered by their association with JohnDow.
>
> o Since cloud data has to be pre-compiled, we have to define what kinds of
> clouds we're interested in.  We currently have tag clouds, and that seems
> like it should stay.  We currently have word-frequency clouds, and I'm not
> sure we should keep this.  If we want to have user-participation clouds,
> etc. we'll need to say so up front.


Additionally we should provide the content of clouds also via an api. (this
decouples ESME from means to present the cloud content.

>
>
> CHARTS
> o System administrators will need to see charts of different kinds to
> monitor system loading, performance, error rates, etc.
>
> o Maybe there are charts that end-users would like to see too.  For
> example, would group administrators want to see charts of activity levels
> over time?
>
> ____________________________________________________________
> CONVERSATIONS
>
> o My idea for conversations is that if you reply to a message you now have
> a two-message conversation: which illustrates the rule that a conversation
> is a set of messages linked in a tree-shaped structure, linked from child to
> parent by virtue of the child being a reply to the parent.
>
> o Within any conversation tree there will thus be parents, children and
> siblings, and users might want to see any/all of those in "viewing" a
> conversation.
>
> o I could see one view of a conversation that is simple a
> forward-chronological listing (a stream) of the messages in the
> conversation.
>
> o But I could also image users wanting to navigate the tree to follow the
> threads of discourse; which means a different UI than just a list of
> messages.  Perhaps, then, we'd have to open conversations into their own
> windows to view them...
>
>
> ____________________________________________________________
> MESSAGES, PAYLOADS, METADATA
>
> o So what is a message (and consequently, what can you do with it)?
>
> o To a first approximation, a message is a short text string, posted by a
> single user, to a designated message pool.
>
> SYSTEM META-DATA
> o But there is also a lot of metadata that's automatically associated with
> the message:
>      -- Sender
>      -- Posting timepoint
>      -- Group (or pool)
>      -- is this a reply?
>      -- if so, then to what message?
>      -- etc.
>
>  USER-SUPPLIED METADATA
>  o There is also metadata that the user can add at his/her discretion:
>       -- tags
>       -- directed recipients (via @username)
>       -- expiration timepoint?
>
>  PAYLOAD
>  o I suggest that we also allow users to attach "payloads" to messages.
>
>  o One form of payload is a URL.  A URL is a long text string, that if
> contained in the message body would make the message too long.  So we should
> make it possible to select a run of text in the message to use as the "name"
> of the URL, then to define the text of the URL elsewhere.  There could be
> two UIs for this:  For the novice, let the user type the URL in a separate
> text box.  For the expert, let the user use a special syntax to type the URL
> inline with the message body.
>
> o Another type of payload would me an email address...
>
> o Another type of payload could be a file...
>
> o In all cases we might require that each payload have a run of text in the
> message body to act as the name, anchor and access point for the payload.
>  payload names might be styled to look like HTML links; and clicking on them
> might invoke the payload (opening a link in a new window or tab, opening an
> email address in the user's mail client, downloading a file to the user's
> computer, etc.).
>
> o It should thus be possible for a message to have an arbitrary number of
> payloads.
>
> o If we require that every payload have some "name" in the body of the
> message, then it should be easy to copy and paste these names (and thus the
> payloads) between messages.  This would be useful, for example, when you
> want to share a file or a link from a message you received with another
> user; or to include it in a new message you are sending out.
>
> FILE PAYLOADS
> o My first idea about payloads that are files is that the ESME server
> should act like a file store and forward server.  As long as there is any
> message in any pool that references a given file we should store it, and
> allow it to be retrieved whenever a user asks for it.

Good point: if we see this extension, we should provide an interface for
virus scanners.
Another one: if there is a library to shorten urls by default, we could
incorporate it here.

>
>
> o Privacy and security is, to a first approximation, enforced by the group
> pools:  if a file is attached to a message, it can only be accessed by users
> in that pool.  If a user chooses to copy the name of (and hence a link to) a
> file and to paste it into a message posted to another pool, then it becomes
> available to that pool too; at the responsibility of the user.
>
> o One might ask how to remove a file from circulation once it's been
> attached to an intial message and uploaded to the ESME server.  Once might
> ask if the file has an owner, who can get a list of his/her files and force
> them to be deleted.  Reasonable questions.  But I'm wondering if it makes
> sense to treat files the same way we do URLs. Each URL points to a resource
> (typically a file of SOME sort) on the internet, and once published there's
> no way to retract the distribution of a URL string.  So maybe we could treat
> files the same way....

How would it be if we supply a personlized list of documents posted and an
option for the user (and the admin) to explicitly delete a certain file.

>
>
> o Or we might say that when the original message either expires (because
> the sender put an expiration timepoint on it) or is purged from the system
> (because there's a server policy in place that purges all messages older
> than a certain age), that only then does the file get deleted from the file
> store.

We should provide a means to archive conversations from ESME.

>
>
> TEXT MESSAGES, ETC.
> o It's easy for a payload reference (in essence, perhaps just a URL) to be
> embedded in the message string itself if the message string is HTML.  If so,
> then we embed URLs as part of anchor tags and the web browser automagically
> displays just the name of the anchor while carrying with it the full URL
> string.
>
> o On the other hand, what happens when we forward a message to a
> non-HTML-based system; such as cellphone text messaging (SMS)?  In these
> cases we'd have to strip out the payload, and perhaps just insert a payload
> stub -- a special symbol that means "there used to be a payload attached to
> this word", etc.
>
>
> ____________________________________________________________
> NAMESPACES, COMMAND LINE MESSAGES
>
> o I'd also like us to nail down the at least the basic set of special
> characters and commands that can be embedded into messages.
>
> o We use @username to make sure a user sees a message (subject to access
> rules and depending on what timeline he/she's viewing).
>
> o We also use #tagname to mark embedded tags.
>
> o It also occurs to me that we could use $servicename to direct a message
> to a service (a machine that will respond to our message)..

Depends on whether we want to handle bots different from users, otherwise we
could call bots also via @

> .
>
> o Twitter has a number of letter commands, such as "d" at the beginning of
> a message to make it private. I remember seeing a long list of special
> commands recognized by Twitter, but they seemed: (a) fragile, and (b) hard
> to extend the command set.
>
> o I wonder about a command language with a syntax like "p: @fred @joe
> here's my message" to cause a private message to be sent to only fred and
> joe?  Does anyone have a handle on the set of commands we envision for our
> command-line language?
>
>
>
>
> Kind regards,


Daniel

-- 
---
Daniel Koller
Jahnstrasse 20
80469 München * dakoller@googlemail.com