You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Nick North (Created) (JIRA)" <ji...@apache.org> on 2011/12/31 16:28:30 UTC
[jira] [Created] (COUCHDB-1373) Time-ordered document ids including the database identity
Time-ordered document ids including the database identity
----------------------------------------------------------
Key: COUCHDB-1373
URL: https://issues.apache.org/jira/browse/COUCHDB-1373
Project: CouchDB
Issue Type: Improvement
Components: Database Core
Reporter: Nick North
Priority: Minor
This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
To addresses this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Robert Newson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185564#comment-13185564 ]
Robert Newson commented on COUCHDB-1373:
----------------------------------------
Some food for thought: http://blog.boundary.com/2012/01/12/Flake-A-decentralized-k-ordered-id-generation-service-in-Erlang.html
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185661#comment-13185661 ]
Nick North commented on COUCHDB-1373:
-------------------------------------
The Flake algorithm is interesting. I also started with Twitter's algorithm and, at the time, shared Twitter's desire to fit ids into a 64-bit long - that affected my choice of suffix. I'm also not an experienced Erlang programmer, so was heavily influenced by the existing CouchDb utc_random algorithm as it could be adapted with minimal changes.
I think Flake is overly complex, because they use a millisecond clock plus a separate sequence id. This is perhaps because they want an algorithm that can be implemented in any language. In Erlang we can use the full microsecond precision of the clock and rely on the fact that repeated calls to Now are guaranteed to be monotonic increasing. However, Flake's use of the MAC address as a machine identifier saves on config file entries, and might be adopted.
Left to my own devices I would stick with the current proposal, but a more Flake-like one would suit me as well. But if I had to implement it, I'm not sure how long it would take with my limited Erlang knowledge and dev environment.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Bob Dionne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178431#comment-13178431 ]
Bob Dionne commented on COUCHDB-1373:
-------------------------------------
Nick,
It looks good at a glance. I'll try to test it and review it some more later. We don't really have a hard convention for patches, the 0002-... approach seems fine. Often folks just post drafts[1] and proposed patches as github links while they are under discussion. Sometimes I also delete older patches if it gets confusing. This project is pioneering the use of Git at ASF so there's been some discussion about workflows and so forth.
Anyway I'll be happy to commit this if no one finds any issue within the next day or so.
Cheers,
Bob
[1] https://github.com/bdionne/couchdb/compare/master...jira-1373
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178765#comment-13178765 ]
Nick North commented on COUCHDB-1373:
-------------------------------------
Many thanks. I can see that it would also be nice to have a TestSuffix etap test that checks that the suffix remains unchanged for multiple utc_id_suffix UUIDs. I've knocked together some speculative code for that but won't propose it unless/until I can get enough bits running on my system to try it out.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Bob Dionne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178201#comment-13178201 ]
Bob Dionne commented on COUCHDB-1373:
-------------------------------------
Nick,
looks pretty good. There's still a trailing whitespace error in default.ini.tpl.in There are git commands for catching those and/or you can set emacs to correct those on save or whatever. I'm not sure also about putting machine_id in the uuids section, in fact it we're going to have it in the ini then it doesn't even need to be "machine_id" it could be "id_suffix". Just a thought.
We use etap [1] for testing, among other things. In test/etap you can see the tests. Attached is a patch that extends 041-uuid-gen.t for your new case.
Cheers,
Bob
[1] https://github.com/ngerakines/etap
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick North updated COUCHDB-1373:
--------------------------------
Description:
This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
was:
This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
To addresses this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick North updated COUCHDB-1373:
--------------------------------
Attachment: 0002-utc_id_suffix.patch
Thanks for all your help. id_suffix does seem a better name if it is only to be used in uuid generation, so the attached patch 0002-utc_id_suffix.patch changes the algorithm name to utc_id_suffix and uses an id_suffix ini setting. It incorporates your etap test, but with renames of machine_id to id_suffix throughout.
I'm not sure of the usual convention for patches here. The idea of 0002-utc_id_suffix.patch is that it supersedes all previous attached files and should be applied instead of them.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Bob Dionne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186164#comment-13186164 ]
Bob Dionne commented on COUCHDB-1373:
-------------------------------------
I think the option of letting the user specify a suffix in the ini file is still good and supports a use case that occurs a lot, for example when some governing body is responsible for the provenance of work that is collaborative and distributed and is responsible for assigning unique suffixes to each instance.
I'm also not convinced that using the MAC address doesn't have problems, so I'd see adding that as as additional approach. It seems reasonable, but it can be changed, what happens when the database is replicated to another, etc...
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186141#comment-13186141 ]
Nick North commented on COUCHDB-1373:
-------------------------------------
Thanks - that looks pretty straightforward, except that the Erlang documentation at http://www.erlang.org/doc/man/inet.html#getifaddrs-0 says that Solaris does not return a {hwaddr, _} tuple. But we can sort out some workaround for that.
I am about to go on holiday for three weeks, but can take a look at incorporating the MAC address when I return, unless someone else wants to have a go in the interim. It might also be nice to give the user the option of specifying a specific suffix if they want, or else give an interface name (and then also have a sensible default if this algorithm is specified with neither a suffix nor an interface).
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178049#comment-13178049 ]
Nick North commented on COUCHDB-1373:
-------------------------------------
Thanks for the reply. I'll submit a patch as soon as I've worked out how to use git.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Robert Newson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Newson resolved COUCHDB-1373.
------------------------------------
Resolution: Fixed
Fix Version/s: 1.3
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Fix For: 1.3
>
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: CouchDB Module System
Posted by Bob Dionne <bo...@cloudant.com>.
Extensions
On Jan 23, 2012, at 4:33 AM, Paul Davis wrote:
> On Mon, Jan 23, 2012 at 3:03 AM, Randall Leeds <ra...@gmail.com> wrote:
>> On Sat, Dec 31, 2011 at 10:34, Jan Johannes <ja...@ntr.io> wrote:
>>> Dear favorite Document-database developer community :-)
>>> This is my first message to the mailing list, so please forgive me and
>>> gently correct me if i am breaking any communication standards whatsoeverŠ
>>
>> Nope! You're just bringing up a deep topic with a lot of text.
>> I hope you didn't expect a speedy response! :)
>>
>>>
>>> I am currently working on a WebDAV interface for CouchDB, that now gets
>>> closer to its release date, somewhere in February or March and as i
>>> approach that, I want to package everything in a proper CouchDB Module
>>> format but the CouchDB Module system is, as i understood, in early
>>> development or pre-development phase and needs a bit of work.
>>>
>>> So as i work on a CouchDB Module anyways i would like to contribute to the
>>> Module system or start kicking the work on that of.
>>>
>>> Hence the following questions:
>>>
>>> Who started working on such a system or would be interested in talking
>>> about the design decisions that need to be done before actual coding can
>>> start?
>>> (I know at least Jan Lehnardt expressed interest, but i can't quite
>>> remember if he started actual work or just had plans for that.)
>>
>> Benoit is particularly interested and there is activity on the refuge
>> project mailing list about it from Nicolas Dufour.
>> I believe Jason Smith might be pretty interested since he's authored
>> some "plugins" already.
>> Others on the list are certainly interested.
>> I'm keen to get a good module system and it's why I was in favor of
>> the effort to get "couch-config" shipped as part of the install.
>>
>>>
>>> If there is any documentation on planning such a system, could someone
>>> please point me to the recourses?
>>
>> There's an umbrella ticket I made for modularizing the code base more:
>> https://issues.apache.org/jira/browse/COUCHDB-1251
>>
>> And these:
>>
>> Utility to help plugin developers manage paths
>> https://issues.apache.org/jira/browse/COUCHDB-1012
>>
>> support load of external erlang modules in couchdb.
>> https://issues.apache.org/jira/browse/COUCHDB-1046
>>
>>>
>>> Lastly here is my two cents on a few topics that need to be addressed:
>>>
>>> - We should build the best module system there is for a database or
>>> content management system to gain back a bit of the traction we lost
>>> during the recent plateau phase of couchdb enthusiasm
>>
>> I find it hard to judge enthusiasm from the inside, but, yeah... a
>> proper module system would rock.
>> One of the things CouchDB struggles with in terms of image, but excels
>> at in terms of function is scaling up and down. The only way to really
>> make one project move from mobile phone to datacenter cluster is to
>> modularize the build and deployment of key components. I think
>> everyone agrees about this. CouchDB is the Gentoo of document
>> databases in this way.
>>
>>>
>>> - I think "module" is the preferable term, as extension, plugin or the
>>> like sound like second class citizens, whereas module implies, that even a
>>> core component of the system can be made into such an instance (Any other
>>> suggestions?)
>>
>> Module is a little bit confusing because it conflicts with Erlang's
>> definition. While the two would often be synonymous for many add-ons,
>> it's not necessarily the case.
>> I would go with calling them "applications" maybe. I've often thought
>> that each such thing could correspond well to a single Erlang
>> application and .app file, which Erlang has significant support and
>> prior art for loading and unloading on the fly. But really, I don't
>> care what color we paint this and long as it's neon awesome.
>>
>>>
>>> - A CouchDB module should be packaged as a stand alone couchdb-document,
>>> with the yet to be specified manifest beeing the json structure of the
>>> document and the files beeing the attachment.
>>
>> I'm pretty sure I disagree here. I think this conflates two issues.
>> The first, to me, is a good hooks system from which more interactions
>> in the core could be scripted with sandboxed user code. Once again, I
>> know I've talked to Benoit about this. The second, though, is the more
>> first-class type of application that requires compiled Erlang
>> bytecode. While these have some degree of portability[1] I
>> instinctively cringe, from a security standpoint, at the idea of hot
>> loading code from a document into the VM. While the document-module
>> idea I find attractive from a share-ability and install-ability
>> standpoint, I don't think it befits the general purpose module system.
>> Although, a good hooks system may make certain kinds of modules work
>> this way.
>>
>> [1] http://stackoverflow.com/questions/2255658/how-portable-are-erlang-beam-files
>>
>>>
>>> - Discovery, Installation, Enabling, Disabling, Module-Specific
>>> Configuration and Distribution should be part of Futon and introduce as
>>> little new interface concepts as possible, also the futon interface
>>> section "configuration" may be needed to be integrated and streamlined
>>> quite nicely
>>
>> Again, this works the dynamic-language, hooks-based module, but I'm
>> not sure it works for all modules. Kanso [2] is doing this to a large
>> extent already, under the CouchApp frame, but I see no reason why a
>> hooks system couldn't accommodate a new, more powerful kind of Kanso
>> package.
>>
>> [2] http://kan.so/
>>
>>>
>>> - Even if a module is not possible to completely install via this method
>>> (e.g. It depends on a os component, that is too low level etc.) such
>>> modules should be as tightly integrated into the system as possible and
>>> the manual admin installation steps should be documented in a way, so that
>>> in a normal case no place but the module system needs to be accessed
>>
>> Agree.
>>
>>>
>>> - on the top level the system should be the same for all OSes
>>>
>>> - The work packages i see at the moment for the structure i have in mind
>>> would be:
>>>
>>> - Module repository: A CouchDB Instance, that hosts a collection of
>>> CouchDB Modules
>>
>> See Kanso, above.
>>
>>> - Futon User Interface to browse available Modules in the configured
>>> repositories, should be close to the Firefox plugin paradigms
>>
>> That would be neat. Or a separate 'shopping' utility for CouchApps.
>> Sounds a bit like a combination between JChris' CouchApp Garden [3]
>> and Max Ogden's Chrome Couch application [4]. Crossing these would
>> Kanso would be really neat.
>>
>> [3] https://github.com/jchris/garden
>> [4] https://github.com/maxogden/chromecouch
>>
>>> - Module Manifest: Descriptive CouchDoc, that references the attached
>>> files or git repository, how files need to be fetched, packaged and what
>>> goes where, also version dependencies, description, rating, Module
>>> Category, needed system privileges (basically if you need to be a couch
>>> admin or if you need to be root on the os), needed configuration, and
>>> basic description of the configuration parameters
>>
>> I'd really prefer to punt on low-level privileges and just call this
>> two separate things. The way to unify them, in my mind, is to create a
>> hooks system and then to make a native module that listens for all
>> hooks and delegates processing to design documents that indicate they
>> handle certain hooks. In other words, build the un-privileged module
>> system on top of the privileged one. Let administrators worry about
>> file systems when installing the privileged ones. It's really not so
>> bad. Sysadmins have been installing things for years, and actually
>> probably prefer to install stuff on the command line than with a POST
>> to /_replicate. The couch-config utility is there to facilitate this.
>> What we probably need is an extra location for user-installed add-on
>> applications that need to be added to the Erlang library path and used
>> for loading configured applications.
>>
>>> - Futon User Interface to the installed modules, with options to enable,
>>> disable, uninstall, rate and configure
>>
>> Yes again. I like the gist, but I don't know if it should be bound up
>> in Futon or not.
>>
>>> - The tools for installation/uninstallation and enabling/disabling as
>>> well as packaging and dependency resolution
>>
>> That sounds like figuring out a good way to structure the
>> configuration sections for modules. See the JIRA tickets I referenced
>> for some discussion of that.
>>
>>>
>>> Open questions: What existing projects could be made use of, maybe there
>>> is something great for the general erlang world, maybe there is a project
>>> like zero install, that could be made use of, maybe that is not
>>> necessary, what are your thoughts?
>>
>> I would also check out Agner: http://erlagner.org/
>>
>> Basically, if we could get to the point where CouchDB administrators
>> could use Agner to install new pieces I'd consider that a huge win.
>> If, in the process, we get a modularized core and support for a
>> design-document-level hooks system I'd consider that an epic win. I
>> never say epic win. I would say it three times out loud.
>>
>>>
>>> I wish you all a happy new year!
>>>
>>>
>>
>> Thanks :)
>> -Randall
>
> +1 to pretty much everything Randall said.
>
> Minor caveat: application is about as overloaded in Erlang as module
> (and process for that matter) when considering prior expectations of
> non-Erlangians. There should be a word for these things. Plug-ins has
> always been my word for them, but the second-class-ness that invokes
> is understandable. Perhaps there's a perfect word, but perhaps
> "component" is good enough?
>
> My other reaction is that installing "whatevers/components/widgets"
> should *not* be a web thing. I understand the desire to make this user
> friendly and the such, but as Randall points out, putting random
> (fairly difficult to sandbox) code into a server/service environment
> seems quite ungood. This basically hearkens back to the
> disabled-by-default native view server. While its definitely
> "possible" to provide some sort of sandboxed thing for Erlang
> extensions, its a "plug all the leaks with fingers" type of possible.
> If we're interested in allowing code execution on the server it should
> be "here's you're iron box with carefully crafted ports of interaction
> to the world" type of sandboxing. Ie, AppEngine vs erm, not-AppEngine.
>
> Other than that, yes. Erlang makes extensions super awesomely easy and
> we should be embracing that more loudly. Both with community resources
> for discovery as well as the technical adjustments to ease their
> usage.
Re: CouchDB Module System
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Jan 23, 2012 at 3:03 AM, Randall Leeds <ra...@gmail.com> wrote:
> On Sat, Dec 31, 2011 at 10:34, Jan Johannes <ja...@ntr.io> wrote:
>> Dear favorite Document-database developer community :-)
>> This is my first message to the mailing list, so please forgive me and
>> gently correct me if i am breaking any communication standards whatsoeverŠ
>
> Nope! You're just bringing up a deep topic with a lot of text.
> I hope you didn't expect a speedy response! :)
>
>>
>> I am currently working on a WebDAV interface for CouchDB, that now gets
>> closer to its release date, somewhere in February or March and as i
>> approach that, I want to package everything in a proper CouchDB Module
>> format but the CouchDB Module system is, as i understood, in early
>> development or pre-development phase and needs a bit of work.
>>
>> So as i work on a CouchDB Module anyways i would like to contribute to the
>> Module system or start kicking the work on that of.
>>
>> Hence the following questions:
>>
>> Who started working on such a system or would be interested in talking
>> about the design decisions that need to be done before actual coding can
>> start?
>> (I know at least Jan Lehnardt expressed interest, but i can't quite
>> remember if he started actual work or just had plans for that.)
>
> Benoit is particularly interested and there is activity on the refuge
> project mailing list about it from Nicolas Dufour.
> I believe Jason Smith might be pretty interested since he's authored
> some "plugins" already.
> Others on the list are certainly interested.
> I'm keen to get a good module system and it's why I was in favor of
> the effort to get "couch-config" shipped as part of the install.
>
>>
>> If there is any documentation on planning such a system, could someone
>> please point me to the recourses?
>
> There's an umbrella ticket I made for modularizing the code base more:
> https://issues.apache.org/jira/browse/COUCHDB-1251
>
> And these:
>
> Utility to help plugin developers manage paths
> https://issues.apache.org/jira/browse/COUCHDB-1012
>
> support load of external erlang modules in couchdb.
> https://issues.apache.org/jira/browse/COUCHDB-1046
>
>>
>> Lastly here is my two cents on a few topics that need to be addressed:
>>
>> - We should build the best module system there is for a database or
>> content management system to gain back a bit of the traction we lost
>> during the recent plateau phase of couchdb enthusiasm
>
> I find it hard to judge enthusiasm from the inside, but, yeah... a
> proper module system would rock.
> One of the things CouchDB struggles with in terms of image, but excels
> at in terms of function is scaling up and down. The only way to really
> make one project move from mobile phone to datacenter cluster is to
> modularize the build and deployment of key components. I think
> everyone agrees about this. CouchDB is the Gentoo of document
> databases in this way.
>
>>
>> - I think "module" is the preferable term, as extension, plugin or the
>> like sound like second class citizens, whereas module implies, that even a
>> core component of the system can be made into such an instance (Any other
>> suggestions?)
>
> Module is a little bit confusing because it conflicts with Erlang's
> definition. While the two would often be synonymous for many add-ons,
> it's not necessarily the case.
> I would go with calling them "applications" maybe. I've often thought
> that each such thing could correspond well to a single Erlang
> application and .app file, which Erlang has significant support and
> prior art for loading and unloading on the fly. But really, I don't
> care what color we paint this and long as it's neon awesome.
>
>>
>> - A CouchDB module should be packaged as a stand alone couchdb-document,
>> with the yet to be specified manifest beeing the json structure of the
>> document and the files beeing the attachment.
>
> I'm pretty sure I disagree here. I think this conflates two issues.
> The first, to me, is a good hooks system from which more interactions
> in the core could be scripted with sandboxed user code. Once again, I
> know I've talked to Benoit about this. The second, though, is the more
> first-class type of application that requires compiled Erlang
> bytecode. While these have some degree of portability[1] I
> instinctively cringe, from a security standpoint, at the idea of hot
> loading code from a document into the VM. While the document-module
> idea I find attractive from a share-ability and install-ability
> standpoint, I don't think it befits the general purpose module system.
> Although, a good hooks system may make certain kinds of modules work
> this way.
>
> [1] http://stackoverflow.com/questions/2255658/how-portable-are-erlang-beam-files
>
>>
>> - Discovery, Installation, Enabling, Disabling, Module-Specific
>> Configuration and Distribution should be part of Futon and introduce as
>> little new interface concepts as possible, also the futon interface
>> section "configuration" may be needed to be integrated and streamlined
>> quite nicely
>
> Again, this works the dynamic-language, hooks-based module, but I'm
> not sure it works for all modules. Kanso [2] is doing this to a large
> extent already, under the CouchApp frame, but I see no reason why a
> hooks system couldn't accommodate a new, more powerful kind of Kanso
> package.
>
> [2] http://kan.so/
>
>>
>> - Even if a module is not possible to completely install via this method
>> (e.g. It depends on a os component, that is too low level etc.) such
>> modules should be as tightly integrated into the system as possible and
>> the manual admin installation steps should be documented in a way, so that
>> in a normal case no place but the module system needs to be accessed
>
> Agree.
>
>>
>> - on the top level the system should be the same for all OSes
>>
>> - The work packages i see at the moment for the structure i have in mind
>> would be:
>>
>> - Module repository: A CouchDB Instance, that hosts a collection of
>> CouchDB Modules
>
> See Kanso, above.
>
>> - Futon User Interface to browse available Modules in the configured
>> repositories, should be close to the Firefox plugin paradigms
>
> That would be neat. Or a separate 'shopping' utility for CouchApps.
> Sounds a bit like a combination between JChris' CouchApp Garden [3]
> and Max Ogden's Chrome Couch application [4]. Crossing these would
> Kanso would be really neat.
>
> [3] https://github.com/jchris/garden
> [4] https://github.com/maxogden/chromecouch
>
>> - Module Manifest: Descriptive CouchDoc, that references the attached
>> files or git repository, how files need to be fetched, packaged and what
>> goes where, also version dependencies, description, rating, Module
>> Category, needed system privileges (basically if you need to be a couch
>> admin or if you need to be root on the os), needed configuration, and
>> basic description of the configuration parameters
>
> I'd really prefer to punt on low-level privileges and just call this
> two separate things. The way to unify them, in my mind, is to create a
> hooks system and then to make a native module that listens for all
> hooks and delegates processing to design documents that indicate they
> handle certain hooks. In other words, build the un-privileged module
> system on top of the privileged one. Let administrators worry about
> file systems when installing the privileged ones. It's really not so
> bad. Sysadmins have been installing things for years, and actually
> probably prefer to install stuff on the command line than with a POST
> to /_replicate. The couch-config utility is there to facilitate this.
> What we probably need is an extra location for user-installed add-on
> applications that need to be added to the Erlang library path and used
> for loading configured applications.
>
>> - Futon User Interface to the installed modules, with options to enable,
>> disable, uninstall, rate and configure
>
> Yes again. I like the gist, but I don't know if it should be bound up
> in Futon or not.
>
>> - The tools for installation/uninstallation and enabling/disabling as
>> well as packaging and dependency resolution
>
> That sounds like figuring out a good way to structure the
> configuration sections for modules. See the JIRA tickets I referenced
> for some discussion of that.
>
>>
>> Open questions: What existing projects could be made use of, maybe there
>> is something great for the general erlang world, maybe there is a project
>> like zero install, that could be made use of, maybe that is not
>> necessary, what are your thoughts?
>
> I would also check out Agner: http://erlagner.org/
>
> Basically, if we could get to the point where CouchDB administrators
> could use Agner to install new pieces I'd consider that a huge win.
> If, in the process, we get a modularized core and support for a
> design-document-level hooks system I'd consider that an epic win. I
> never say epic win. I would say it three times out loud.
>
>>
>> I wish you all a happy new year!
>>
>>
>
> Thanks :)
> -Randall
+1 to pretty much everything Randall said.
Minor caveat: application is about as overloaded in Erlang as module
(and process for that matter) when considering prior expectations of
non-Erlangians. There should be a word for these things. Plug-ins has
always been my word for them, but the second-class-ness that invokes
is understandable. Perhaps there's a perfect word, but perhaps
"component" is good enough?
My other reaction is that installing "whatevers/components/widgets"
should *not* be a web thing. I understand the desire to make this user
friendly and the such, but as Randall points out, putting random
(fairly difficult to sandbox) code into a server/service environment
seems quite ungood. This basically hearkens back to the
disabled-by-default native view server. While its definitely
"possible" to provide some sort of sandboxed thing for Erlang
extensions, its a "plug all the leaks with fingers" type of possible.
If we're interested in allowing code execution on the server it should
be "here's you're iron box with carefully crafted ports of interaction
to the world" type of sandboxing. Ie, AppEngine vs erm, not-AppEngine.
Other than that, yes. Erlang makes extensions super awesomely easy and
we should be embracing that more loudly. Both with community resources
for discovery as well as the technical adjustments to ease their
usage.
Re: CouchDB Module System
Posted by Randall Leeds <ra...@gmail.com>.
Add this one to the pile:
Improve config file write-back behavior.
https://issues.apache.org/jira/browse/COUCHDB-1086
On Mon, Jan 23, 2012 at 01:03, Randall Leeds <ra...@gmail.com> wrote:
> On Sat, Dec 31, 2011 at 10:34, Jan Johannes <ja...@ntr.io> wrote:
>> Dear favorite Document-database developer community :-)
>> This is my first message to the mailing list, so please forgive me and
>> gently correct me if i am breaking any communication standards whatsoeverŠ
>
> Nope! You're just bringing up a deep topic with a lot of text.
> I hope you didn't expect a speedy response! :)
>
>>
>> I am currently working on a WebDAV interface for CouchDB, that now gets
>> closer to its release date, somewhere in February or March and as i
>> approach that, I want to package everything in a proper CouchDB Module
>> format but the CouchDB Module system is, as i understood, in early
>> development or pre-development phase and needs a bit of work.
>>
>> So as i work on a CouchDB Module anyways i would like to contribute to the
>> Module system or start kicking the work on that of.
>>
>> Hence the following questions:
>>
>> Who started working on such a system or would be interested in talking
>> about the design decisions that need to be done before actual coding can
>> start?
>> (I know at least Jan Lehnardt expressed interest, but i can't quite
>> remember if he started actual work or just had plans for that.)
>
> Benoit is particularly interested and there is activity on the refuge
> project mailing list about it from Nicolas Dufour.
> I believe Jason Smith might be pretty interested since he's authored
> some "plugins" already.
> Others on the list are certainly interested.
> I'm keen to get a good module system and it's why I was in favor of
> the effort to get "couch-config" shipped as part of the install.
>
>>
>> If there is any documentation on planning such a system, could someone
>> please point me to the recourses?
>
> There's an umbrella ticket I made for modularizing the code base more:
> https://issues.apache.org/jira/browse/COUCHDB-1251
>
> And these:
>
> Utility to help plugin developers manage paths
> https://issues.apache.org/jira/browse/COUCHDB-1012
>
> support load of external erlang modules in couchdb.
> https://issues.apache.org/jira/browse/COUCHDB-1046
>
>>
>> Lastly here is my two cents on a few topics that need to be addressed:
>>
>> - We should build the best module system there is for a database or
>> content management system to gain back a bit of the traction we lost
>> during the recent plateau phase of couchdb enthusiasm
>
> I find it hard to judge enthusiasm from the inside, but, yeah... a
> proper module system would rock.
> One of the things CouchDB struggles with in terms of image, but excels
> at in terms of function is scaling up and down. The only way to really
> make one project move from mobile phone to datacenter cluster is to
> modularize the build and deployment of key components. I think
> everyone agrees about this. CouchDB is the Gentoo of document
> databases in this way.
>
>>
>> - I think "module" is the preferable term, as extension, plugin or the
>> like sound like second class citizens, whereas module implies, that even a
>> core component of the system can be made into such an instance (Any other
>> suggestions?)
>
> Module is a little bit confusing because it conflicts with Erlang's
> definition. While the two would often be synonymous for many add-ons,
> it's not necessarily the case.
> I would go with calling them "applications" maybe. I've often thought
> that each such thing could correspond well to a single Erlang
> application and .app file, which Erlang has significant support and
> prior art for loading and unloading on the fly. But really, I don't
> care what color we paint this and long as it's neon awesome.
>
>>
>> - A CouchDB module should be packaged as a stand alone couchdb-document,
>> with the yet to be specified manifest beeing the json structure of the
>> document and the files beeing the attachment.
>
> I'm pretty sure I disagree here. I think this conflates two issues.
> The first, to me, is a good hooks system from which more interactions
> in the core could be scripted with sandboxed user code. Once again, I
> know I've talked to Benoit about this. The second, though, is the more
> first-class type of application that requires compiled Erlang
> bytecode. While these have some degree of portability[1] I
> instinctively cringe, from a security standpoint, at the idea of hot
> loading code from a document into the VM. While the document-module
> idea I find attractive from a share-ability and install-ability
> standpoint, I don't think it befits the general purpose module system.
> Although, a good hooks system may make certain kinds of modules work
> this way.
>
> [1] http://stackoverflow.com/questions/2255658/how-portable-are-erlang-beam-files
>
>>
>> - Discovery, Installation, Enabling, Disabling, Module-Specific
>> Configuration and Distribution should be part of Futon and introduce as
>> little new interface concepts as possible, also the futon interface
>> section "configuration" may be needed to be integrated and streamlined
>> quite nicely
>
> Again, this works the dynamic-language, hooks-based module, but I'm
> not sure it works for all modules. Kanso [2] is doing this to a large
> extent already, under the CouchApp frame, but I see no reason why a
> hooks system couldn't accommodate a new, more powerful kind of Kanso
> package.
>
> [2] http://kan.so/
>
>>
>> - Even if a module is not possible to completely install via this method
>> (e.g. It depends on a os component, that is too low level etc.) such
>> modules should be as tightly integrated into the system as possible and
>> the manual admin installation steps should be documented in a way, so that
>> in a normal case no place but the module system needs to be accessed
>
> Agree.
>
>>
>> - on the top level the system should be the same for all OSes
>>
>> - The work packages i see at the moment for the structure i have in mind
>> would be:
>>
>> - Module repository: A CouchDB Instance, that hosts a collection of
>> CouchDB Modules
>
> See Kanso, above.
>
>> - Futon User Interface to browse available Modules in the configured
>> repositories, should be close to the Firefox plugin paradigms
>
> That would be neat. Or a separate 'shopping' utility for CouchApps.
> Sounds a bit like a combination between JChris' CouchApp Garden [3]
> and Max Ogden's Chrome Couch application [4]. Crossing these would
> Kanso would be really neat.
>
> [3] https://github.com/jchris/garden
> [4] https://github.com/maxogden/chromecouch
>
>> - Module Manifest: Descriptive CouchDoc, that references the attached
>> files or git repository, how files need to be fetched, packaged and what
>> goes where, also version dependencies, description, rating, Module
>> Category, needed system privileges (basically if you need to be a couch
>> admin or if you need to be root on the os), needed configuration, and
>> basic description of the configuration parameters
>
> I'd really prefer to punt on low-level privileges and just call this
> two separate things. The way to unify them, in my mind, is to create a
> hooks system and then to make a native module that listens for all
> hooks and delegates processing to design documents that indicate they
> handle certain hooks. In other words, build the un-privileged module
> system on top of the privileged one. Let administrators worry about
> file systems when installing the privileged ones. It's really not so
> bad. Sysadmins have been installing things for years, and actually
> probably prefer to install stuff on the command line than with a POST
> to /_replicate. The couch-config utility is there to facilitate this.
> What we probably need is an extra location for user-installed add-on
> applications that need to be added to the Erlang library path and used
> for loading configured applications.
>
>> - Futon User Interface to the installed modules, with options to enable,
>> disable, uninstall, rate and configure
>
> Yes again. I like the gist, but I don't know if it should be bound up
> in Futon or not.
>
>> - The tools for installation/uninstallation and enabling/disabling as
>> well as packaging and dependency resolution
>
> That sounds like figuring out a good way to structure the
> configuration sections for modules. See the JIRA tickets I referenced
> for some discussion of that.
>
>>
>> Open questions: What existing projects could be made use of, maybe there
>> is something great for the general erlang world, maybe there is a project
>> like zero install, that could be made use of, maybe that is not
>> necessary, what are your thoughts?
>
> I would also check out Agner: http://erlagner.org/
>
> Basically, if we could get to the point where CouchDB administrators
> could use Agner to install new pieces I'd consider that a huge win.
> If, in the process, we get a modularized core and support for a
> design-document-level hooks system I'd consider that an epic win. I
> never say epic win. I would say it three times out loud.
>
>>
>> I wish you all a happy new year!
>>
>>
>
> Thanks :)
> -Randall
Re: CouchDB Module System
Posted by Randall Leeds <ra...@gmail.com>.
On Sat, Dec 31, 2011 at 10:34, Jan Johannes <ja...@ntr.io> wrote:
> Dear favorite Document-database developer community :-)
> This is my first message to the mailing list, so please forgive me and
> gently correct me if i am breaking any communication standards whatsoeverŠ
Nope! You're just bringing up a deep topic with a lot of text.
I hope you didn't expect a speedy response! :)
>
> I am currently working on a WebDAV interface for CouchDB, that now gets
> closer to its release date, somewhere in February or March and as i
> approach that, I want to package everything in a proper CouchDB Module
> format but the CouchDB Module system is, as i understood, in early
> development or pre-development phase and needs a bit of work.
>
> So as i work on a CouchDB Module anyways i would like to contribute to the
> Module system or start kicking the work on that of.
>
> Hence the following questions:
>
> Who started working on such a system or would be interested in talking
> about the design decisions that need to be done before actual coding can
> start?
> (I know at least Jan Lehnardt expressed interest, but i can't quite
> remember if he started actual work or just had plans for that.)
Benoit is particularly interested and there is activity on the refuge
project mailing list about it from Nicolas Dufour.
I believe Jason Smith might be pretty interested since he's authored
some "plugins" already.
Others on the list are certainly interested.
I'm keen to get a good module system and it's why I was in favor of
the effort to get "couch-config" shipped as part of the install.
>
> If there is any documentation on planning such a system, could someone
> please point me to the recourses?
There's an umbrella ticket I made for modularizing the code base more:
https://issues.apache.org/jira/browse/COUCHDB-1251
And these:
Utility to help plugin developers manage paths
https://issues.apache.org/jira/browse/COUCHDB-1012
support load of external erlang modules in couchdb.
https://issues.apache.org/jira/browse/COUCHDB-1046
>
> Lastly here is my two cents on a few topics that need to be addressed:
>
> - We should build the best module system there is for a database or
> content management system to gain back a bit of the traction we lost
> during the recent plateau phase of couchdb enthusiasm
I find it hard to judge enthusiasm from the inside, but, yeah... a
proper module system would rock.
One of the things CouchDB struggles with in terms of image, but excels
at in terms of function is scaling up and down. The only way to really
make one project move from mobile phone to datacenter cluster is to
modularize the build and deployment of key components. I think
everyone agrees about this. CouchDB is the Gentoo of document
databases in this way.
>
> - I think "module" is the preferable term, as extension, plugin or the
> like sound like second class citizens, whereas module implies, that even a
> core component of the system can be made into such an instance (Any other
> suggestions?)
Module is a little bit confusing because it conflicts with Erlang's
definition. While the two would often be synonymous for many add-ons,
it's not necessarily the case.
I would go with calling them "applications" maybe. I've often thought
that each such thing could correspond well to a single Erlang
application and .app file, which Erlang has significant support and
prior art for loading and unloading on the fly. But really, I don't
care what color we paint this and long as it's neon awesome.
>
> - A CouchDB module should be packaged as a stand alone couchdb-document,
> with the yet to be specified manifest beeing the json structure of the
> document and the files beeing the attachment.
I'm pretty sure I disagree here. I think this conflates two issues.
The first, to me, is a good hooks system from which more interactions
in the core could be scripted with sandboxed user code. Once again, I
know I've talked to Benoit about this. The second, though, is the more
first-class type of application that requires compiled Erlang
bytecode. While these have some degree of portability[1] I
instinctively cringe, from a security standpoint, at the idea of hot
loading code from a document into the VM. While the document-module
idea I find attractive from a share-ability and install-ability
standpoint, I don't think it befits the general purpose module system.
Although, a good hooks system may make certain kinds of modules work
this way.
[1] http://stackoverflow.com/questions/2255658/how-portable-are-erlang-beam-files
>
> - Discovery, Installation, Enabling, Disabling, Module-Specific
> Configuration and Distribution should be part of Futon and introduce as
> little new interface concepts as possible, also the futon interface
> section "configuration" may be needed to be integrated and streamlined
> quite nicely
Again, this works the dynamic-language, hooks-based module, but I'm
not sure it works for all modules. Kanso [2] is doing this to a large
extent already, under the CouchApp frame, but I see no reason why a
hooks system couldn't accommodate a new, more powerful kind of Kanso
package.
[2] http://kan.so/
>
> - Even if a module is not possible to completely install via this method
> (e.g. It depends on a os component, that is too low level etc.) such
> modules should be as tightly integrated into the system as possible and
> the manual admin installation steps should be documented in a way, so that
> in a normal case no place but the module system needs to be accessed
Agree.
>
> - on the top level the system should be the same for all OSes
>
> - The work packages i see at the moment for the structure i have in mind
> would be:
>
> - Module repository: A CouchDB Instance, that hosts a collection of
> CouchDB Modules
See Kanso, above.
> - Futon User Interface to browse available Modules in the configured
> repositories, should be close to the Firefox plugin paradigms
That would be neat. Or a separate 'shopping' utility for CouchApps.
Sounds a bit like a combination between JChris' CouchApp Garden [3]
and Max Ogden's Chrome Couch application [4]. Crossing these would
Kanso would be really neat.
[3] https://github.com/jchris/garden
[4] https://github.com/maxogden/chromecouch
> - Module Manifest: Descriptive CouchDoc, that references the attached
> files or git repository, how files need to be fetched, packaged and what
> goes where, also version dependencies, description, rating, Module
> Category, needed system privileges (basically if you need to be a couch
> admin or if you need to be root on the os), needed configuration, and
> basic description of the configuration parameters
I'd really prefer to punt on low-level privileges and just call this
two separate things. The way to unify them, in my mind, is to create a
hooks system and then to make a native module that listens for all
hooks and delegates processing to design documents that indicate they
handle certain hooks. In other words, build the un-privileged module
system on top of the privileged one. Let administrators worry about
file systems when installing the privileged ones. It's really not so
bad. Sysadmins have been installing things for years, and actually
probably prefer to install stuff on the command line than with a POST
to /_replicate. The couch-config utility is there to facilitate this.
What we probably need is an extra location for user-installed add-on
applications that need to be added to the Erlang library path and used
for loading configured applications.
> - Futon User Interface to the installed modules, with options to enable,
> disable, uninstall, rate and configure
Yes again. I like the gist, but I don't know if it should be bound up
in Futon or not.
> - The tools for installation/uninstallation and enabling/disabling as
> well as packaging and dependency resolution
That sounds like figuring out a good way to structure the
configuration sections for modules. See the JIRA tickets I referenced
for some discussion of that.
>
> Open questions: What existing projects could be made use of, maybe there
> is something great for the general erlang world, maybe there is a project
> like zero install, that could be made use of, maybe that is not
> necessary, what are your thoughts?
I would also check out Agner: http://erlagner.org/
Basically, if we could get to the point where CouchDB administrators
could use Agner to install new pieces I'd consider that a huge win.
If, in the process, we get a modularized core and support for a
design-document-level hooks system I'd consider that an epic win. I
never say epic win. I would say it three times out loud.
>
> I wish you all a happy new year!
>
>
Thanks :)
-Randall
CouchDB Module System
Posted by Jan Johannes <ja...@ntr.io>.
Dear favorite Document-database developer community :-)
This is my first message to the mailing list, so please forgive me and
gently correct me if i am breaking any communication standards whatsoever
I am currently working on a WebDAV interface for CouchDB, that now gets
closer to its release date, somewhere in February or March and as i
approach that, I want to package everything in a proper CouchDB Module
format but the CouchDB Module system is, as i understood, in early
development or pre-development phase and needs a bit of work.
So as i work on a CouchDB Module anyways i would like to contribute to the
Module system or start kicking the work on that of.
Hence the following questions:
Who started working on such a system or would be interested in talking
about the design decisions that need to be done before actual coding can
start?
(I know at least Jan Lehnardt expressed interest, but i can't quite
remember if he started actual work or just had plans for that.)
If there is any documentation on planning such a system, could someone
please point me to the recourses?
Lastly here is my two cents on a few topics that need to be addressed:
- We should build the best module system there is for a database or
content management system to gain back a bit of the traction we lost
during the recent plateau phase of couchdb enthusiasm
- I think "module" is the preferable term, as extension, plugin or the
like sound like second class citizens, whereas module implies, that even a
core component of the system can be made into such an instance (Any other
suggestions?)
- A CouchDB module should be packaged as a stand alone couchdb-document,
with the yet to be specified manifest beeing the json structure of the
document and the files beeing the attachment.
- Discovery, Installation, Enabling, Disabling, Module-Specific
Configuration and Distribution should be part of Futon and introduce as
little new interface concepts as possible, also the futon interface
section "configuration" may be needed to be integrated and streamlined
quite nicely
- Even if a module is not possible to completely install via this method
(e.g. It depends on a os component, that is too low level etc.) such
modules should be as tightly integrated into the system as possible and
the manual admin installation steps should be documented in a way, so that
in a normal case no place but the module system needs to be accessed
- on the top level the system should be the same for all OSes
- The work packages i see at the moment for the structure i have in mind
would be:
- Module repository: A CouchDB Instance, that hosts a collection of
CouchDB Modules
- Futon User Interface to browse available Modules in the configured
repositories, should be close to the Firefox plugin paradigms
- Module Manifest: Descriptive CouchDoc, that references the attached
files or git repository, how files need to be fetched, packaged and what
goes where, also version dependencies, description, rating, Module
Category, needed system privileges (basically if you need to be a couch
admin or if you need to be root on the os), needed configuration, and
basic description of the configuration parameters
- Futon User Interface to the installed modules, with options to enable,
disable, uninstall, rate and configure
- The tools for installation/uninstallation and enabling/disabling as
well as packaging and dependency resolution
Open questions: What existing projects could be made use of, maybe there
is something great for the general erlang world, maybe there is a project
like zero install, that could be made use of, maybe that is not
necessary, what are your thoughts?
I wish you all a happy new year!
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Bob Dionne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178038#comment-13178038 ]
Bob Dionne commented on COUCHDB-1373:
-------------------------------------
Nick,
This seems reasonable, I'm sure there might be some issues to consider but if it's done the way the others were I don't see why not. I've run into this sort of thing before in systems where you want unique ids with a certain provenance so you assign each player an additional unique id that becomes part of the generated keys.
couch_uuids is a gen_server whose state is determined by an algorithm defined in the config file. I see no problem with adding another. If you could submit a patch it would be easier to assess or better yet a link to a source repo.
Best,
Bob
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick North updated COUCHDB-1373:
--------------------------------
Attachment: couch_uuids.patch
Patch to implement utc-machine_id uuid algorithm
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: couch_uuids.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408678#comment-13408678 ]
Nick North commented on COUCHDB-1373:
-------------------------------------
Bumping this as there were no comments on my pull request https://github.com/apache/couchdb/pull/28 which was intended to address it. If I'm going about this the wrong way, then please do let me know so that I can improve my approach. Many thanks.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick North updated COUCHDB-1373:
--------------------------------
Attachment: 0003-utc_id_suffix.patch
One more tweak: added tests of utc_id_suffix to share/www/script/test/uuids.js. Patch 0003-utc_id_suffix.patch includes this, and all previous changes.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Robert Newson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185690#comment-13185690 ]
Robert Newson commented on COUCHDB-1373:
----------------------------------------
grabbing the MAC by name is as simple as;
fun(Name) -> {ok, List} = inet:getifaddrs(), If = proplists:get_value(Name, List), proplists:get_value(hwaddr, If) end.
So I suggest the value in the .ini file is the name of the interface ("eth0", for example) and not a uuid, this will ensure that any live node is using a different value.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, 0002-utc_id_suffix.patch, 0003-utc_id_suffix.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Nick North (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick North updated COUCHDB-1373:
--------------------------------
Attachment: utc_machine_id.patch
Thanks for the comments. Whitespace is now fixed, I've commented default.ini.tpl.in, and moved machine_id to the uuids section of the file as that's less intrusive. Revised patch attached as utc_machine_id.patch.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Bob Dionne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bob Dionne updated COUCHDB-1373:
--------------------------------
Attachment: 0001-Add-etap-for-jira-1373.patch
test for couchdb-1373
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: 0001-Add-etap-for-jira-1373.patch, couch_uuids.patch, utc_machine_id.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1373) Time-ordered document ids including the database identity
Posted by "Paul Joseph Davis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178095#comment-13178095 ]
Paul Joseph Davis commented on COUCHDB-1373:
--------------------------------------------
Your whitespace is wonky. Indentation in Erlang code should be four spaces and not tabs. There's also some pure white-space changes.
Also, you should add a note in the default.ini.tpl.in file to indicate that the new algorithm exists as well as the fact that it requires the couchdb/machine_id setting to do what it's supposed to. I'm also hesitating if machine_id shouldn't be in the uuid section or not. Not super important but I find it a bit odd to have it there.
Also, I wonder if it'd be possible to generate some sort of default machine id if one isn't specified. Something like, a sha of all mac addresses in sorted order? Then again I'm not sure how hard it'd be to get that list from Erlang in a portable fashion.
> Time-ordered document ids including the database identity
> ----------------------------------------------------------
>
> Key: COUCHDB-1373
> URL: https://issues.apache.org/jira/browse/COUCHDB-1373
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Nick North
> Priority: Minor
> Labels: uuid
> Attachments: couch_uuids.patch
>
>
> This suggestion is for an enhancement to the document id generation algorithms in CouchDb. I am new to CouchDb, and this question addresses an old issue (https://issues.apache.org/jira/browse/COUCHDB-465) so please forgive me if I am retreading old ground.
> My application has a number of mutually replicating CouchDb instances and I would like document ids to be monotonically-increasing per-instance, and globally unique, and for the instance where the document was created to be determinable from the id. (To be more accurate - I don't need to know anything about the instance itself; just whether any two documents originated from the same instance.) The utc_random algorithm is not far from meeting these requirements, as ids are monotonic and almost certainly globally unique. However, the instance cannot be determined from the id, and there is a tiny chance of an id clash between two instances. Both of these issues could be solved if the random part of the id could be replaced with a suffix that is fixed in the ini file for each instance.
> To address this I have a modified version of couch_uuids.erl introducing a new utc_machine_id algorithm which reads a machine_id string from the ini file and then generates ids using an internal utc_suffix method that just appends the string to the usual utc 14-byte string. utc_random then also uses the utc_suffix method, but its suffix is the usual random byte string.
> However, it is obviously a nuisance to have to maintain a non-standard distribution, so I wondered if there is enough call for this sort of thing to make it a part of the standard distribution? If there is, I'd be very happy to make my code available for discussion/modification/inclusion. If there are good reasons why this is a bad idea, then I'd also be very interested to hear them so that I can rethink my ideas. (It happens that the privacy and guessability concerns raised in the original discussion do not apply in my case.) If this question has been beaten to death, then I'm sorry for bothering the group, and would be grateful if someone could point me to the discussions so that I can understand the issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira