You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@archiva.apache.org by "Kishore Gopalakrishna (JIRA)" <ji...@codehaus.org> on 2012/12/21 17:59:13 UTC
[jira] (MRM-1733) Add replication and fault tolerance using Helix
Kishore Gopalakrishna created MRM-1733:
------------------------------------------
Summary: Add replication and fault tolerance using Helix
Key: MRM-1733
URL: https://jira.codehaus.org/browse/MRM-1733
Project: Archiva
Issue Type: New Feature
Reporter: Kishore Gopalakrishna
As of today, Archiva does not support replication and does not support automatic fail over. Archiva has two main storage types, the files uploaded to the repository which uses file system for storage and metadata storage for which Jack Rabbit is used. Archiva also supports notification mechanism where a consumer can be notified of the changes in the repository.
In order to have fault tolerance and replication, we can have multiple archiva instances running for redundancy. One of them will be elected as the master and will accept writes/reads. And the remaining will be slaves and only serve reads. The slaves can get notified from the master of every change and it will apply the changes. When the master dies, one of the slaves will become the master and serve writes.
Apache Helix is a newly incubated project and provides the basic building blocks to add partition management, recovery from failure and cluster expansion with ease. I have built a sample prototype how one can build such a replicated file store using Helix. More information can be found here. http://helix.incubator.apache.org/recipes/rsync_replicated_file_store.html
I have used rsync for replication and apache jci module to detect file system changes to show case the recipe as a generic use case. However in case of Archiva, one can use the notification mechanism provided by archiva consumer for detecting changes and using archiva api's to fetch the changed files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MRM-1733) Add replication and fault tolerance using Helix
Posted by "Kishore Gopalakrishna (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MRM-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=316114#comment-316114 ]
Kishore Gopalakrishna edited comment on MRM-1733 at 12/21/12 11:01 AM:
-----------------------------------------------------------------------
>From Olivier,
Hi Kishore,
Thanks for your email and explanations !
@Others
As I'm involved in Helix incubator, my idea was to use it to a sync
mechanism with Archiva for artifacts.
Basically a primary/master instance where users deploy their artifacts
then n slave instances for reading artifacts.
Does it make sense to you ?
was (Author: kishoreg):
Hi Kishore,
Thanks for your email and explanations !
@Others
As I'm involved in Helix incubator, my idea was to use it to a sync
mechanism with Archiva for artifacts.
Basically a primary/master instance where users deploy their artifacts
then n slave instances for reading artifacts.
Does it make sense to you ?
> Add replication and fault tolerance using Helix
> -----------------------------------------------
>
> Key: MRM-1733
> URL: https://jira.codehaus.org/browse/MRM-1733
> Project: Archiva
> Issue Type: New Feature
> Reporter: Kishore Gopalakrishna
>
> As of today, Archiva does not support replication and does not support automatic fail over. Archiva has two main storage types, the files uploaded to the repository which uses file system for storage and metadata storage for which Jack Rabbit is used. Archiva also supports notification mechanism where a consumer can be notified of the changes in the repository.
> In order to have fault tolerance and replication, we can have multiple archiva instances running for redundancy. One of them will be elected as the master and will accept writes/reads. And the remaining will be slaves and only serve reads. The slaves can get notified from the master of every change and it will apply the changes. When the master dies, one of the slaves will become the master and serve writes.
> Apache Helix is a newly incubated project and provides the basic building blocks to add partition management, recovery from failure and cluster expansion with ease. I have built a sample prototype how one can build such a replicated file store using Helix. More information can be found here. http://helix.incubator.apache.org/recipes/rsync_replicated_file_store.html
> I have used rsync for replication and apache jci module to detect file system changes to show case the recipe as a generic use case. However in case of Archiva, one can use the notification mechanism provided by archiva consumer for detecting changes and using archiva api's to fetch the changed files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MRM-1733) Add replication and fault tolerance using Helix
Posted by "Kishore Gopalakrishna (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MRM-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=316114#comment-316114 ]
Kishore Gopalakrishna commented on MRM-1733:
--------------------------------------------
Hi Kishore,
Thanks for your email and explanations !
@Others
As I'm involved in Helix incubator, my idea was to use it to a sync
mechanism with Archiva for artifacts.
Basically a primary/master instance where users deploy their artifacts
then n slave instances for reading artifacts.
Does it make sense to you ?
> Add replication and fault tolerance using Helix
> -----------------------------------------------
>
> Key: MRM-1733
> URL: https://jira.codehaus.org/browse/MRM-1733
> Project: Archiva
> Issue Type: New Feature
> Reporter: Kishore Gopalakrishna
>
> As of today, Archiva does not support replication and does not support automatic fail over. Archiva has two main storage types, the files uploaded to the repository which uses file system for storage and metadata storage for which Jack Rabbit is used. Archiva also supports notification mechanism where a consumer can be notified of the changes in the repository.
> In order to have fault tolerance and replication, we can have multiple archiva instances running for redundancy. One of them will be elected as the master and will accept writes/reads. And the remaining will be slaves and only serve reads. The slaves can get notified from the master of every change and it will apply the changes. When the master dies, one of the slaves will become the master and serve writes.
> Apache Helix is a newly incubated project and provides the basic building blocks to add partition management, recovery from failure and cluster expansion with ease. I have built a sample prototype how one can build such a replicated file store using Helix. More information can be found here. http://helix.incubator.apache.org/recipes/rsync_replicated_file_store.html
> I have used rsync for replication and apache jci module to detect file system changes to show case the recipe as a generic use case. However in case of Archiva, one can use the notification mechanism provided by archiva consumer for detecting changes and using archiva api's to fetch the changed files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MRM-1733) Add replication and fault tolerance using Helix
Posted by "Kishore Gopalakrishna (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MRM-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=316117#comment-316117 ]
Kishore Gopalakrishna commented on MRM-1733:
--------------------------------------------
Thanks. Few things to add.
>From Kishore
* Replication in the recipe was done using rsync. Not sure if rsync is ok for Archiva. Its possible for Archiva to not use rsync. As I mentioned in my previous email, Archiva supports notification mechanism in the form of consumers. This means the slave instance can act as a consumer and master can push the notifications to slave. Each notification will contain the list of files changed. Once the slave gets the list of changes, it can request those files from master and save it locally. Slave will need to have ACL's to fetch all the files. This can make it independent of rsync.
* I have not considered the user information that is stored in database/ldap. Is it possible to enhance the notification mechanism to notify consumers of change in user information.
Thoughts ?
> Add replication and fault tolerance using Helix
> -----------------------------------------------
>
> Key: MRM-1733
> URL: https://jira.codehaus.org/browse/MRM-1733
> Project: Archiva
> Issue Type: New Feature
> Reporter: Kishore Gopalakrishna
>
> As of today, Archiva does not support replication and does not support automatic fail over. Archiva has two main storage types, the files uploaded to the repository which uses file system for storage and metadata storage for which Jack Rabbit is used. Archiva also supports notification mechanism where a consumer can be notified of the changes in the repository.
> In order to have fault tolerance and replication, we can have multiple archiva instances running for redundancy. One of them will be elected as the master and will accept writes/reads. And the remaining will be slaves and only serve reads. The slaves can get notified from the master of every change and it will apply the changes. When the master dies, one of the slaves will become the master and serve writes.
> Apache Helix is a newly incubated project and provides the basic building blocks to add partition management, recovery from failure and cluster expansion with ease. I have built a sample prototype how one can build such a replicated file store using Helix. More information can be found here. http://helix.incubator.apache.org/recipes/rsync_replicated_file_store.html
> I have used rsync for replication and apache jci module to detect file system changes to show case the recipe as a generic use case. However in case of Archiva, one can use the notification mechanism provided by archiva consumer for detecting changes and using archiva api's to fetch the changed files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MRM-1733) Add replication and fault tolerance using Helix
Posted by "Kishore Gopalakrishna (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MRM-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=316116#comment-316116 ]
Kishore Gopalakrishna commented on MRM-1733:
--------------------------------------------
Adrien Lecharpentier wrote,
IMO, it is a very good idea. I've already saw some config like this but
using proxy connector to "replicate" artifacts to archiva-instance close to
users (specific teams/dept). Using Helix could be a time saving and great
performance enhancement solution.
> Add replication and fault tolerance using Helix
> -----------------------------------------------
>
> Key: MRM-1733
> URL: https://jira.codehaus.org/browse/MRM-1733
> Project: Archiva
> Issue Type: New Feature
> Reporter: Kishore Gopalakrishna
>
> As of today, Archiva does not support replication and does not support automatic fail over. Archiva has two main storage types, the files uploaded to the repository which uses file system for storage and metadata storage for which Jack Rabbit is used. Archiva also supports notification mechanism where a consumer can be notified of the changes in the repository.
> In order to have fault tolerance and replication, we can have multiple archiva instances running for redundancy. One of them will be elected as the master and will accept writes/reads. And the remaining will be slaves and only serve reads. The slaves can get notified from the master of every change and it will apply the changes. When the master dies, one of the slaves will become the master and serve writes.
> Apache Helix is a newly incubated project and provides the basic building blocks to add partition management, recovery from failure and cluster expansion with ease. I have built a sample prototype how one can build such a replicated file store using Helix. More information can be found here. http://helix.incubator.apache.org/recipes/rsync_replicated_file_store.html
> I have used rsync for replication and apache jci module to detect file system changes to show case the recipe as a generic use case. However in case of Archiva, one can use the notification mechanism provided by archiva consumer for detecting changes and using archiva api's to fetch the changed files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira