You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Ashish Chopra <as...@adobe.com.INVALID> on 2019/11/18 10:23:07 UTC
Resource-backed Passive Queue impl in Content Distribution
Hi,
I'm trying to better understand agent instances created via QueueDistributionAgentFactory [0], and the Resource-backed queues they leverage [1].
The Sling issue through which this implementation was added makes a mention of Active and Passive queues (as defined as [2]), with [1] being a Passive queue primarily useful for Reverse Distribution.
I hope to be able to leverage Resource-backed queues for forward-distribution scenarios, but currently it is not possible (given [1] and [2]). I'm working to understand existing "active" queue impls in sling-dist-core bundle [3] to have a better sense of what would it take to develop an "active" variant of ResourceQueue impl.
I have a few queries on the topic above:
* is there any documentation (beyond [2], that is) which explains the concept of Passive and Active queues (in context of SCD) in more detail?
* is there any background information on why ResourceQueue impl [1] is currently Passive? i.e., was it due to any known limitations, or a case of YAGNI?
I read through SLING-7754 and searched the archives, but couldn't get any pointers. Would be great if the community can help me discover more information on this topic.
Thanks
Ashish
[0] https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/agent/impl/QueueDistributionAgentFactory.java
[1] https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/queue/impl/resource/ResourceQueue.java
[2] https://issues.apache.org/jira/browse/SLING-7754?focusedCommentId=16522146
[3] https://github.com/apache/sling-org-apache-sling-distribution-core/blob/deb3d2ae/src/main/java/org/apache/sling/distribution/agent/impl/ForwardDistributionAgentFactory.java#L171-L177
RE: Resource-backed Passive Queue impl in Content Distribution
Posted by Ashish Chopra <as...@adobe.com.INVALID>.
Hi,
Thanks for the insight Marius.
> the first usecase was to implement a multi-datacenter sync...
> [publish instances use active queues and author] saved the packages in a passive queue for other datacenters to pull the updates.
> second usecase was to pause delivery of some queues [in case one endpoint is unavailable for a long time]
Above is indeed helpful to know!
I also understood from above that ResourceQueue was not implemented as an Active Queue because there was no such use case.
Do you think there's merit in developing an active variant of ResourceQueue implementation?
This will enable o.a.s.d.agent.impl.ForwardDistributionAgentFactory to use queues persisted in JCR without relying in Sling Jobs (for the reasons mentioned in SLING-7754).
The caveat here would be that a queue-processor attached to the "Active" variant of ResourceQueue impl can only work on a single instance in the cluster [0], but this would still be helpful since Resource-backed queues (in contrast to a "In Memory" or "On Disk" queue) will survive terminations of Sling instances running in containers without any affinity to disk.
I've opened SLING-8853 for that and I hope to be able to work on it.
Please let me know in case this sounds like a bad idea at the outset.
Regards
-Ashish
[0] https://sling.apache.org/documentation/bundles/scheduler-service-commons-scheduler.html#scheduling-the-job-just-once-in-a-cluster
[1] https://issues.apache.org/jira/browse/SLING-8853
-----Original Message-----
From: Marius Petria <mp...@adobe.com.INVALID>
Sent: Monday, November 18, 2019 8:10 PM
To: dev@sling.apache.org
Subject: Re: Resource-backed Passive Queue impl in Content Distribution
Hi,
I introduced the passive queues in SLING-4629 and I am trying to recall the exact context __.
An active queue is a queue for which a processor was defined and a passive queue is one for which no processor was defined.
If I recall correctly the first usecase was to implement a multi-datacenter sync. In a datacenter there was an author instance that coordinated the sync between publish instances in that datacenter using active queues but also saved the packages in a passive queue for other datacenters to pull the updates.
The second usecase was to pause delivery of some queues. Assume that one endpoint is down for a long time and you want to pause delivery to it, but still build the queue, then you make the queue for that endpoint passive.
Marius
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fsling-org-apache-sling-distribution-core%2Fcommit%2F0a6508e42034721a97eb2f86f9ede591f09db7fe&data=02%7C01%7Cashishc%40adobe.com%7C1b128bd347084327e2b308d76c354016%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096848275000581&sdata=r%2F3hTgHQ0Ba8J4hT9zLxstzRIdKZRNQxAVTq0l5AhFA%3D&reserved=0
On 18/11/2019, 12:23, "Ashish Chopra" <as...@adobe.com.INVALID> wrote:
Hi,
I'm trying to better understand agent instances created via QueueDistributionAgentFactory [0], and the Resource-backed queues they leverage [1].
The Sling issue through which this implementation was added makes a mention of Active and Passive queues (as defined as [2]), with [1] being a Passive queue primarily useful for Reverse Distribution.
I hope to be able to leverage Resource-backed queues for forward-distribution scenarios, but currently it is not possible (given [1] and [2]). I'm working to understand existing "active" queue impls in sling-dist-core bundle [3] to have a better sense of what would it take to develop an "active" variant of ResourceQueue impl.
I have a few queries on the topic above:
* is there any documentation (beyond [2], that is) which explains the concept of Passive and Active queues (in context of SCD) in more detail?
* is there any background information on why ResourceQueue impl [1] is currently Passive? i.e., was it due to any known limitations, or a case of YAGNI?
I read through SLING-7754 and searched the archives, but couldn't get any pointers. Would be great if the community can help me discover more information on this topic.
Thanks
Ashish
[0] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fsling-org-apache-sling-distribution-core%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fsling%2Fdistribution%2Fagent%2Fimpl%2FQueueDistributionAgentFactory.java&data=02%7C01%7Cashishc%40adobe.com%7C1b128bd347084327e2b308d76c354016%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096848275000581&sdata=K0f%2BgG7ERQ99CcSniWrY5hw3g6XtuyMCUkXwTbKTig8%3D&reserved=0
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fsling-org-apache-sling-distribution-core%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fsling%2Fdistribution%2Fqueue%2Fimpl%2Fresource%2FResourceQueue.java&data=02%7C01%7Cashishc%40adobe.com%7C1b128bd347084327e2b308d76c354016%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096848275000581&sdata=9X2WblTXcCUZLZjRyHlY%2F017WOplq2EwDvlzRqcgpLo%3D&reserved=0
[2] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSLING-7754%3FfocusedCommentId%3D16522146&data=02%7C01%7Cashishc%40adobe.com%7C1b128bd347084327e2b308d76c354016%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096848275010574&sdata=y%2FEkVWuIL4o%2FwRHaBty%2FyVDwqRhrY0tOoEKcD%2FjniUk%3D&reserved=0
[3] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fsling-org-apache-sling-distribution-core%2Fblob%2Fdeb3d2ae%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fsling%2Fdistribution%2Fagent%2Fimpl%2FForwardDistributionAgentFactory.java%23L171-L177&data=02%7C01%7Cashishc%40adobe.com%7C1b128bd347084327e2b308d76c354016%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096848275010574&sdata=33t61wifNoixJDcF8maGreR2mtoN5PAmsbTT8UrlN08%3D&reserved=0
Re: Resource-backed Passive Queue impl in Content Distribution
Posted by Marius Petria <mp...@adobe.com.INVALID>.
Hi,
I introduced the passive queues in SLING-4629 and I am trying to recall the exact context __.
An active queue is a queue for which a processor was defined and a passive queue is one for which no processor was defined.
If I recall correctly the first usecase was to implement a multi-datacenter sync. In a datacenter there was an author instance that coordinated the sync between publish instances in that datacenter using active queues but also saved the packages in a passive queue for other datacenters to pull the updates.
The second usecase was to pause delivery of some queues. Assume that one endpoint is down for a long time and you want to pause delivery to it, but still build the queue, then you make the queue for that endpoint passive.
Marius
https://github.com/apache/sling-org-apache-sling-distribution-core/commit/0a6508e42034721a97eb2f86f9ede591f09db7fe
On 18/11/2019, 12:23, "Ashish Chopra" <as...@adobe.com.INVALID> wrote:
Hi,
I'm trying to better understand agent instances created via QueueDistributionAgentFactory [0], and the Resource-backed queues they leverage [1].
The Sling issue through which this implementation was added makes a mention of Active and Passive queues (as defined as [2]), with [1] being a Passive queue primarily useful for Reverse Distribution.
I hope to be able to leverage Resource-backed queues for forward-distribution scenarios, but currently it is not possible (given [1] and [2]). I'm working to understand existing "active" queue impls in sling-dist-core bundle [3] to have a better sense of what would it take to develop an "active" variant of ResourceQueue impl.
I have a few queries on the topic above:
* is there any documentation (beyond [2], that is) which explains the concept of Passive and Active queues (in context of SCD) in more detail?
* is there any background information on why ResourceQueue impl [1] is currently Passive? i.e., was it due to any known limitations, or a case of YAGNI?
I read through SLING-7754 and searched the archives, but couldn't get any pointers. Would be great if the community can help me discover more information on this topic.
Thanks
Ashish
[0] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fsling-org-apache-sling-distribution-core%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fsling%2Fdistribution%2Fagent%2Fimpl%2FQueueDistributionAgentFactory.java&data=02%7C01%7Cmpetria%40adobe.com%7Cbf1f58a3e4f247ffeb4408d76c115729%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096694043583838&sdata=pTMcvSgxUdQSyRV3Fj0TjFZ8DF4HVHENRDAM60plVyE%3D&reserved=0
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fsling-org-apache-sling-distribution-core%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fsling%2Fdistribution%2Fqueue%2Fimpl%2Fresource%2FResourceQueue.java&data=02%7C01%7Cmpetria%40adobe.com%7Cbf1f58a3e4f247ffeb4408d76c115729%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096694043583838&sdata=BX8vB7fwK6Z4kHJkZBtitVwBZzm5KDgaxNPEUMgkE7w%3D&reserved=0
[2] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSLING-7754%3FfocusedCommentId%3D16522146&data=02%7C01%7Cmpetria%40adobe.com%7Cbf1f58a3e4f247ffeb4408d76c115729%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096694043583838&sdata=64mh6dnL%2BaS9oEWMQVfyBd0J1b9vpUs0KrMYD2qBYXI%3D&reserved=0
[3] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fsling-org-apache-sling-distribution-core%2Fblob%2Fdeb3d2ae%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fsling%2Fdistribution%2Fagent%2Fimpl%2FForwardDistributionAgentFactory.java%23L171-L177&data=02%7C01%7Cmpetria%40adobe.com%7Cbf1f58a3e4f247ffeb4408d76c115729%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637096694043583838&sdata=IlkJnzFPR1wS5IrnWjhk798a8eW41oXHWUiQm5xR17s%3D&reserved=0