You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/12/08 09:11:02 UTC
Slack digest for #general - 2018-12-08
2018-12-07 09:22:55 UTC - Maarten Tielemans: @Maarten Tielemans has joined the channel
----
2018-12-07 09:55:28 UTC - Maarten Tielemans: Hi All, somewhat similar question to above.
Running Apache Pulsar (currently) in standalone mode on an i3.large node.
I have formatted (as xfs) and mounted the NVMe disk. I do not have separate disks for journal and ledger (which one should be the NVMe?)
At times I see a big variance and huge spikes in tail latency. Wondering what my best way of debugging this would be? I need lower overall and tail latency ..
Furthermore, if I were to run Kafka standalone on the same instance/disk, my average latency is around 1 ms. As far as I understand, Pulsar standalone does not have the fsync requirement. Are there things which I should finetune?
Some extracts from the pulsar-perf log:
----
2018-12-07 09:55:37 UTC - Maarten Tielemans:
----
2018-12-07 10:10:07 UTC - Sijie Guo: @Maarten Tielemans: since your disk is mounted in /mnt/data, you need to point the standalone to use this directory `bin/pulsar standalone --bookkeeper-dir /mnt/data/bookkeeper`. also I would suggest not putting the binary distribution under /mnt/data
----
2018-12-07 10:13:09 UTC - Sijie Guo: (actually I think it might actually use the disk, since you put the binary in /mnt/data)
----
2018-12-07 10:19:52 UTC - Maarten Tielemans: @Sijie Guo made the changes you suggested, but doesn't seem to have a noticeable impact ; still avg around 3.x ms, 99.9 around 10ms and spikes in the 99.99
----
2018-12-07 10:30:51 UTC - Sijie Guo: @Maarten Tielemans: I see . the bookie journal group commit is set to 1ms, since you are sending 60 msg/second . almost every message is waiting for 1ms second at storage to commit, and there is 1ms batching time at client side as well.
I would suggest you test with 10000 messages/s and 100000 messages/s (basically changing your rate)
----
2018-12-07 10:32:12 UTC - Maarten Tielemans: @Sijie Guo is there a way to change that behaviour? The message rates and sizes I use are closer to my real use case
----
2018-12-07 10:53:26 UTC - Sijie Guo: can you add `journalFlushWhenQueueEmpty=true` to `conf/standalone.conf`?
and add `--batch-time-window 0` to the command you used for running pulsar-perf
----
2018-12-07 10:53:32 UTC - Sijie Guo: @Maarten Tielemans ^
----
2018-12-07 10:57:50 UTC - David Tinker: Does anyone have any idea why the ProducerImpl from the Java client (2.2.0) would get stuck in some kind of loop after creation but before any messages have been published? These 2 debugs are endlessly repeated and calls to producer.send(..) block indefinitely.
----
2018-12-07 10:57:59 UTC - David Tinker: 2018-12-07 12:52:16,776 [,] [pulsar-timer-4-1] DEBUG impl.ProducerImpl - [<persistent://public/feed-mentions/TWITTER_SAMPLE>] [sprout@porker] Batching the messages from the batch container from timer thread
2018-12-07 12:52:16,776 [,] [pulsar-timer-4-1] DEBUG impl.ProducerImpl - [<persistent://public/feed-mentions/TWITTER_SAMPLE>] [sprout@porker] Batching the messages from the batch container with 0 messages
----
2018-12-07 10:58:59 UTC - David Tinker: Exactly the same code is running fine in some other services.
----
2018-12-07 11:01:52 UTC - Sijie Guo: @David Tinker I haven’t heard anyone seen this issue. one possibility (just guessing here) - if there are dependency conflicts with your other dependencies and causing RuntimeException.
----
2018-12-07 11:38:48 UTC - David Tinker: Looks like its normal behaviour for that code to run every batchingMaxPublishDelayMicros
----
2018-12-07 12:19:00 UTC - David Tinker: I had a clash between `net.jpountz.lz4:lz4:1.2.0` and `org.lz4:lz4-java:1.5.0`. Maybe ProducerImpl should log failures in the `batchMessageAndSend catch (Throwable) { .. }` as errors instead of warnings? I would have found this quicker :slightly_smiling_face:
----
2018-12-07 12:21:08 UTC - Sijie Guo: yes that’s probably a good idea to catch throwable
----
2018-12-07 12:22:30 UTC - David Tinker: It does catch Throwable but only logs it as a warning so I missed it. When I turned on debugging it got drowned out. I should have had at least warnings on originally I suppose.
----
2018-12-07 12:23:10 UTC - Sijie Guo: oh I see I misread your comment :slightly_smiling_face:
----
2018-12-07 12:23:15 UTC - Sijie Guo: yes error is probably better
----
2018-12-07 12:39:46 UTC - Maarten Tielemans: @Sijie Guo the changes you mention bring the mean/med latency down to around 1.5-2ms ; so is a noticeable improvement. tail/max latency still spikes at times
----
2018-12-07 12:40:20 UTC - Maarten Tielemans: I assume that if I were to run two bookkeepers, and I use ack = 1, that would help my tail latency?
----
2018-12-07 12:41:10 UTC - Sijie Guo: yes
----
2018-12-07 12:48:57 UTC - Maarten Tielemans: is there value in splitting journal and ledger to two separate disks? can I do that with the standalone?
----
2018-12-07 13:01:51 UTC - jia zhai: Here is the related config, it should work with standalone
```
ledgerDirectories=data/bookkeeper/ledgers
journalDirectory=data/bookkeeper/journal
```
----
2018-12-07 13:41:57 UTC - Maarten Tielemans: I am not sure if I am correctly setting the above config. Can I add that to the `conf/standalone.conf` ? Asking because it does not seem to have effect
----
2018-12-07 13:47:09 UTC - Sijie Guo: 1) don’t think those settings take effect for standalone mode. Standalone mode is taking the directory setting from —bookkeeper-dir
2) multiple disks only change throughput. Doesn’t change latency.
3) for tail or max latency, I would suggest setting up Prometheus and Grafana, so you can get metrics to understand where the latency comes from.
----
2018-12-07 13:52:15 UTC - Maarten Tielemans: @Sijie Guo thanks for the feedback
----
2018-12-07 16:49:33 UTC - Melvin: @Melvin has joined the channel
----
2018-12-07 17:07:45 UTC - Melvin: Hi I am using the pulsar python client and have trouble subscribing. I am seeing these errors. Do any of you know what's going on?
```2018-12-07 12:05:45.866 ERROR ClientConnection:709 | [sourceip:57139 -> destip:6651] Failed partition-metadata lookup req_id: 1 error: 4
2018-12-07 12:05:45.866 ERROR ClientImpl:374 | Error Checking/Getting Partition Metadata while Subscribing- 5```
----
2018-12-07 17:26:47 UTC - Sanjeev Kulkarni: what does pulsar-admin topics stats <topicname> tell you?
----
2018-12-07 19:46:17 UTC - Beast in Black: @Melvin I have this exact problem when using the pulsar CPP client as well. If you check the broker logs you will see something like the following (even if you are not using partitioned topics):
```
20:39:38.028 [pulsar-io-21-2] ERROR org.apache.pulsar.broker.web.PulsarWebResource - Policies not found for <YOUR_NAMESPACE> namespace
20:39:38.033 [pulsar-io-21-2] WARN org.apache.pulsar.broker.service.ServerCnx - Failed to get Partitioned Metadata [/100.96.1.12:50564] <YOUR_NAMESPACE/YOUR_TOPIC>: Policies not found for <YOUR_NAMESPACE> namespaceorg.apache.pulsar.broker.web.RestException: Policies not found for <YOUR_NAMESPACE> namespace
<JAVA_EXCEPTION_FOLLOWS>
```
So far I have not received any responses here, nor been able to figure this out yet especially since I am not using partitioned topics or using explicit partitioning.
I did see in the pulsar CPP client test source code the following, and implemented it in my own code but it does not seem to solve the issue re the subscriber failure:
```
// handling dangling subscriptions
consumer.unsubscribe();
client.subscribe(topicName, subName, consumer);
```
One thing I have noticed is that it consistently reports these errors/failures when a namespace is created and you immediately try to subscribe to it. However, I have seen these errors occur sporadically even after an initial subscribe succeeds (I have the subscription in a retry thread).
----
2018-12-07 20:38:50 UTC - David Kjerrumgaard: @Melvin @Beast in Black What does pulsar-admin namespaces permissions <YOUR-TENANT>/<YOUR NAMESPACE> return?
----
2018-12-07 20:44:18 UTC - David Kjerrumgaard: what does pulsar-admin topics stats <topicname> tell you?
----
2018-12-07 20:45:23 UTC - David Kjerrumgaard: what does pulsar-admin topics stats-internal <topicname> return?
----
2018-12-07 20:49:16 UTC - Beast in Black: @David Kjerrumgaard I get an empty map for each of my namespaces: `{ }`
----
2018-12-07 20:51:52 UTC - Beast in Black: @David Kjerrumgaard stats and stats-internal both return the same thing:
```
Policies not found for <MYTENANT/MYNAMESPACE> namespace
Reason: Policies not found for <MYTENANT/MYNAMESPACE> namespace
command terminated with exit code 1
```
----
2018-12-07 20:55:27 UTC - David Kjerrumgaard: and the host at 100.96.1.12:50564 is up and reachable by your Pulsar broker?
----
2018-12-07 21:01:09 UTC - Beast in Black: @David Kjerrumgaard yep
----
2018-12-07 21:02:10 UTC - Beast in Black: I see that it does work, but every so often I get this problem and then my subscribers have conniptions
----
2018-12-07 21:08:09 UTC - Beast in Black: @David Kjerrumgaard and I checked that my tenants have the admin role set. here's what `tenants get` shows me:
```
{
"adminRoles" : [ "admin" ],
"allowedClusters" : [<MY_LIST_OF_REPLICATION_CLUSTERS> ]
}
```
----
2018-12-07 21:13:48 UTC - Beast in Black: @David Kjerrumgaard do I need to explicitly set permissions on the namespace for the `"admin"` user set above in the `tenants get` call result?
----
2018-12-07 21:17:07 UTC - David Kjerrumgaard: You should, but I don't think it will solve the issue you are experiencing.
----
2018-12-07 21:32:47 UTC - Beast in Black: @David Kjerrumgaard OK, I'll change my code to explicitly set permissions on the namespaces. I'll have to do this via a REST call to the admin interface from my C++ app since the pulsar CPP client sources do not seem to have any way to do this. I checked the documentation at the pulsar REST API docs but it does not specify what the contents of the `POST` payload for the call to `/admin/v2/namespaces/:tenant/:namespace/permissions/:role` ought to be - could you point me to where I could find that info or an example?
Also, I am willing to experiment with explicitly setting a reasonable default policy on the namespace - are there any examples/docs showing how to do this?
----
2018-12-07 21:37:01 UTC - David Kjerrumgaard: @Beast in Black Can we sync in 30 minutes?
----
2018-12-07 21:39:22 UTC - Beast in Black: @David Kjerrumgaard sure, no problem, and thank you
----
2018-12-07 21:57:21 UTC - Mike Card: @Matteo Merli well went in and added the following to my userdata:
```
# Set up nvme disks
sudo mkdir /mnt/bookie-journal
sudo mkdir /mnt/bookie-ledger
sudo mkfs.ext4 -E nodiscard /dev/nvme0n1
sudo mount -o discard /dev/nvme0n1 /mnt/bookie-journal
sudo mkfs.ext4 -E nodiscard /dev/nvme1n1
sudo mount -o discard /dev/nvme1n1 /mnt/bookie-ledger
```
<SNIP>
```
# configure Bookkeeper
cat <<EOF >> conf/bookkeeper.conf
zkServers=zookeeper-1.${ZookeeperEnsemble}:2181,zookeeper-2.${ZookeeperEnsemble}:2181,zookeeper-3.${ZookeeperEnsemble}:2181
journalDirectory=/mnt/bookie-journal
ledgerDirectory=/mnt/bookie-ledger
EOF
```
----
2018-12-07 21:58:17 UTC - Mike Card: Now on one of the Pulsar instances when I examine the content of bookeeper.conf I see this at the end:
```
# Other RocksDB specific tunables
dbStorage_rocksDB_writeBufferSizeMB=64
dbStorage_rocksDB_sstSizeInMB=64
dbStorage_rocksDB_blockSize=65536
dbStorage_rocksDB_bloomFilterBitsPerKey=10
dbStorage_rocksDB_numLevels=-1
dbStorage_rocksDB_numFilesInLevel0=4
dbStorage_rocksDB_maxSizeInLevel1MB=256
zkServers=zookeeper-1.emo3:2181,zookeeper-2.emo3:2181,zookeeper-3.emo3:2181
journalDirectory=/mnt/bookie-journal
ledgerDirectory=/mnt/bookie-ledger
```
----
2018-12-07 21:59:02 UTC - Mike Card: And if I list the contents of these dirs I see:
```
drwx------ 2 root root 16384 Dec 7 21:37 lost+found
[mcard@pulsar-0d712b2f6d8e3259e conf]$ ls -la /mnt/bookie*
/mnt/bookie-journal:
total 20
drwxr-xr-x 3 root root 4096 Dec 7 21:37 .
drwxr-xr-x 5 root root 63 Dec 7 21:37 ..
drwx------ 2 root root 16384 Dec 7 21:37 lost+found
/mnt/bookie-ledger:
total 20
drwxr-xr-x 3 root root 4096 Dec 7 21:37 .
drwxr-xr-x 5 root root 63 Dec 7 21:37 ..
drwx------ 2 root root 16384 Dec 7 21:37 lost+found
```
----
2018-12-07 21:59:49 UTC - Mike Card: All of this looks OK but behavior is identical to yesterday. There is no I/O to the nvme disks, it is all to the local disk so latency is terrible etc.
----
2018-12-07 21:59:58 UTC - David Kjerrumgaard: @Beast in Black <http://pulsar.apache.org/en/admin-rest-api/#operation/grantPermissionOnNamespace>
----
2018-12-07 22:01:18 UTC - Mike Card: Oh I just saw I should've used ledgerDirectories!
----
2018-12-07 22:01:38 UTC - Mike Card: Saw that in the original conf will fix that and see if that fixes this
----
2018-12-07 22:03:38 UTC - Matteo Merli: Yes
----
2018-12-07 22:04:22 UTC - Matteo Merli: Also, side note: xfs is generally preferred to ext4 (in terms of latency), not huge but noticeable especially under load
----
2018-12-07 22:07:48 UTC - Mike Card: oh OK will switch that too
----
2018-12-07 22:08:04 UTC - Mike Card: Otherwise do the userdata contents look right?
----
2018-12-07 22:11:59 UTC - Beast in Black: @David Kjerrumgaard thank you. Would the contents of the `POST` payload be something like `"admin": [ "produce", "consume"]` ? Is there a list of operations available? I basically want to be able to grant permissions for all operations, something like `"admin": [ "*" ]`
----
2018-12-07 22:15:26 UTC - David Kjerrumgaard: The REST API doesn't have the ns-isolation-policies listed as an option. So you may have to use the admin CLI instead. I am guessing that your goal is to automate this entire process, and that using the CLI would not be a viable solution for this. However, we should try setting this via the CLI just to confirm that it is indeed the fix we are looking for. Once we confirm that, we can work on automating it.
----
2018-12-07 22:15:27 UTC - David Kjerrumgaard: <http://pulsar.apache.org/docs/en/pulsar-admin/#ns-isolation-policy>
----
2018-12-07 22:21:22 UTC - Beast in Black: @David Kjerrumgaard yes, the goal is indeed to be able to dynamically create tenants/namespaces/topics and perform pub/sub from the C++ app. I don't have a K8s cluster right now to test the CLI (we use K8s on AWS and our app is a C++ app running in a pod, with the pulsar admin pod being separate) but the next time I deploy (later tonight) I will issue a CLI command to explicitly set the NS policy and see if it makes the partition metadata/policy errors go away.
I am also wondering if there is some way to specify a default policy - perhaps in the broker config - to be applied for each namespace that gets created by the app pod...that way, we can avoid having to muck about in the app itself
----
2018-12-07 22:27:19 UTC - David Kjerrumgaard: @Beast in Black <http://pulsar.apache.org/en/admin-rest-api/#operation/grantPermissionOnNamespace> is the API for the permissions, and you would want specify; tenant=, namespace=, role="admin", actions = produce
----
2018-12-07 22:27:28 UTC - David Kjerrumgaard: tenant=, namespace=, role="admin", actions = consume
----
2018-12-07 22:27:47 UTC - David Kjerrumgaard: ^^^ Just make two calls with different actions to achieve the same result...
----
2018-12-07 22:28:08 UTC - David Kjerrumgaard: since I am not 100% sure how it would handle an array of values
----
2018-12-07 22:28:51 UTC - David Kjerrumgaard: @Beast in Black I will get back to on the default policy settings, as I don't have an answer at the moment.
----
2018-12-07 22:34:21 UTC - Beast in Black: @David Kjerrumgaard no problem, and thank you so much for your help and patience. I will try the CLI option for the policy settings and report back here on my findings (I'll ping you so you can find the message). Thanks again!
----
2018-12-07 23:11:07 UTC - Matteo Merli: I mean, the 2 directories are empty :slightly_smiling_face:
----
2018-12-07 23:11:21 UTC - Matteo Merli: so it’s definitely not writing there
----
2018-12-07 23:14:05 UTC - Mike Card: Well made fixes now I see this:
```
[mcard@pulsar-096813e6006a55e68 conf]$ ls -la /mnt/bookie*
/mnt/bookie-journal:
total 0
drwxr-xr-x 2 root root 6 Dec 7 22:55 .
drwxr-xr-x 5 root root 63 Dec 7 22:55 ..
/mnt/bookie-ledger:
total 0
drwxr-xr-x 3 root root 21 Dec 7 23:08 .
drwxr-xr-x 5 root root 63 Dec 7 22:55 ..
drwxr-xr-x 4 root root 109 Dec 7 23:08 current
```
----
2018-12-07 23:14:24 UTC - Mike Card: ```
[mcard@pulsar-096813e6006a55e68 conf]$ ls -la /mnt/bookie-ledger/current/
total 16
drwxr-xr-x 4 root root 109 Dec 7 23:08 .
drwxr-xr-x 3 root root 21 Dec 7 23:08 ..
-rw-r--r-- 1 root root 1024 Dec 7 23:08 0.log
-rw-r--r-- 1 root root 0 Dec 7 23:08 1.log
-rw-r--r-- 1 root root 2 Dec 7 23:08 lastId
-rw-r--r-- 1 root root 16 Dec 7 23:08 lastMark
drwxr-xr-x 2 root root 123 Dec 7 23:08 ledgers
drwxr-xr-x 2 root root 123 Dec 7 23:08 locations
-rw-r--r-- 1 root root 185 Dec 7 23:08 VERSION
```
----
2018-12-07 23:14:36 UTC - Mike Card: so looks like ledger is being written to at expected place
----
2018-12-07 23:14:42 UTC - Matteo Merli: looks better nwo
----
2018-12-07 23:14:49 UTC - Matteo Merli: did the latency change?
----
2018-12-07 23:14:51 UTC - Mike Card: however performance is still really bad
----
2018-12-07 23:14:55 UTC - Mike Card: no
----
2018-12-07 23:15:21 UTC - Mike Card: looks like all I/O is still to local disk but that can't be given directory contents
----
2018-12-07 23:15:41 UTC - Matteo Merli: uhm, there’s something else then. In this setup it should be ~2 millis at 99pct
----
2018-12-07 23:21:14 UTC - Mike Card:
----
2018-12-07 23:21:25 UTC - Mike Card: That is bookkeeper.conf
----
2018-12-07 23:22:25 UTC - Mike Card:
----
2018-12-07 23:22:34 UTC - Mike Card: and that is broker.conf
----
2018-12-07 23:22:48 UTC - Mike Card: Those are the configs running up in AWS on i3.4xlarge instances
----
2018-12-07 23:26:42 UTC - Matteo Merli: journalDirectory is defined at the beginning of the file as well
----
2018-12-07 23:27:05 UTC - Matteo Merli: same for `ledgerDirectories` at line 351
----
2018-12-07 23:27:36 UTC - Mike Card: Yeah I was doing an append to the default configs, doesn't the second entry override the first?
----
2018-12-07 23:27:57 UTC - Matteo Merli: I don’t think so :slightly_smiling_face:
----
2018-12-07 23:29:01 UTC - Mike Card: Hmm thought it did, esp. since it seems to be writing the ledger in the right place. I'll remove the other entries and see if that matters
----
2018-12-07 23:55:16 UTC - Mike Card: Now would it be bad to have an ELB as the single URL for the pulsar cluster in the call to set up the ZK metadata, to wit:
pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper zookeeper-1.emo3:2181,zookeeper-2.emo3:2181,zookeeper-3.emo3:2181 \
--configuration-store zookeeper-1.emo3:2181,zookeeper-2.emo3:2181,zookeeper-3.emo3:2181 \
--web-service-url <http://emodb-pulsar-databus-pulsar-cluster-1.qa.us-east-1.nexus.bazaarvoice.com:8080> \
--web-service-url-tls <https://emodb-pulsar-databus-pulsar-cluster-1.qa.us-east-1.nexus.bazaarvoice.com:8443> \
--broker-service-url <pulsar://emodb-pulsar-databus-pulsar-cluster-1.qa.us-east-1.nexus.bazaarvoice.com:6650> \
--broker-service-url-tls <pulsar+ssl://emodb-pulsar-databus-pulsar-cluster-1.qa.us-east-1.nexus.bazaarvoice.com:6651>
where here <http://emodb-pulsar-databus-pulsar-cluster-1.qa.us-east-1.nexus.bazaarvoice.com|emodb-pulsar-databus-pulsar-cluster-1.qa.us-east-1.nexus.bazaarvoice.com> points to a load balancer? Would it be better to make these parameters a comma-separated list of the pulsar instances e.g.
pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-1 \
--zookeeper zookeeper-1.emo3:2181,zookeeper-2.emo3:2181,zookeeper-3.emo3:2181 \
--configuration-store zookeeper-1.emo3:2181,zookeeper-2.emo3:2181,zookeeper-3.emo3:2181 \
--web-service-url <http://pulsar1:8080>, <http://pulsar2:8080,http://pulsar3:8080,\>
--web-service-url-tls <https://pulsar1:8443,https://pulsar2:8443,https://pulsar3:8443> \
--broker-service-url <pulsar://pulsar1:6650,pulsar://pulsar2:6650,pulsar://pulsar3:6650> \
--broker-service-url-tls <pulsar://pulsar1:6651,pulsar://pulsar2:6651,pulsar://pulsar3:6651>
----
2018-12-08 00:12:01 UTC - Mike Card: Well results are better this time still not great though
----
2018-12-08 00:13:00 UTC - Mike Card:
----
2018-12-08 00:44:28 UTC - Beast in Black: Does anyone have an example of setting `ns-isolation-policy` via the apache pulsar command line? I looked at the docs and the java admin source code/tests but wasn't able to find anything that I could use...
----
2018-12-08 00:46:24 UTC - Beast in Black: To elaborate, any help with what the `--auto-failover-policy-params` and `--auto-failover-policy-type` params accept/expect would be greatly appreciated
----
2018-12-08 02:32:09 UTC - Mike Card: OK @Matteo Merli I went through those confs and found that zkServers was defined twice too, so I commented out the first one I think that made the difference latency is now running steady around 4 ms
----
2018-12-08 03:33:06 UTC - Samuel Sun: Hi any command to stop bookie(running background) gracefully ?
----
2018-12-08 04:28:32 UTC - Matteo Merli: @Samuel Sun if you started with `pulsar-daemon start bookie` you can do with `pulsar-daemon stop bookie`.
In any case, sending SIGTERM will trigger the graceful shutdown
----
2018-12-08 04:29:34 UTC - Matteo Merli: :+1: yes this numbers make more sense! :)
----
2018-12-08 04:56:57 UTC - Samuel Sun: got it, thanks
----
2018-12-08 09:05:28 UTC - Christophe Bornet: @Beast in Black I'm also testing namespace isolation. To set a policy I used `bin/pulsar-admin --admin-url <http://pulsar1-eu:8080> ns-isolation-policy set test ns-is-policy1 --auto-failover-policy-params min_limit=0,usage_threshold=100 --auto-failover-policy-type min_available --namespaces public/default --secondary pulsar*-us --primary pulsar*-eu`
----
2018-12-08 09:06:26 UTC - Christophe Bornet: Don't hesitate on sharing the results of your tests. I'll share mine.
----
2018-12-08 09:07:55 UTC - Christophe Bornet: Note that the min_available policy type is the only one currently acceptable
----
2018-12-08 09:09:19 UTC - Christophe Bornet: And providing min_limit and usage_threshold is mandatory for this policy
----