You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mynewt.apache.org by will sanfilippo <wi...@runtime.io> on 2017/01/11 19:00:27 UTC
MBUF sizing for the bluetooth stack
Hello:
Since this has come up on a number of different occasions I wanted to send out an email which discusses how the nimble stack uses mbufs. This will be a controller-centric discussion but the concepts apply to the host as well.
A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs for networking stack packet data. A “packet” is simply a chain of mbufs with the first mbuf in the chain being a packet header mbuf and all others being “normal” mbufs. A packet header mbuf contains a mbuf header, a packet header and an optional user-defined header.
The length of the packet (i.e. all the data contained in all the mbuf chains) is stored in the packet header. Each individual mbuf in the chain also contains a length which is the length of the data in that mbuf. The sum of all the mbuf data lengths = length of packet.
The amount of overhead in an mbuf and its size determine the amount of data that can be carried in a mbuf. All mbufs have a 16-byte mbuf header. Packet header mbufs have an additional 8 bytes for the packet header structure and an optional user-data header. The nimble stack uses either an 8-byte or 12-byte user data header. If you turn on multi-advertising support, the user header is 12 bytes; otherwise it is 8 bytes. This means the total packet header mbuf overhead is 32 or 36 bytes.
The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg variables. Currently, there is one mbuf pool added to msys (MSYS_1) with a block size of 292 bytes.
Controller constraints:
The controller assumes that a certain minimum data size is available in a packet header mbuf. This size is equal to the largest advertising PDU, or 37 bytes, and must also contain the 2-byte LL PDU header (for a total of 39 bytes). Additionally, the controller requires an additional 4 bytes at the start of the packet header mbuf to prepend the HCI ACL data packet header. This means that the minimum mbuf size that can be allocated in any msys mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv). Since memory pools are always rounded up to the nearest 4 byte boundary, this means that the minimum size should be 76 (or 80) bytes.
For most applications that dont use large packets, setting the mbuf size to 80 should be fine as this will accommodate the typical BLE PDU and also meets the minimum requirement. If your application generally uses larger packets it might be benefical to allocate large mbufs as you dont lose the 16-byte overhead per mbuf.
Finally, here is an example of how many mbufs will be used by the controller for received packets. This assumes multi-advertising enabled (36 byte packet header overhead).
Example 1: PDU length = 251, msys_1_block_size = 80
Controller needs to store 251 + 2 = 253 total bytes.
Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64 bytes while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
Example 2: PDU length = 251, msys_1_block_size = 112
Total mbufs: 3 (72 + 96 + 85)
Hope this helps.
Re: MBUF sizing for the bluetooth stack
Posted by Simon Ratner <si...@proxy.co>.
For the benefit of anyone (incl. my future self) searching for this info:
In 1.1.0, sizeof(struct ble_mbuf_hdr) = 14 (22 with BLE_EXT_ADV), so
minimum mbuf is (38 + 4 + 39 PDU) = 81 (89 with BLE_EXT_ADV). Rounded up to
4-byte boundary: 84/92 bytes.
In 1.2.0, sizeof(struct ble_mbuf_hdr) = 14 (18 with BLE_EXT_ADV), so
minimum mbuf is (38 + 4 + 39 PDU) = 81 (85 with BLE_EXT_ADV). Rounded up to
4-byte boundary: 84/88 bytes.
With BLE_EXT_ADV, you should also consider that your BLE_EXT_ADV_MAX_SIZE
may be >31, affecting the size of your largest PDU, so adjust accordingly.
Similarly, with BLE_LL_CFG_FEAT_DATA_LEN_EXT.
On Wed, Jan 11, 2017 at 4:17 PM, will sanfilippo <wi...@runtime.io> wrote:
> Well, yes, there are “definitions” for these things. They are in various
> places but they are there. Using them might get a bit tricky as you have
> mentioned; not sure. You would have to make sure the right header files get
> included in the proper places...
>
> Anyway, here are the definitions:
> os mbuf header: sizeof(struct os_mbuf). Size = 16
> os mbuf packet header: sizeof(struct os_mbuf_pkthdr) Size = 8
> user header: sizeof(struct ble_mbuf_hdr) Size = 8 or 12
> The HCI ACL data header: BLE_HCI_DATA_HSDR_SZ. 4 bytes
> The LL PDU header: BLE_LL_PDU_HDR_LEN. 2 bytes
>
> I would always make the size a multiple of 4 but the code should do that
> for you; I just like to do it so the size you see in the syscfg variable is
> the actual memory block size.
>
> Another thing I should mention: you should never add a buffer pool to msys
> smaller than the minimum size I mentioned if you are using the controller.
> This is something we will address in the future but for now it would be
> bad. :-)
>
>
>
> > On Jan 11, 2017, at 3:49 PM, Simon Ratner <si...@proxy.co> wrote:
> >
> > Thanks for the detailed write-up, Will - very useful.
> >
> > Are there defines for these things?
> > Ideally, if I want a payload size of N, I'd like to specify in
> syscfg.yml:
> >
> > MSYS_1_BLOCK_SIZE: '(N + MBUF_HEADER + PKT_HEADER + LL_OVERHEAD +
> ...)'
> >
> > And magically have optimally-sized buffers.
> >
> >
> > On Wed, Jan 11, 2017 at 11:00 AM, will sanfilippo <wi...@runtime.io>
> wrote:
> >
> >> Hello:
> >>
> >> Since this has come up on a number of different occasions I wanted to
> send
> >> out an email which discusses how the nimble stack uses mbufs. This will
> be
> >> a controller-centric discussion but the concepts apply to the host as
> well.
> >>
> >> A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs for
> >> networking stack packet data. A “packet” is simply a chain of mbufs with
> >> the first mbuf in the chain being a packet header mbuf and all others
> being
> >> “normal” mbufs. A packet header mbuf contains a mbuf header, a packet
> >> header and an optional user-defined header.
> >>
> >> The length of the packet (i.e. all the data contained in all the mbuf
> >> chains) is stored in the packet header. Each individual mbuf in the
> chain
> >> also contains a length which is the length of the data in that mbuf. The
> >> sum of all the mbuf data lengths = length of packet.
> >>
> >> The amount of overhead in an mbuf and its size determine the amount of
> >> data that can be carried in a mbuf. All mbufs have a 16-byte mbuf
> header.
> >> Packet header mbufs have an additional 8 bytes for the packet header
> >> structure and an optional user-data header. The nimble stack uses
> either an
> >> 8-byte or 12-byte user data header. If you turn on multi-advertising
> >> support, the user header is 12 bytes; otherwise it is 8 bytes. This
> means
> >> the total packet header mbuf overhead is 32 or 36 bytes.
> >>
> >> The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg
> >> variables. Currently, there is one mbuf pool added to msys (MSYS_1)
> with a
> >> block size of 292 bytes.
> >>
> >> Controller constraints:
> >> The controller assumes that a certain minimum data size is available in
> a
> >> packet header mbuf. This size is equal to the largest advertising PDU,
> or
> >> 37 bytes, and must also contain the 2-byte LL PDU header (for a total
> of 39
> >> bytes). Additionally, the controller requires an additional 4 bytes at
> the
> >> start of the packet header mbuf to prepend the HCI ACL data packet
> header.
> >> This means that the minimum mbuf size that can be allocated in any msys
> >> mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv).
> >> Since memory pools are always rounded up to the nearest 4 byte boundary,
> >> this means that the minimum size should be 76 (or 80) bytes.
> >>
> >> For most applications that dont use large packets, setting the mbuf size
> >> to 80 should be fine as this will accommodate the typical BLE PDU and
> also
> >> meets the minimum requirement. If your application generally uses larger
> >> packets it might be benefical to allocate large mbufs as you dont lose
> the
> >> 16-byte overhead per mbuf.
> >>
> >> Finally, here is an example of how many mbufs will be used by the
> >> controller for received packets. This assumes multi-advertising enabled
> (36
> >> byte packet header overhead).
> >>
> >> Example 1: PDU length = 251, msys_1_block_size = 80
> >>
> >> Controller needs to store 251 + 2 = 253 total bytes.
> >>
> >> Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
> >> Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
> >> Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64 bytes
> >> while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
> >>
> >> Example 2: PDU length = 251, msys_1_block_size = 112
> >> Total mbufs: 3 (72 + 96 + 85)
> >>
> >> Hope this helps.
> >>
> >>
> >>
> >>
> >>
> >>
>
>
Re: MBUF sizing for the bluetooth stack
Posted by will sanfilippo <wi...@runtime.io>.
That is a good question. I should let Chris answer this one as he knows for sure. I suspect you will have a chain of mbufs but I would have to look over the code to be sure.
> On Jan 19, 2017, at 3:46 AM, Simon Ratner <si...@proxy.co> wrote:
>
> Hi Will,
>
> A related question: how does this map to large ATT_MTU and fragmented
> packets at the L2CAP level (assuming no data length extension)? Does each
> fragment get its own mbuf, which are then chained together, or does the
> entire packet get reassembled into a single mbuf if there is room?
>
>
>
> On Wed, Jan 11, 2017 at 4:57 PM, will sanfilippo <wi...@runtime.io> wrote:
>
>> Yes; 76 or 80. Note that I have not actually tested with 80 byte mbuf
>> blocks. That is the theory though :-)
>>
>>> On Jan 11, 2017, at 4:31 PM, Simon Ratner <si...@proxy.co> wrote:
>>>
>>> Got it; by minimum size you mean the 76/80 bytes?
>>>
>>> On Wed, Jan 11, 2017 at 4:17 PM, will sanfilippo <wi...@runtime.io>
>> wrote:
>>>
>>>> Well, yes, there are “definitions” for these things. They are in various
>>>> places but they are there. Using them might get a bit tricky as you have
>>>> mentioned; not sure. You would have to make sure the right header files
>> get
>>>> included in the proper places...
>>>>
>>>> Anyway, here are the definitions:
>>>> os mbuf header: sizeof(struct os_mbuf). Size = 16
>>>> os mbuf packet header: sizeof(struct os_mbuf_pkthdr) Size = 8
>>>> user header: sizeof(struct ble_mbuf_hdr) Size = 8 or 12
>>>> The HCI ACL data header: BLE_HCI_DATA_HSDR_SZ. 4 bytes
>>>> The LL PDU header: BLE_LL_PDU_HDR_LEN. 2 bytes
>>>>
>>>> I would always make the size a multiple of 4 but the code should do that
>>>> for you; I just like to do it so the size you see in the syscfg
>> variable is
>>>> the actual memory block size.
>>>>
>>>> Another thing I should mention: you should never add a buffer pool to
>> msys
>>>> smaller than the minimum size I mentioned if you are using the
>> controller.
>>>> This is something we will address in the future but for now it would be
>>>> bad. :-)
>>>>
>>>>
>>>>
>>>>> On Jan 11, 2017, at 3:49 PM, Simon Ratner <si...@proxy.co> wrote:
>>>>>
>>>>> Thanks for the detailed write-up, Will - very useful.
>>>>>
>>>>> Are there defines for these things?
>>>>> Ideally, if I want a payload size of N, I'd like to specify in
>>>> syscfg.yml:
>>>>>
>>>>> MSYS_1_BLOCK_SIZE: '(N + MBUF_HEADER + PKT_HEADER + LL_OVERHEAD +
>>>> ...)'
>>>>>
>>>>> And magically have optimally-sized buffers.
>>>>>
>>>>>
>>>>> On Wed, Jan 11, 2017 at 11:00 AM, will sanfilippo <wi...@runtime.io>
>>>> wrote:
>>>>>
>>>>>> Hello:
>>>>>>
>>>>>> Since this has come up on a number of different occasions I wanted to
>>>> send
>>>>>> out an email which discusses how the nimble stack uses mbufs. This
>> will
>>>> be
>>>>>> a controller-centric discussion but the concepts apply to the host as
>>>> well.
>>>>>>
>>>>>> A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs
>> for
>>>>>> networking stack packet data. A “packet” is simply a chain of mbufs
>> with
>>>>>> the first mbuf in the chain being a packet header mbuf and all others
>>>> being
>>>>>> “normal” mbufs. A packet header mbuf contains a mbuf header, a packet
>>>>>> header and an optional user-defined header.
>>>>>>
>>>>>> The length of the packet (i.e. all the data contained in all the mbuf
>>>>>> chains) is stored in the packet header. Each individual mbuf in the
>>>> chain
>>>>>> also contains a length which is the length of the data in that mbuf.
>> The
>>>>>> sum of all the mbuf data lengths = length of packet.
>>>>>>
>>>>>> The amount of overhead in an mbuf and its size determine the amount of
>>>>>> data that can be carried in a mbuf. All mbufs have a 16-byte mbuf
>>>> header.
>>>>>> Packet header mbufs have an additional 8 bytes for the packet header
>>>>>> structure and an optional user-data header. The nimble stack uses
>>>> either an
>>>>>> 8-byte or 12-byte user data header. If you turn on multi-advertising
>>>>>> support, the user header is 12 bytes; otherwise it is 8 bytes. This
>>>> means
>>>>>> the total packet header mbuf overhead is 32 or 36 bytes.
>>>>>>
>>>>>> The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg
>>>>>> variables. Currently, there is one mbuf pool added to msys (MSYS_1)
>>>> with a
>>>>>> block size of 292 bytes.
>>>>>>
>>>>>> Controller constraints:
>>>>>> The controller assumes that a certain minimum data size is available
>> in
>>>> a
>>>>>> packet header mbuf. This size is equal to the largest advertising PDU,
>>>> or
>>>>>> 37 bytes, and must also contain the 2-byte LL PDU header (for a total
>>>> of 39
>>>>>> bytes). Additionally, the controller requires an additional 4 bytes at
>>>> the
>>>>>> start of the packet header mbuf to prepend the HCI ACL data packet
>>>> header.
>>>>>> This means that the minimum mbuf size that can be allocated in any
>> msys
>>>>>> mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv).
>>>>>> Since memory pools are always rounded up to the nearest 4 byte
>> boundary,
>>>>>> this means that the minimum size should be 76 (or 80) bytes.
>>>>>>
>>>>>> For most applications that dont use large packets, setting the mbuf
>> size
>>>>>> to 80 should be fine as this will accommodate the typical BLE PDU and
>>>> also
>>>>>> meets the minimum requirement. If your application generally uses
>> larger
>>>>>> packets it might be benefical to allocate large mbufs as you dont lose
>>>> the
>>>>>> 16-byte overhead per mbuf.
>>>>>>
>>>>>> Finally, here is an example of how many mbufs will be used by the
>>>>>> controller for received packets. This assumes multi-advertising
>> enabled
>>>> (36
>>>>>> byte packet header overhead).
>>>>>>
>>>>>> Example 1: PDU length = 251, msys_1_block_size = 80
>>>>>>
>>>>>> Controller needs to store 251 + 2 = 253 total bytes.
>>>>>>
>>>>>> Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
>>>>>> Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
>>>>>> Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64
>> bytes
>>>>>> while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
>>>>>>
>>>>>> Example 2: PDU length = 251, msys_1_block_size = 112
>>>>>> Total mbufs: 3 (72 + 96 + 85)
>>>>>>
>>>>>> Hope this helps.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: MBUF sizing for the bluetooth stack
Posted by Christopher Collins <cc...@apache.org>.
On Sun, Jan 22, 2017 at 09:16:59AM -0800, Christopher Collins wrote:
> There are some optimizations that could be done here. In most cases,
> the host is interested in the full attribute value. The host should
> probably allocate GATT read mbufs using ble_hs_mbuf_att_pkt() in these
> cases so that it can just send the mbuf after the application has filled
> it.
Btw, I went ahead and implemented / pushed this optimization.
Chris
Re: MBUF sizing for the bluetooth stack
Posted by Christopher Collins <cc...@apache.org>.
On Fri, Jan 20, 2017 at 02:27:41PM -0800, Simon Ratner wrote:
> I am talking about GATT reads, where the mbuf is pre-allocated for the app
> to fill in:
>
> - static int
> - gatt_svr_chr_access_gap(
> - uint16_t conn_handle,
> - uint16_t attr_handle,
> - struct ble_gatt_access_ctxt *ctxt, void *arg)
> - {
> - ...
> - os_mbuf_append(ctxt->om, data, len);
> - ...
> - }
> -
> - Similarly, GATT read ops triggered by ble_gatt_chr_updated for sending
> out notifications.
These mbufs never actually get sent to the controller, so they don't
need the extra leading space. After the application fills one of these
mbufs with a characteristic's value, the host copies the relevant data
into a properly allocated mbuf and sends the copy.
This may seem wasteful (in fact, that's what I'm thinking as I write
this!), but there is at least some logic here. In the case of a GATT
read, the peer may read from different offsets of the characteristic.
In an effort to simplify the API, I decided the requested offset should
be hidden from the application. Instead, the application always
provides the full characteristic value, and the host copies the
requested portion out of the user-filled mbuf and into the ACL data
packet.
There are some optimizations that could be done here. In most cases,
the host is interested in the full attribute value. The host should
probably allocate GATT read mbufs using ble_hs_mbuf_att_pkt() in these
cases so that it can just send the mbuf after the application has filled
it.
Alternatively, the host could specify the offset and length being read,
and require the application to provide only the requested range of
bytes. This would be a little messier, but would be the most mbuf
efficient.
I'm thinking we should do the first optimization immediately. I don't
believe it would be much work, and I don't see any downsides. I'll take
a closer look at it today.
Chris
Re: MBUF sizing for the bluetooth stack
Posted by Simon Ratner <si...@proxy.co>.
I am talking about GATT reads, where the mbuf is pre-allocated for the app
to fill in:
- static int
- gatt_svr_chr_access_gap(
- uint16_t conn_handle,
- uint16_t attr_handle,
- struct ble_gatt_access_ctxt *ctxt, void *arg)
- {
- ...
- os_mbuf_append(ctxt->om, data, len);
- ...
- }
-
- Similarly, GATT read ops triggered by ble_gatt_chr_updated for sending
out notifications.
-
On 20 Jan. 2017 1:48 pm, "Christopher Collins" <cc...@apache.org> wrote:
> On Fri, Jan 20, 2017 at 01:21:20PM -0800, Simon Ratner wrote:
> > On Fri, Jan 20, 2017 at 9:01 AM, Christopher Collins <
> ccollins@apache.org>
> > wrote:
> > > On Thu, Jan 19, 2017 at 11:57:01PM -0800, Simon Ratner wrote:
> > > > When allocating mbufs for the payload, is there something I should
> do to
> > > > reserve enough leading space for the ACL header to make sure host
> doesn't
> > > > need to re-allocate it?
> > >
> > > Yes - you should use ble_hs_mbuf_att_pkt() to allocate all your mbufs.
> > > This function ensures the resulting mbuf has sufficient leading space
> > > for:
> > > * ACL data header
> > > * Basic L2CAP header
> > > * Four bytes for an ATT command
> > >
> >
> > And I assume the mbuf passed into gatt read handlers (ctxt->om) has been
> > allocated that way? I only need to worry about the mbufs I allocate
> myself,
> > such as for notify payload.
>
> The mbufs handed up from the host are typically allocated by something
> else (e.g., for the combined host-controller, it is the controller which
> allocates them). If you pass these mbufs back to the host, there is a
> chance the host will need to allocate an additional buffer while
> prepending to the chain. The host stripped the headers from the front
> of the mbuf before handing it to the application, so there will be
> sufficient leading space to add them back. However, the problem is the
> variable length ATT command. Your outgoing command may require more
> "ATT bytes" than the one you received.
>
> Do you often pass these mbufs back to the host? I thought that would be
> an uncommon use case.
>
> Chris
>
Re: MBUF sizing for the bluetooth stack
Posted by Christopher Collins <cc...@apache.org>.
On Fri, Jan 20, 2017 at 01:21:20PM -0800, Simon Ratner wrote:
> On Fri, Jan 20, 2017 at 9:01 AM, Christopher Collins <cc...@apache.org>
> wrote:
> > On Thu, Jan 19, 2017 at 11:57:01PM -0800, Simon Ratner wrote:
> > > When allocating mbufs for the payload, is there something I should do to
> > > reserve enough leading space for the ACL header to make sure host doesn't
> > > need to re-allocate it?
> >
> > Yes - you should use ble_hs_mbuf_att_pkt() to allocate all your mbufs.
> > This function ensures the resulting mbuf has sufficient leading space
> > for:
> > * ACL data header
> > * Basic L2CAP header
> > * Four bytes for an ATT command
> >
>
> And I assume the mbuf passed into gatt read handlers (ctxt->om) has been
> allocated that way? I only need to worry about the mbufs I allocate myself,
> such as for notify payload.
The mbufs handed up from the host are typically allocated by something
else (e.g., for the combined host-controller, it is the controller which
allocates them). If you pass these mbufs back to the host, there is a
chance the host will need to allocate an additional buffer while
prepending to the chain. The host stripped the headers from the front
of the mbuf before handing it to the application, so there will be
sufficient leading space to add them back. However, the problem is the
variable length ATT command. Your outgoing command may require more
"ATT bytes" than the one you received.
Do you often pass these mbufs back to the host? I thought that would be
an uncommon use case.
Chris
Re: MBUF sizing for the bluetooth stack
Posted by Simon Ratner <si...@proxy.co>.
On Fri, Jan 20, 2017 at 9:01 AM, Christopher Collins <cc...@apache.org>
wrote:
> On Thu, Jan 19, 2017 at 11:57:01PM -0800, Simon Ratner wrote:
> > Thanks Chris,
> >
> > It appears to me that there is questionable benefit to having mbufs sized
> > larger than the largest L2CAP fragment size (plus overhead), i.e. the 80
> > bytes that Will mentioned. Is that a reasonable statement, or am I
> missing
> > something?
> >
> > For incoming data, you always waste memory with larger mbufs, and for
> > outgoing data host will take longer to free the memory (since you can't
> > free the payload mbuf until the last fragment, as opposed to freeing
> > smaller mbufs as you go), and you don't save on the number of copies in
> the
> > host. You will save something on mbuf allocations and mbuf header
> overhead
> > in the app as you are generating the payload, though.
>
> I agree with your analysis. This is just for BLE data, of course. If
> you use msys for other things (e.g., an alternative to malloc for
> internal use), that obviously has an affect on the optimum msys buffer
> size.
>
> > When allocating mbufs for the payload, is there something I should do to
> > reserve enough leading space for the ACL header to make sure host doesn't
> > need to re-allocate it?
>
> Yes - you should use ble_hs_mbuf_att_pkt() to allocate all your mbufs.
> This function ensures the resulting mbuf has sufficient leading space
> for:
> * ACL data header
> * Basic L2CAP header
> * Four bytes for an ATT command
>
And I assume the mbuf passed into gatt read handlers (ctxt->om) has been
allocated that way? I only need to worry about the mbufs I allocate myself,
such as for notify payload.
> The last item (four bytes) is imprecise because different ATT commands
> have different payload sizes, and this function doesn't know which
> command you'll be sending. Four bytes is the most the host would never
> need to prepend to application attribute data.
>
> We should expose some more functions from
> net/nimble/host/src/ble_hs_mbuf.c that give the user more control over
> the amount of leading space. This would be useful for when the
> application wants to send an ATT command that doesn't need the full four
> bytes.
>
> Just to be clear - if you allocate an mbuf that lacks sufficient leading
> space (e.g., via os_msys_get_pkthdr()) and pass it to the host, the host
> won't reallocate and copy the entire chain; it will just allocate a
> single buffer and prepend it to the chain. This is still wasteful, but
> it can be completely avoided by using the ble_hs_mbuf_att_pkt()
> function.
>
> > Also, at least in theory, it sounds like you could size mbufs to match
> the
> > fragment exactly -- or pre-fragment the mbuf chain as you are generating
> > the payload -- and have zero copies in the host. Could be useful in a
> > low-memory situation, if the host was smart enough to take advantage of
> > that?
>
> The host isn't smart enough :). The complication arises from the
> "os_mbuf_pkthdr" that the leading buffer contains. The presence of this
> header causes the first buffer to have less capacity than subsequent
> buffers. The fragmentation procedure never frees the chain's leading
> buffer. The assumption is that the data is packed in the mbuf chain,
> and that the second buffer doesn't have sufficient leading space to
> contain the pkthdr.
>
> A smarter procedure might check how much unused space the second buffer
> contains. If there is sufficient room for the pkthdr, the procedure
> would move the data to make room for the pkthdr at the front of the
> buffer, then copy the pkthdr into the buffer. After this, the leading
> buffer could be freed. This might be worth looking into.
>
> Thanks,
> Chris
>
Re: MBUF sizing for the bluetooth stack
Posted by Christopher Collins <cc...@apache.org>.
On Thu, Jan 19, 2017 at 11:57:01PM -0800, Simon Ratner wrote:
> Thanks Chris,
>
> It appears to me that there is questionable benefit to having mbufs sized
> larger than the largest L2CAP fragment size (plus overhead), i.e. the 80
> bytes that Will mentioned. Is that a reasonable statement, or am I missing
> something?
>
> For incoming data, you always waste memory with larger mbufs, and for
> outgoing data host will take longer to free the memory (since you can't
> free the payload mbuf until the last fragment, as opposed to freeing
> smaller mbufs as you go), and you don't save on the number of copies in the
> host. You will save something on mbuf allocations and mbuf header overhead
> in the app as you are generating the payload, though.
I agree with your analysis. This is just for BLE data, of course. If
you use msys for other things (e.g., an alternative to malloc for
internal use), that obviously has an affect on the optimum msys buffer
size.
> When allocating mbufs for the payload, is there something I should do to
> reserve enough leading space for the ACL header to make sure host doesn't
> need to re-allocate it?
Yes - you should use ble_hs_mbuf_att_pkt() to allocate all your mbufs.
This function ensures the resulting mbuf has sufficient leading space
for:
* ACL data header
* Basic L2CAP header
* Four bytes for an ATT command
The last item (four bytes) is imprecise because different ATT commands
have different payload sizes, and this function doesn't know which
command you'll be sending. Four bytes is the most the host would never
need to prepend to application attribute data.
We should expose some more functions from
net/nimble/host/src/ble_hs_mbuf.c that give the user more control over
the amount of leading space. This would be useful for when the
application wants to send an ATT command that doesn't need the full four
bytes.
Just to be clear - if you allocate an mbuf that lacks sufficient leading
space (e.g., via os_msys_get_pkthdr()) and pass it to the host, the host
won't reallocate and copy the entire chain; it will just allocate a
single buffer and prepend it to the chain. This is still wasteful, but
it can be completely avoided by using the ble_hs_mbuf_att_pkt()
function.
> Also, at least in theory, it sounds like you could size mbufs to match the
> fragment exactly -- or pre-fragment the mbuf chain as you are generating
> the payload -- and have zero copies in the host. Could be useful in a
> low-memory situation, if the host was smart enough to take advantage of
> that?
The host isn't smart enough :). The complication arises from the
"os_mbuf_pkthdr" that the leading buffer contains. The presence of this
header causes the first buffer to have less capacity than subsequent
buffers. The fragmentation procedure never frees the chain's leading
buffer. The assumption is that the data is packed in the mbuf chain,
and that the second buffer doesn't have sufficient leading space to
contain the pkthdr.
A smarter procedure might check how much unused space the second buffer
contains. If there is sufficient room for the pkthdr, the procedure
would move the data to make room for the pkthdr at the front of the
buffer, then copy the pkthdr into the buffer. After this, the leading
buffer could be freed. This might be worth looking into.
Thanks,
Chris
Re: MBUF sizing for the bluetooth stack
Posted by will sanfilippo <wi...@runtime.io>.
Simon:
I think you are pretty much correct; generally you are better off with smaller size mbufs. However, there are cases where larger mbufs are better (for example, a very large portion of your data packets are large).
> On Jan 19, 2017, at 11:57 PM, Simon Ratner <si...@proxy.co> wrote:
>
> Thanks Chris,
>
> It appears to me that there is questionable benefit to having mbufs sized
> larger than the largest L2CAP fragment size (plus overhead), i.e. the 80
> bytes that Will mentioned. Is that a reasonable statement, or am I missing
> something?
>
> For incoming data, you always waste memory with larger mbufs, and for
> outgoing data host will take longer to free the memory (since you can't
> free the payload mbuf until the last fragment, as opposed to freeing
> smaller mbufs as you go), and you don't save on the number of copies in the
> host. You will save something on mbuf allocations and mbuf header overhead
> in the app as you are generating the payload, though.
>
> When allocating mbufs for the payload, is there something I should do to
> reserve enough leading space for the ACL header to make sure host doesn't
> need to re-allocate it?
>
> Also, at least in theory, it sounds like you could size mbufs to match the
> fragment exactly -- or pre-fragment the mbuf chain as you are generating
> the payload -- and have zero copies in the host. Could be useful in a
> low-memory situation, if the host was smart enough to take advantage of
> that?
>
>
>
>
> On Thu, Jan 19, 2017 at 11:13 AM, Christopher Collins <cc...@apache.org>
> wrote:
>
>> On Thu, Jan 19, 2017 at 10:57:58AM -0800, Christopher Collins wrote:
>>> On Thu, Jan 19, 2017 at 03:46:49AM -0800, Simon Ratner wrote:
>>>> A related question: how does this map to large ATT_MTU and fragmented
>>>> packets at the L2CAP level (assuming no data length extension)? Does
>> each
>>>> fragment get its own mbuf, which are then chained together, or does the
>>>> entire packet get reassembled into a single mbuf if there is room?
>>>
>>> If the host needs to send a large packet, it packs the payload into an
>>> mbuf chain. By "packs," I mean each buffer holds as much data as
>>> possible with no regard to the maximum L2CAP fragment size.
>>>
>>> When the host sends an L2CAP fragment, it splits the fragment payload
>>> off from the front of the mbuf chain, constructs an ACL data packet, and
>>> sends it to the controller. If a buffer at the front of mbuf can be
>>> freed, now that data has been removed, the host frees it.
>>>
>>> If you are interested, the function which handles fragmentation and
>>> freeing is mem_split_frag() (util/mem/src/mem.c).
>>
>> I rushed this response a bit, and there are some important details I
>> neglected.
>>
>> * For the final L2CAP fragment in a packet, the host doesn't
>> do an allocating or copying. Instead, it just prepends an ACL data
>> header to the mbuf chain and sends it to the controller.
>>
>> * For all L2CAP fragments *other than the last*, the host allocates an
>> additional mbuf chain to hold the ACL data packet. The host then copies
>> the fragment data into this new chain, sends it, and frees buffers from
>> the front of the original chain if possible. The number of buffers that
>> get allocated for the fragment depends on how the maximum L2CAP fragment
>> size compares to the msys mbuf size. If an msys mbuf buffer has
>> sufficient capacity for a maximum size L2CAP fragment, then only one
>> buffer will get allocated. If the mbuf capacity is less, the chain that
>> gets allocated will consist of multiple buffers.
>>
>> * An L2CAP fragment mbuf chain contains the following:
>> * mbuf pkthdr (8 bytes)
>> * HCI ACL data header (4 bytes)
>> * Basic L2CAP header (4 bytes)
>> * Payload (varies)
>>
>> * For incoming data, the host does not do any packing. Each L2CAP
>> fragment is simply chained together.
>>
Re: MBUF sizing for the bluetooth stack
Posted by Simon Ratner <si...@proxy.co>.
Thanks Chris,
It appears to me that there is questionable benefit to having mbufs sized
larger than the largest L2CAP fragment size (plus overhead), i.e. the 80
bytes that Will mentioned. Is that a reasonable statement, or am I missing
something?
For incoming data, you always waste memory with larger mbufs, and for
outgoing data host will take longer to free the memory (since you can't
free the payload mbuf until the last fragment, as opposed to freeing
smaller mbufs as you go), and you don't save on the number of copies in the
host. You will save something on mbuf allocations and mbuf header overhead
in the app as you are generating the payload, though.
When allocating mbufs for the payload, is there something I should do to
reserve enough leading space for the ACL header to make sure host doesn't
need to re-allocate it?
Also, at least in theory, it sounds like you could size mbufs to match the
fragment exactly -- or pre-fragment the mbuf chain as you are generating
the payload -- and have zero copies in the host. Could be useful in a
low-memory situation, if the host was smart enough to take advantage of
that?
On Thu, Jan 19, 2017 at 11:13 AM, Christopher Collins <cc...@apache.org>
wrote:
> On Thu, Jan 19, 2017 at 10:57:58AM -0800, Christopher Collins wrote:
> > On Thu, Jan 19, 2017 at 03:46:49AM -0800, Simon Ratner wrote:
> > > A related question: how does this map to large ATT_MTU and fragmented
> > > packets at the L2CAP level (assuming no data length extension)? Does
> each
> > > fragment get its own mbuf, which are then chained together, or does the
> > > entire packet get reassembled into a single mbuf if there is room?
> >
> > If the host needs to send a large packet, it packs the payload into an
> > mbuf chain. By "packs," I mean each buffer holds as much data as
> > possible with no regard to the maximum L2CAP fragment size.
> >
> > When the host sends an L2CAP fragment, it splits the fragment payload
> > off from the front of the mbuf chain, constructs an ACL data packet, and
> > sends it to the controller. If a buffer at the front of mbuf can be
> > freed, now that data has been removed, the host frees it.
> >
> > If you are interested, the function which handles fragmentation and
> > freeing is mem_split_frag() (util/mem/src/mem.c).
>
> I rushed this response a bit, and there are some important details I
> neglected.
>
> * For the final L2CAP fragment in a packet, the host doesn't
> do an allocating or copying. Instead, it just prepends an ACL data
> header to the mbuf chain and sends it to the controller.
>
> * For all L2CAP fragments *other than the last*, the host allocates an
> additional mbuf chain to hold the ACL data packet. The host then copies
> the fragment data into this new chain, sends it, and frees buffers from
> the front of the original chain if possible. The number of buffers that
> get allocated for the fragment depends on how the maximum L2CAP fragment
> size compares to the msys mbuf size. If an msys mbuf buffer has
> sufficient capacity for a maximum size L2CAP fragment, then only one
> buffer will get allocated. If the mbuf capacity is less, the chain that
> gets allocated will consist of multiple buffers.
>
> * An L2CAP fragment mbuf chain contains the following:
> * mbuf pkthdr (8 bytes)
> * HCI ACL data header (4 bytes)
> * Basic L2CAP header (4 bytes)
> * Payload (varies)
>
> * For incoming data, the host does not do any packing. Each L2CAP
> fragment is simply chained together.
>
Re: MBUF sizing for the bluetooth stack
Posted by Christopher Collins <cc...@apache.org>.
On Thu, Jan 19, 2017 at 10:57:58AM -0800, Christopher Collins wrote:
> On Thu, Jan 19, 2017 at 03:46:49AM -0800, Simon Ratner wrote:
> > A related question: how does this map to large ATT_MTU and fragmented
> > packets at the L2CAP level (assuming no data length extension)? Does each
> > fragment get its own mbuf, which are then chained together, or does the
> > entire packet get reassembled into a single mbuf if there is room?
>
> If the host needs to send a large packet, it packs the payload into an
> mbuf chain. By "packs," I mean each buffer holds as much data as
> possible with no regard to the maximum L2CAP fragment size.
>
> When the host sends an L2CAP fragment, it splits the fragment payload
> off from the front of the mbuf chain, constructs an ACL data packet, and
> sends it to the controller. If a buffer at the front of mbuf can be
> freed, now that data has been removed, the host frees it.
>
> If you are interested, the function which handles fragmentation and
> freeing is mem_split_frag() (util/mem/src/mem.c).
I rushed this response a bit, and there are some important details I
neglected.
* For the final L2CAP fragment in a packet, the host doesn't
do an allocating or copying. Instead, it just prepends an ACL data
header to the mbuf chain and sends it to the controller.
* For all L2CAP fragments *other than the last*, the host allocates an
additional mbuf chain to hold the ACL data packet. The host then copies
the fragment data into this new chain, sends it, and frees buffers from
the front of the original chain if possible. The number of buffers that
get allocated for the fragment depends on how the maximum L2CAP fragment
size compares to the msys mbuf size. If an msys mbuf buffer has
sufficient capacity for a maximum size L2CAP fragment, then only one
buffer will get allocated. If the mbuf capacity is less, the chain that
gets allocated will consist of multiple buffers.
* An L2CAP fragment mbuf chain contains the following:
* mbuf pkthdr (8 bytes)
* HCI ACL data header (4 bytes)
* Basic L2CAP header (4 bytes)
* Payload (varies)
* For incoming data, the host does not do any packing. Each L2CAP
fragment is simply chained together.
Re: MBUF sizing for the bluetooth stack
Posted by Christopher Collins <cc...@apache.org>.
On Thu, Jan 19, 2017 at 03:46:49AM -0800, Simon Ratner wrote:
> A related question: how does this map to large ATT_MTU and fragmented
> packets at the L2CAP level (assuming no data length extension)? Does each
> fragment get its own mbuf, which are then chained together, or does the
> entire packet get reassembled into a single mbuf if there is room?
If the host needs to send a large packet, it packs the payload into an
mbuf chain. By "packs," I mean each buffer holds as much data as
possible with no regard to the maximum L2CAP fragment size.
When the host sends an L2CAP fragment, it splits the fragment payload
off from the front of the mbuf chain, constructs an ACL data packet, and
sends it to the controller. If a buffer at the front of mbuf can be
freed, now that data has been removed, the host frees it.
If you are interested, the function which handles fragmentation and
freeing is mem_split_frag() (util/mem/src/mem.c).
Chris
Re: MBUF sizing for the bluetooth stack
Posted by Simon Ratner <si...@proxy.co>.
Hi Will,
A related question: how does this map to large ATT_MTU and fragmented
packets at the L2CAP level (assuming no data length extension)? Does each
fragment get its own mbuf, which are then chained together, or does the
entire packet get reassembled into a single mbuf if there is room?
On Wed, Jan 11, 2017 at 4:57 PM, will sanfilippo <wi...@runtime.io> wrote:
> Yes; 76 or 80. Note that I have not actually tested with 80 byte mbuf
> blocks. That is the theory though :-)
>
> > On Jan 11, 2017, at 4:31 PM, Simon Ratner <si...@proxy.co> wrote:
> >
> > Got it; by minimum size you mean the 76/80 bytes?
> >
> > On Wed, Jan 11, 2017 at 4:17 PM, will sanfilippo <wi...@runtime.io>
> wrote:
> >
> >> Well, yes, there are “definitions” for these things. They are in various
> >> places but they are there. Using them might get a bit tricky as you have
> >> mentioned; not sure. You would have to make sure the right header files
> get
> >> included in the proper places...
> >>
> >> Anyway, here are the definitions:
> >> os mbuf header: sizeof(struct os_mbuf). Size = 16
> >> os mbuf packet header: sizeof(struct os_mbuf_pkthdr) Size = 8
> >> user header: sizeof(struct ble_mbuf_hdr) Size = 8 or 12
> >> The HCI ACL data header: BLE_HCI_DATA_HSDR_SZ. 4 bytes
> >> The LL PDU header: BLE_LL_PDU_HDR_LEN. 2 bytes
> >>
> >> I would always make the size a multiple of 4 but the code should do that
> >> for you; I just like to do it so the size you see in the syscfg
> variable is
> >> the actual memory block size.
> >>
> >> Another thing I should mention: you should never add a buffer pool to
> msys
> >> smaller than the minimum size I mentioned if you are using the
> controller.
> >> This is something we will address in the future but for now it would be
> >> bad. :-)
> >>
> >>
> >>
> >>> On Jan 11, 2017, at 3:49 PM, Simon Ratner <si...@proxy.co> wrote:
> >>>
> >>> Thanks for the detailed write-up, Will - very useful.
> >>>
> >>> Are there defines for these things?
> >>> Ideally, if I want a payload size of N, I'd like to specify in
> >> syscfg.yml:
> >>>
> >>> MSYS_1_BLOCK_SIZE: '(N + MBUF_HEADER + PKT_HEADER + LL_OVERHEAD +
> >> ...)'
> >>>
> >>> And magically have optimally-sized buffers.
> >>>
> >>>
> >>> On Wed, Jan 11, 2017 at 11:00 AM, will sanfilippo <wi...@runtime.io>
> >> wrote:
> >>>
> >>>> Hello:
> >>>>
> >>>> Since this has come up on a number of different occasions I wanted to
> >> send
> >>>> out an email which discusses how the nimble stack uses mbufs. This
> will
> >> be
> >>>> a controller-centric discussion but the concepts apply to the host as
> >> well.
> >>>>
> >>>> A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs
> for
> >>>> networking stack packet data. A “packet” is simply a chain of mbufs
> with
> >>>> the first mbuf in the chain being a packet header mbuf and all others
> >> being
> >>>> “normal” mbufs. A packet header mbuf contains a mbuf header, a packet
> >>>> header and an optional user-defined header.
> >>>>
> >>>> The length of the packet (i.e. all the data contained in all the mbuf
> >>>> chains) is stored in the packet header. Each individual mbuf in the
> >> chain
> >>>> also contains a length which is the length of the data in that mbuf.
> The
> >>>> sum of all the mbuf data lengths = length of packet.
> >>>>
> >>>> The amount of overhead in an mbuf and its size determine the amount of
> >>>> data that can be carried in a mbuf. All mbufs have a 16-byte mbuf
> >> header.
> >>>> Packet header mbufs have an additional 8 bytes for the packet header
> >>>> structure and an optional user-data header. The nimble stack uses
> >> either an
> >>>> 8-byte or 12-byte user data header. If you turn on multi-advertising
> >>>> support, the user header is 12 bytes; otherwise it is 8 bytes. This
> >> means
> >>>> the total packet header mbuf overhead is 32 or 36 bytes.
> >>>>
> >>>> The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg
> >>>> variables. Currently, there is one mbuf pool added to msys (MSYS_1)
> >> with a
> >>>> block size of 292 bytes.
> >>>>
> >>>> Controller constraints:
> >>>> The controller assumes that a certain minimum data size is available
> in
> >> a
> >>>> packet header mbuf. This size is equal to the largest advertising PDU,
> >> or
> >>>> 37 bytes, and must also contain the 2-byte LL PDU header (for a total
> >> of 39
> >>>> bytes). Additionally, the controller requires an additional 4 bytes at
> >> the
> >>>> start of the packet header mbuf to prepend the HCI ACL data packet
> >> header.
> >>>> This means that the minimum mbuf size that can be allocated in any
> msys
> >>>> mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv).
> >>>> Since memory pools are always rounded up to the nearest 4 byte
> boundary,
> >>>> this means that the minimum size should be 76 (or 80) bytes.
> >>>>
> >>>> For most applications that dont use large packets, setting the mbuf
> size
> >>>> to 80 should be fine as this will accommodate the typical BLE PDU and
> >> also
> >>>> meets the minimum requirement. If your application generally uses
> larger
> >>>> packets it might be benefical to allocate large mbufs as you dont lose
> >> the
> >>>> 16-byte overhead per mbuf.
> >>>>
> >>>> Finally, here is an example of how many mbufs will be used by the
> >>>> controller for received packets. This assumes multi-advertising
> enabled
> >> (36
> >>>> byte packet header overhead).
> >>>>
> >>>> Example 1: PDU length = 251, msys_1_block_size = 80
> >>>>
> >>>> Controller needs to store 251 + 2 = 253 total bytes.
> >>>>
> >>>> Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
> >>>> Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
> >>>> Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64
> bytes
> >>>> while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
> >>>>
> >>>> Example 2: PDU length = 251, msys_1_block_size = 112
> >>>> Total mbufs: 3 (72 + 96 + 85)
> >>>>
> >>>> Hope this helps.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: MBUF sizing for the bluetooth stack
Posted by will sanfilippo <wi...@runtime.io>.
Yes; 76 or 80. Note that I have not actually tested with 80 byte mbuf blocks. That is the theory though :-)
> On Jan 11, 2017, at 4:31 PM, Simon Ratner <si...@proxy.co> wrote:
>
> Got it; by minimum size you mean the 76/80 bytes?
>
> On Wed, Jan 11, 2017 at 4:17 PM, will sanfilippo <wi...@runtime.io> wrote:
>
>> Well, yes, there are “definitions” for these things. They are in various
>> places but they are there. Using them might get a bit tricky as you have
>> mentioned; not sure. You would have to make sure the right header files get
>> included in the proper places...
>>
>> Anyway, here are the definitions:
>> os mbuf header: sizeof(struct os_mbuf). Size = 16
>> os mbuf packet header: sizeof(struct os_mbuf_pkthdr) Size = 8
>> user header: sizeof(struct ble_mbuf_hdr) Size = 8 or 12
>> The HCI ACL data header: BLE_HCI_DATA_HSDR_SZ. 4 bytes
>> The LL PDU header: BLE_LL_PDU_HDR_LEN. 2 bytes
>>
>> I would always make the size a multiple of 4 but the code should do that
>> for you; I just like to do it so the size you see in the syscfg variable is
>> the actual memory block size.
>>
>> Another thing I should mention: you should never add a buffer pool to msys
>> smaller than the minimum size I mentioned if you are using the controller.
>> This is something we will address in the future but for now it would be
>> bad. :-)
>>
>>
>>
>>> On Jan 11, 2017, at 3:49 PM, Simon Ratner <si...@proxy.co> wrote:
>>>
>>> Thanks for the detailed write-up, Will - very useful.
>>>
>>> Are there defines for these things?
>>> Ideally, if I want a payload size of N, I'd like to specify in
>> syscfg.yml:
>>>
>>> MSYS_1_BLOCK_SIZE: '(N + MBUF_HEADER + PKT_HEADER + LL_OVERHEAD +
>> ...)'
>>>
>>> And magically have optimally-sized buffers.
>>>
>>>
>>> On Wed, Jan 11, 2017 at 11:00 AM, will sanfilippo <wi...@runtime.io>
>> wrote:
>>>
>>>> Hello:
>>>>
>>>> Since this has come up on a number of different occasions I wanted to
>> send
>>>> out an email which discusses how the nimble stack uses mbufs. This will
>> be
>>>> a controller-centric discussion but the concepts apply to the host as
>> well.
>>>>
>>>> A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs for
>>>> networking stack packet data. A “packet” is simply a chain of mbufs with
>>>> the first mbuf in the chain being a packet header mbuf and all others
>> being
>>>> “normal” mbufs. A packet header mbuf contains a mbuf header, a packet
>>>> header and an optional user-defined header.
>>>>
>>>> The length of the packet (i.e. all the data contained in all the mbuf
>>>> chains) is stored in the packet header. Each individual mbuf in the
>> chain
>>>> also contains a length which is the length of the data in that mbuf. The
>>>> sum of all the mbuf data lengths = length of packet.
>>>>
>>>> The amount of overhead in an mbuf and its size determine the amount of
>>>> data that can be carried in a mbuf. All mbufs have a 16-byte mbuf
>> header.
>>>> Packet header mbufs have an additional 8 bytes for the packet header
>>>> structure and an optional user-data header. The nimble stack uses
>> either an
>>>> 8-byte or 12-byte user data header. If you turn on multi-advertising
>>>> support, the user header is 12 bytes; otherwise it is 8 bytes. This
>> means
>>>> the total packet header mbuf overhead is 32 or 36 bytes.
>>>>
>>>> The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg
>>>> variables. Currently, there is one mbuf pool added to msys (MSYS_1)
>> with a
>>>> block size of 292 bytes.
>>>>
>>>> Controller constraints:
>>>> The controller assumes that a certain minimum data size is available in
>> a
>>>> packet header mbuf. This size is equal to the largest advertising PDU,
>> or
>>>> 37 bytes, and must also contain the 2-byte LL PDU header (for a total
>> of 39
>>>> bytes). Additionally, the controller requires an additional 4 bytes at
>> the
>>>> start of the packet header mbuf to prepend the HCI ACL data packet
>> header.
>>>> This means that the minimum mbuf size that can be allocated in any msys
>>>> mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv).
>>>> Since memory pools are always rounded up to the nearest 4 byte boundary,
>>>> this means that the minimum size should be 76 (or 80) bytes.
>>>>
>>>> For most applications that dont use large packets, setting the mbuf size
>>>> to 80 should be fine as this will accommodate the typical BLE PDU and
>> also
>>>> meets the minimum requirement. If your application generally uses larger
>>>> packets it might be benefical to allocate large mbufs as you dont lose
>> the
>>>> 16-byte overhead per mbuf.
>>>>
>>>> Finally, here is an example of how many mbufs will be used by the
>>>> controller for received packets. This assumes multi-advertising enabled
>> (36
>>>> byte packet header overhead).
>>>>
>>>> Example 1: PDU length = 251, msys_1_block_size = 80
>>>>
>>>> Controller needs to store 251 + 2 = 253 total bytes.
>>>>
>>>> Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
>>>> Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
>>>> Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64 bytes
>>>> while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
>>>>
>>>> Example 2: PDU length = 251, msys_1_block_size = 112
>>>> Total mbufs: 3 (72 + 96 + 85)
>>>>
>>>> Hope this helps.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
Re: MBUF sizing for the bluetooth stack
Posted by Simon Ratner <si...@proxy.co>.
Got it; by minimum size you mean the 76/80 bytes?
On Wed, Jan 11, 2017 at 4:17 PM, will sanfilippo <wi...@runtime.io> wrote:
> Well, yes, there are “definitions” for these things. They are in various
> places but they are there. Using them might get a bit tricky as you have
> mentioned; not sure. You would have to make sure the right header files get
> included in the proper places...
>
> Anyway, here are the definitions:
> os mbuf header: sizeof(struct os_mbuf). Size = 16
> os mbuf packet header: sizeof(struct os_mbuf_pkthdr) Size = 8
> user header: sizeof(struct ble_mbuf_hdr) Size = 8 or 12
> The HCI ACL data header: BLE_HCI_DATA_HSDR_SZ. 4 bytes
> The LL PDU header: BLE_LL_PDU_HDR_LEN. 2 bytes
>
> I would always make the size a multiple of 4 but the code should do that
> for you; I just like to do it so the size you see in the syscfg variable is
> the actual memory block size.
>
> Another thing I should mention: you should never add a buffer pool to msys
> smaller than the minimum size I mentioned if you are using the controller.
> This is something we will address in the future but for now it would be
> bad. :-)
>
>
>
> > On Jan 11, 2017, at 3:49 PM, Simon Ratner <si...@proxy.co> wrote:
> >
> > Thanks for the detailed write-up, Will - very useful.
> >
> > Are there defines for these things?
> > Ideally, if I want a payload size of N, I'd like to specify in
> syscfg.yml:
> >
> > MSYS_1_BLOCK_SIZE: '(N + MBUF_HEADER + PKT_HEADER + LL_OVERHEAD +
> ...)'
> >
> > And magically have optimally-sized buffers.
> >
> >
> > On Wed, Jan 11, 2017 at 11:00 AM, will sanfilippo <wi...@runtime.io>
> wrote:
> >
> >> Hello:
> >>
> >> Since this has come up on a number of different occasions I wanted to
> send
> >> out an email which discusses how the nimble stack uses mbufs. This will
> be
> >> a controller-centric discussion but the concepts apply to the host as
> well.
> >>
> >> A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs for
> >> networking stack packet data. A “packet” is simply a chain of mbufs with
> >> the first mbuf in the chain being a packet header mbuf and all others
> being
> >> “normal” mbufs. A packet header mbuf contains a mbuf header, a packet
> >> header and an optional user-defined header.
> >>
> >> The length of the packet (i.e. all the data contained in all the mbuf
> >> chains) is stored in the packet header. Each individual mbuf in the
> chain
> >> also contains a length which is the length of the data in that mbuf. The
> >> sum of all the mbuf data lengths = length of packet.
> >>
> >> The amount of overhead in an mbuf and its size determine the amount of
> >> data that can be carried in a mbuf. All mbufs have a 16-byte mbuf
> header.
> >> Packet header mbufs have an additional 8 bytes for the packet header
> >> structure and an optional user-data header. The nimble stack uses
> either an
> >> 8-byte or 12-byte user data header. If you turn on multi-advertising
> >> support, the user header is 12 bytes; otherwise it is 8 bytes. This
> means
> >> the total packet header mbuf overhead is 32 or 36 bytes.
> >>
> >> The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg
> >> variables. Currently, there is one mbuf pool added to msys (MSYS_1)
> with a
> >> block size of 292 bytes.
> >>
> >> Controller constraints:
> >> The controller assumes that a certain minimum data size is available in
> a
> >> packet header mbuf. This size is equal to the largest advertising PDU,
> or
> >> 37 bytes, and must also contain the 2-byte LL PDU header (for a total
> of 39
> >> bytes). Additionally, the controller requires an additional 4 bytes at
> the
> >> start of the packet header mbuf to prepend the HCI ACL data packet
> header.
> >> This means that the minimum mbuf size that can be allocated in any msys
> >> mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv).
> >> Since memory pools are always rounded up to the nearest 4 byte boundary,
> >> this means that the minimum size should be 76 (or 80) bytes.
> >>
> >> For most applications that dont use large packets, setting the mbuf size
> >> to 80 should be fine as this will accommodate the typical BLE PDU and
> also
> >> meets the minimum requirement. If your application generally uses larger
> >> packets it might be benefical to allocate large mbufs as you dont lose
> the
> >> 16-byte overhead per mbuf.
> >>
> >> Finally, here is an example of how many mbufs will be used by the
> >> controller for received packets. This assumes multi-advertising enabled
> (36
> >> byte packet header overhead).
> >>
> >> Example 1: PDU length = 251, msys_1_block_size = 80
> >>
> >> Controller needs to store 251 + 2 = 253 total bytes.
> >>
> >> Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
> >> Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
> >> Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64 bytes
> >> while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
> >>
> >> Example 2: PDU length = 251, msys_1_block_size = 112
> >> Total mbufs: 3 (72 + 96 + 85)
> >>
> >> Hope this helps.
> >>
> >>
> >>
> >>
> >>
> >>
>
>
Re: MBUF sizing for the bluetooth stack
Posted by will sanfilippo <wi...@runtime.io>.
Well, yes, there are “definitions” for these things. They are in various places but they are there. Using them might get a bit tricky as you have mentioned; not sure. You would have to make sure the right header files get included in the proper places...
Anyway, here are the definitions:
os mbuf header: sizeof(struct os_mbuf). Size = 16
os mbuf packet header: sizeof(struct os_mbuf_pkthdr) Size = 8
user header: sizeof(struct ble_mbuf_hdr) Size = 8 or 12
The HCI ACL data header: BLE_HCI_DATA_HSDR_SZ. 4 bytes
The LL PDU header: BLE_LL_PDU_HDR_LEN. 2 bytes
I would always make the size a multiple of 4 but the code should do that for you; I just like to do it so the size you see in the syscfg variable is the actual memory block size.
Another thing I should mention: you should never add a buffer pool to msys smaller than the minimum size I mentioned if you are using the controller. This is something we will address in the future but for now it would be bad. :-)
> On Jan 11, 2017, at 3:49 PM, Simon Ratner <si...@proxy.co> wrote:
>
> Thanks for the detailed write-up, Will - very useful.
>
> Are there defines for these things?
> Ideally, if I want a payload size of N, I'd like to specify in syscfg.yml:
>
> MSYS_1_BLOCK_SIZE: '(N + MBUF_HEADER + PKT_HEADER + LL_OVERHEAD + ...)'
>
> And magically have optimally-sized buffers.
>
>
> On Wed, Jan 11, 2017 at 11:00 AM, will sanfilippo <wi...@runtime.io> wrote:
>
>> Hello:
>>
>> Since this has come up on a number of different occasions I wanted to send
>> out an email which discusses how the nimble stack uses mbufs. This will be
>> a controller-centric discussion but the concepts apply to the host as well.
>>
>> A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs for
>> networking stack packet data. A “packet” is simply a chain of mbufs with
>> the first mbuf in the chain being a packet header mbuf and all others being
>> “normal” mbufs. A packet header mbuf contains a mbuf header, a packet
>> header and an optional user-defined header.
>>
>> The length of the packet (i.e. all the data contained in all the mbuf
>> chains) is stored in the packet header. Each individual mbuf in the chain
>> also contains a length which is the length of the data in that mbuf. The
>> sum of all the mbuf data lengths = length of packet.
>>
>> The amount of overhead in an mbuf and its size determine the amount of
>> data that can be carried in a mbuf. All mbufs have a 16-byte mbuf header.
>> Packet header mbufs have an additional 8 bytes for the packet header
>> structure and an optional user-data header. The nimble stack uses either an
>> 8-byte or 12-byte user data header. If you turn on multi-advertising
>> support, the user header is 12 bytes; otherwise it is 8 bytes. This means
>> the total packet header mbuf overhead is 32 or 36 bytes.
>>
>> The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg
>> variables. Currently, there is one mbuf pool added to msys (MSYS_1) with a
>> block size of 292 bytes.
>>
>> Controller constraints:
>> The controller assumes that a certain minimum data size is available in a
>> packet header mbuf. This size is equal to the largest advertising PDU, or
>> 37 bytes, and must also contain the 2-byte LL PDU header (for a total of 39
>> bytes). Additionally, the controller requires an additional 4 bytes at the
>> start of the packet header mbuf to prepend the HCI ACL data packet header.
>> This means that the minimum mbuf size that can be allocated in any msys
>> mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv).
>> Since memory pools are always rounded up to the nearest 4 byte boundary,
>> this means that the minimum size should be 76 (or 80) bytes.
>>
>> For most applications that dont use large packets, setting the mbuf size
>> to 80 should be fine as this will accommodate the typical BLE PDU and also
>> meets the minimum requirement. If your application generally uses larger
>> packets it might be benefical to allocate large mbufs as you dont lose the
>> 16-byte overhead per mbuf.
>>
>> Finally, here is an example of how many mbufs will be used by the
>> controller for received packets. This assumes multi-advertising enabled (36
>> byte packet header overhead).
>>
>> Example 1: PDU length = 251, msys_1_block_size = 80
>>
>> Controller needs to store 251 + 2 = 253 total bytes.
>>
>> Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
>> Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
>> Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64 bytes
>> while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
>>
>> Example 2: PDU length = 251, msys_1_block_size = 112
>> Total mbufs: 3 (72 + 96 + 85)
>>
>> Hope this helps.
>>
>>
>>
>>
>>
>>
Re: MBUF sizing for the bluetooth stack
Posted by Simon Ratner <si...@proxy.co>.
Thanks for the detailed write-up, Will - very useful.
Are there defines for these things?
Ideally, if I want a payload size of N, I'd like to specify in syscfg.yml:
MSYS_1_BLOCK_SIZE: '(N + MBUF_HEADER + PKT_HEADER + LL_OVERHEAD + ...)'
And magically have optimally-sized buffers.
On Wed, Jan 11, 2017 at 11:00 AM, will sanfilippo <wi...@runtime.io> wrote:
> Hello:
>
> Since this has come up on a number of different occasions I wanted to send
> out an email which discusses how the nimble stack uses mbufs. This will be
> a controller-centric discussion but the concepts apply to the host as well.
>
> A quick refresher on mbufs: Mynewt, and the nimble stack, use mbufs for
> networking stack packet data. A “packet” is simply a chain of mbufs with
> the first mbuf in the chain being a packet header mbuf and all others being
> “normal” mbufs. A packet header mbuf contains a mbuf header, a packet
> header and an optional user-defined header.
>
> The length of the packet (i.e. all the data contained in all the mbuf
> chains) is stored in the packet header. Each individual mbuf in the chain
> also contains a length which is the length of the data in that mbuf. The
> sum of all the mbuf data lengths = length of packet.
>
> The amount of overhead in an mbuf and its size determine the amount of
> data that can be carried in a mbuf. All mbufs have a 16-byte mbuf header.
> Packet header mbufs have an additional 8 bytes for the packet header
> structure and an optional user-data header. The nimble stack uses either an
> 8-byte or 12-byte user data header. If you turn on multi-advertising
> support, the user header is 12 bytes; otherwise it is 8 bytes. This means
> the total packet header mbuf overhead is 32 or 36 bytes.
>
> The total mbuf size is defined by the various MSYS_X_BLOCK_SIZE syscfg
> variables. Currently, there is one mbuf pool added to msys (MSYS_1) with a
> block size of 292 bytes.
>
> Controller constraints:
> The controller assumes that a certain minimum data size is available in a
> packet header mbuf. This size is equal to the largest advertising PDU, or
> 37 bytes, and must also contain the 2-byte LL PDU header (for a total of 39
> bytes). Additionally, the controller requires an additional 4 bytes at the
> start of the packet header mbuf to prepend the HCI ACL data packet header.
> This means that the minimum mbuf size that can be allocated in any msys
> mbuf pool is: packet header overhead + 4 + 39 = 75 (79 for multi-adv).
> Since memory pools are always rounded up to the nearest 4 byte boundary,
> this means that the minimum size should be 76 (or 80) bytes.
>
> For most applications that dont use large packets, setting the mbuf size
> to 80 should be fine as this will accommodate the typical BLE PDU and also
> meets the minimum requirement. If your application generally uses larger
> packets it might be benefical to allocate large mbufs as you dont lose the
> 16-byte overhead per mbuf.
>
> Finally, here is an example of how many mbufs will be used by the
> controller for received packets. This assumes multi-advertising enabled (36
> byte packet header overhead).
>
> Example 1: PDU length = 251, msys_1_block_size = 80
>
> Controller needs to store 251 + 2 = 253 total bytes.
>
> Packet header mbuf can hold 80 - 36 - 4 bytes, or 40 bytes.
> Each additional mbuf can hold 80 - 16 bytes, or 64 bytes.
> Total mbufs = 5. First mbuf holds 40 bytes, the next three hold 64 bytes
> while the final mbuf holds 21 bytes (40 + 64*3 + 21 = 253).
>
> Example 2: PDU length = 251, msys_1_block_size = 112
> Total mbufs: 3 (72 + 96 + 85)
>
> Hope this helps.
>
>
>
>
>
>