You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mynewt.apache.org by Christopher Collins <cc...@apache.org> on 2017/01/24 22:49:02 UTC
Newtmgr over BLE
Hello all,
Recently, I have mentioned some planned BLE-related changes to the
newtmgr tool. I wanted to share some of what I was thinking. Please
feel free to comment and criticize as needed.
* All BLE code gets removed from the newtmgr tool. The gatt library is
also removed.
* A separate tool, blehostd, runs in its own process. This tool:
* Is a Mynewt app running in the Mynewt simulator.
* Contains the NimBLE host and UART BLE transport.
* Caches information about connected peers (who is connected and how
their services, characteristics, and descriptors map to attribute
handles).
* Communicates with other processes via JSON over a streaming UNIX domain
socket.
* Exposes the host API via JSON-encoded requests, responses, and
events.
* Controller is accessed via UART device (/dev/cu.<...>). This could be
a Mynewt device running the blehci app, or any other controller.
The blehostd app solves a few problems for us:
* Fixes the bugginess and lack of cross platform support in
newtmgr (caused by the gatt library).
* Exposes a generic interface into the NimBLE host for software
other than newtmgr.
Here is how I envision newtmgr using BLE:
1. If an instance of blehostd isn't already running for this
controller [*]:
a. Open and bind to a socket.
b. Start an instance of blehostd, passing it the socket filename
and the controller's /dev filename.
2. Newtmgr tells blehostd to connect to the destination device.
If the device is already connected: blehostd immediately responds,
indicating the connection handle of the device.
Else: The response indicates that the connection attempt is in
progress. Eventually, blehostd sends an event indicating
success and a connection handle, or failure.
3. Newtmgr asks blehostd if the peer's newtmgr characteristic has been
discovered yet.
Yes: blehostd's response indicates the characteristic's
attribute handle.
No: newtmgr tells blehostd to discover the newtmgr
characteristic. After discovery completes, newtmgr repeats
this step so that it knows the characteristic's value handle.
4. Newtmgr builds the CBOR newtmgr request, as always.
5. Newtmgr encodes the CBOR newtmgr request as a bluetooth ATT write
command, and tells blehostd to write it to the peer's newtmgr
characteristic.
6. blehostd immediately responds, indicating success or failure.
7. If successful, blehostd sends an event containing to the peer's
notification (corresponding to its newtmgr response).
[*] This might be a bit tricky to get right. The goal is to not have to
connect and perform service discovery for each newtmgr command.
Ideally, blehostd would stick around until some idle timeout is
exceeded. Then, newtmgr could just reuse the existing socket and
blehostd instance to immediately send a follow-up command. This is more
of an optimization, so it can probably come later.
All comments welcome.
Thanks,
Chris
Re: Newtmgr over BLE
Posted by Sterling Hughes <st...@gmail.com>.
Hi Jacob,
Chris has been working on getting the Mynewt stack working in a cross
platform way. You can see some of his progress here:
*
https://github.com/ccollins476ad/incubator-mynewt-core/tree/nativehost/apps/blehostd/src
*
https://github.com/ccollins476ad/incubator-mynewt-newt/tree/nmxact/newtmgr
The idea is to use Mynewt’s sim architecture to run in a *nix process,
and then speak HCI to whatever controller is running on the device.
Newtmgr forks mynewt, and then communicates over a unix domain socket
using JSON, and then drives the bluetooth stack. I actually pinged
Sandeep about this, because it should now work portably across Linux and
Mac with the same Bluetooth host, no wacky kernel stuff necessary.
The caveat on Mac is that you need a controller (NRF* dongle for
example), on Linux you can either run against a Mynewt controller or use
HCI itself.
I think Chris is a week or two off on getting this ready - and I think
it’s still TBD as whether this belongs in core, or as a separate
project. My vote would be to start with core, and then as it grows,
potentially move it out of core. But it is very nice to have the Mynewt
stack running as a linux process — and fairly easy to map into other
languages, because the interface is all JSON.
Best,
Sterling
On 7 Mar 2017, at 14:47, Jacob Rosenthal wrote:
> Whats the internal thinking on this? Is it being discussed elsewhere?
> Is
> the aforementioned rewrite going on, or delayed so it make sense for
> me to
> try to get work on the xpc calls?
>
> On Wed, Jan 25, 2017 at 1:07 PM, Jacob Rosenthal
> <ja...@gmail.com>
> wrote:
>
>> The gatt dependency actually uses the xpc's reverse engineered by
>> Sandeep
>> for noble github.com/sandeepmistry/noble (which I have some code on)
>> which has three separate xpc bindings for mavericks
>> <https://github.com/sandeepmistry/noble/blob/master/lib/mac/mavericks.js>
>> yosemite
>> <https://github.com/sandeepmistry/noble/blob/master/lib/mac/yosemite.js>
>> and legacy
>> <https://github.com/sandeepmistry/noble/blob/master/lib/mac/legacy.js>
>>
>> Now that the native apis are fairly stable and cover most of what the
>> xpcs
>> have access to, I think even sandeep doesnt love having to redo the
>> xpcs on
>> new os releases..But it takes very little to piggyback on the work
>> hes
>> already done and modernize gatt's xpc handling if thats the only
>> issue...
>>
>>
>>
>>
>>
>> On Wed, Jan 25, 2017 at 12:10 PM, Jacob Rosenthal
>> <jakerosenthal@gmail.com
>>> wrote:
>>
>>> Or at least the newtmgr code as a dependency ;)
>>>
>>> On Wed, Jan 25, 2017 at 6:52 AM, David G. Simmons <sa...@mac.com>
>>> wrote:
>>>
>>>> ook at this app ...
>>>>
>>>
>>>
>>>
>>
Re: Newtmgr over BLE
Posted by Jacob Rosenthal <ja...@gmail.com>.
Whats the internal thinking on this? Is it being discussed elsewhere? Is
the aforementioned rewrite going on, or delayed so it make sense for me to
try to get work on the xpc calls?
On Wed, Jan 25, 2017 at 1:07 PM, Jacob Rosenthal <ja...@gmail.com>
wrote:
> The gatt dependency actually uses the xpc's reverse engineered by Sandeep
> for noble github.com/sandeepmistry/noble (which I have some code on)
> which has three separate xpc bindings for mavericks
> <https://github.com/sandeepmistry/noble/blob/master/lib/mac/mavericks.js>
> yosemite
> <https://github.com/sandeepmistry/noble/blob/master/lib/mac/yosemite.js>
> and legacy
> <https://github.com/sandeepmistry/noble/blob/master/lib/mac/legacy.js>
>
> Now that the native apis are fairly stable and cover most of what the xpcs
> have access to, I think even sandeep doesnt love having to redo the xpcs on
> new os releases..But it takes very little to piggyback on the work hes
> already done and modernize gatt's xpc handling if thats the only issue...
>
>
>
>
>
> On Wed, Jan 25, 2017 at 12:10 PM, Jacob Rosenthal <jakerosenthal@gmail.com
> > wrote:
>
>> Or at least the newtmgr code as a dependency ;)
>>
>> On Wed, Jan 25, 2017 at 6:52 AM, David G. Simmons <sa...@mac.com>
>> wrote:
>>
>>> ook at this app ...
>>>
>>
>>
>>
>
Re: Newtmgr over BLE
Posted by Jacob Rosenthal <ja...@gmail.com>.
The gatt dependency actually uses the xpc's reverse engineered by Sandeep
for noble github.com/sandeepmistry/noble (which I have some code on) which
has three separate xpc bindings for mavericks
<https://github.com/sandeepmistry/noble/blob/master/lib/mac/mavericks.js>
yosemite
<https://github.com/sandeepmistry/noble/blob/master/lib/mac/yosemite.js>
and legacy
<https://github.com/sandeepmistry/noble/blob/master/lib/mac/legacy.js>
Now that the native apis are fairly stable and cover most of what the xpcs
have access to, I think even sandeep doesnt love having to redo the xpcs on
new os releases..But it takes very little to piggyback on the work hes
already done and modernize gatt's xpc handling if thats the only issue...
On Wed, Jan 25, 2017 at 12:10 PM, Jacob Rosenthal <ja...@gmail.com>
wrote:
> Or at least the newtmgr code as a dependency ;)
>
> On Wed, Jan 25, 2017 at 6:52 AM, David G. Simmons <sa...@mac.com>
> wrote:
>
>> ook at this app ...
>>
>
>
>
Re: Newtmgr over BLE
Posted by Jacob Rosenthal <ja...@gmail.com>.
Or at least the newtmgr code as a dependency ;)
On Wed, Jan 25, 2017 at 6:52 AM, David G. Simmons <sa...@mac.com> wrote:
> ook at this app ...
>
Re: Newtmgr over BLE
Posted by "David G. Simmons" <sa...@mac.com>.
I'd love to get a look at this app ...
dg
> On Jan 24, 2017, at 8:18 PM, Kevin Townsend <ke...@adafruit.com> wrote:
>
> I'm not sure what the implications would be here on the iOS app we wrote that is based on the newtmgr protocol and makes use of the current BLE service and characteristics, although we could rewrite it if necessary. We use the currently defined newtmgr protocol and GATT services/chars to get task info and statistic and stream them live onto the mobile device, which I think provides a unique window into your device in a non-invasive way.
--
David G. Simmons
(919) 534-5099
Web <https://davidgs.com/> • Blog <https://davidgs.com/davidgs_blog> • Linkedin <http://linkedin.com/in/davidgsimmons> • Twitter <http://twitter.com/TechEvangelist1> • GitHub <http://github.com/davidgs>
/** Message digitally signed for security and authenticity.
* If you cannot read the PGP.sig attachment, please go to
* http://www.gnupg.com/ <http://www.gnupg.com/> Secure your email!!!
* Public key available at keyserver.pgp.com <http://keyserver.pgp.com/>
**/
♺ This email uses 100% recycled electrons. Don't blow it by printing!
There are only 2 hard things in computer science: Cache invalidation, naming things, and off-by-one errors.
Re: Newtmgr over BLE
Posted by Christopher Collins <cc...@apache.org>.
Hi Kevin,
On Wed, Jan 25, 2017 at 02:18:01AM +0100, Kevin Townsend wrote:
> I'm not sure what the implications would be here on the iOS app we wrote
> that is based on the newtmgr protocol and makes use of the current BLE
> service and characteristics, although we could rewrite it if necessary.
> We use the currently defined newtmgr protocol and GATT services/chars to
> get task info and statistic and stream them live onto the mobile device,
> which I think provides a unique window into your device in a
> non-invasive way.
The newtmgr protocol wouldn't be affected. This would just be a way to
get the newtmgr tool to speak BLE.
> I'm certainly OK with BLE being removed from the newtmgr tool and having
> that supported by a dedicated tool or service based on UART/HCI, but
> just to be sure are there any implications here for the newtmgr protocol
> running on a mynewt app? It sounds like we should still be OK with the
> app, though, and this is limited to the newtmgr tool itself not the
> newtmgr protocol implementation or the BLE transport in the Mynewt app.
>
> One advantage of taking the HCI approach talking to a Mynewt target over
> UART is that you can add support to any OS, not just systems that
> support BLE, and not having to depend on a specific OS or BlueZ version.
> You need to do some work up front and it would require another nRF5x
> board, but it solves more problems than it creates in my opinion.
>
> Alternatively, you could use the simulator and try to add some hooks
> into CoreBluetooth and BlueZ (so no additional HW required), but that's
> ultimately frustrating and a significant maintenance burden. UART/HCI
> seems like less work both short and long term, and the requirements for
> an additional nRF5x board seems relatively trivial.
All good points. I really think BLE on Linux vs. MacOS is so different
that it would be less of a headache to make our own.
> Removing the dependency on the current GO lib and moving to HCI/UART and
> a dedicated Mynewt app is a +1 for me and worth doing before 1.0, even
> if it's a major change ... with the caveat that I hope this won't break
> anything with the newtmgr protocol that will cause us to have to rewrite
> the newtmgr tool we wrote and want to release around the same time as
> 1.0.0 final. :)
Thanks,
Chris
> On 24/01/17 23:49, Christopher Collins wrote:
> > Hello all,
> >
> > Recently, I have mentioned some planned BLE-related changes to the
> > newtmgr tool. I wanted to share some of what I was thinking. Please
> > feel free to comment and criticize as needed.
> >
> > * All BLE code gets removed from the newtmgr tool. The gatt library is
> > also removed.
> >
> > * A separate tool, blehostd, runs in its own process. This tool:
> > * Is a Mynewt app running in the Mynewt simulator.
> > * Contains the NimBLE host and UART BLE transport.
> > * Caches information about connected peers (who is connected and how
> > their services, characteristics, and descriptors map to attribute
> > handles).
> > * Communicates with other processes via JSON over a streaming UNIX domain
> > socket.
> > * Exposes the host API via JSON-encoded requests, responses, and
> > events.
> >
> > * Controller is accessed via UART device (/dev/cu.<...>). This could be
> > a Mynewt device running the blehci app, or any other controller.
> >
> > The blehostd app solves a few problems for us:
> > * Fixes the bugginess and lack of cross platform support in
> > newtmgr (caused by the gatt library).
> > * Exposes a generic interface into the NimBLE host for software
> > other than newtmgr.
> >
> > Here is how I envision newtmgr using BLE:
> >
> > 1. If an instance of blehostd isn't already running for this
> > controller [*]:
> > a. Open and bind to a socket.
> > b. Start an instance of blehostd, passing it the socket filename
> > and the controller's /dev filename.
> >
> > 2. Newtmgr tells blehostd to connect to the destination device.
> > If the device is already connected: blehostd immediately responds,
> > indicating the connection handle of the device.
> > Else: The response indicates that the connection attempt is in
> > progress. Eventually, blehostd sends an event indicating
> > success and a connection handle, or failure.
> >
> > 3. Newtmgr asks blehostd if the peer's newtmgr characteristic has been
> > discovered yet.
> > Yes: blehostd's response indicates the characteristic's
> > attribute handle.
> > No: newtmgr tells blehostd to discover the newtmgr
> > characteristic. After discovery completes, newtmgr repeats
> > this step so that it knows the characteristic's value handle.
> >
> > 4. Newtmgr builds the CBOR newtmgr request, as always.
> >
> > 5. Newtmgr encodes the CBOR newtmgr request as a bluetooth ATT write
> > command, and tells blehostd to write it to the peer's newtmgr
> > characteristic.
> >
> > 6. blehostd immediately responds, indicating success or failure.
> >
> > 7. If successful, blehostd sends an event containing to the peer's
> > notification (corresponding to its newtmgr response).
> >
> > [*] This might be a bit tricky to get right. The goal is to not have to
> > connect and perform service discovery for each newtmgr command.
> > Ideally, blehostd would stick around until some idle timeout is
> > exceeded. Then, newtmgr could just reuse the existing socket and
> > blehostd instance to immediately send a follow-up command. This is more
> > of an optimization, so it can probably come later.
> >
> > All comments welcome.
> >
> > Thanks,
> > Chris
>
Re: Newtmgr over BLE
Posted by Kevin Townsend <ke...@adafruit.com>.
I'm not sure what the implications would be here on the iOS app we wrote
that is based on the newtmgr protocol and makes use of the current BLE
service and characteristics, although we could rewrite it if necessary.
We use the currently defined newtmgr protocol and GATT services/chars to
get task info and statistic and stream them live onto the mobile device,
which I think provides a unique window into your device in a
non-invasive way.
I'm certainly OK with BLE being removed from the newtmgr tool and having
that supported by a dedicated tool or service based on UART/HCI, but
just to be sure are there any implications here for the newtmgr protocol
running on a mynewt app? It sounds like we should still be OK with the
app, though, and this is limited to the newtmgr tool itself not the
newtmgr protocol implementation or the BLE transport in the Mynewt app.
One advantage of taking the HCI approach talking to a Mynewt target over
UART is that you can add support to any OS, not just systems that
support BLE, and not having to depend on a specific OS or BlueZ version.
You need to do some work up front and it would require another nRF5x
board, but it solves more problems than it creates in my opinion.
Alternatively, you could use the simulator and try to add some hooks
into CoreBluetooth and BlueZ (so no additional HW required), but that's
ultimately frustrating and a significant maintenance burden. UART/HCI
seems like less work both short and long term, and the requirements for
an additional nRF5x board seems relatively trivial.
Removing the dependency on the current GO lib and moving to HCI/UART and
a dedicated Mynewt app is a +1 for me and worth doing before 1.0, even
if it's a major change ... with the caveat that I hope this won't break
anything with the newtmgr protocol that will cause us to have to rewrite
the newtmgr tool we wrote and want to release around the same time as
1.0.0 final. :)
On 24/01/17 23:49, Christopher Collins wrote:
> Hello all,
>
> Recently, I have mentioned some planned BLE-related changes to the
> newtmgr tool. I wanted to share some of what I was thinking. Please
> feel free to comment and criticize as needed.
>
> * All BLE code gets removed from the newtmgr tool. The gatt library is
> also removed.
>
> * A separate tool, blehostd, runs in its own process. This tool:
> * Is a Mynewt app running in the Mynewt simulator.
> * Contains the NimBLE host and UART BLE transport.
> * Caches information about connected peers (who is connected and how
> their services, characteristics, and descriptors map to attribute
> handles).
> * Communicates with other processes via JSON over a streaming UNIX domain
> socket.
> * Exposes the host API via JSON-encoded requests, responses, and
> events.
>
> * Controller is accessed via UART device (/dev/cu.<...>). This could be
> a Mynewt device running the blehci app, or any other controller.
>
> The blehostd app solves a few problems for us:
> * Fixes the bugginess and lack of cross platform support in
> newtmgr (caused by the gatt library).
> * Exposes a generic interface into the NimBLE host for software
> other than newtmgr.
>
> Here is how I envision newtmgr using BLE:
>
> 1. If an instance of blehostd isn't already running for this
> controller [*]:
> a. Open and bind to a socket.
> b. Start an instance of blehostd, passing it the socket filename
> and the controller's /dev filename.
>
> 2. Newtmgr tells blehostd to connect to the destination device.
> If the device is already connected: blehostd immediately responds,
> indicating the connection handle of the device.
> Else: The response indicates that the connection attempt is in
> progress. Eventually, blehostd sends an event indicating
> success and a connection handle, or failure.
>
> 3. Newtmgr asks blehostd if the peer's newtmgr characteristic has been
> discovered yet.
> Yes: blehostd's response indicates the characteristic's
> attribute handle.
> No: newtmgr tells blehostd to discover the newtmgr
> characteristic. After discovery completes, newtmgr repeats
> this step so that it knows the characteristic's value handle.
>
> 4. Newtmgr builds the CBOR newtmgr request, as always.
>
> 5. Newtmgr encodes the CBOR newtmgr request as a bluetooth ATT write
> command, and tells blehostd to write it to the peer's newtmgr
> characteristic.
>
> 6. blehostd immediately responds, indicating success or failure.
>
> 7. If successful, blehostd sends an event containing to the peer's
> notification (corresponding to its newtmgr response).
>
> [*] This might be a bit tricky to get right. The goal is to not have to
> connect and perform service discovery for each newtmgr command.
> Ideally, blehostd would stick around until some idle timeout is
> exceeded. Then, newtmgr could just reuse the existing socket and
> blehostd instance to immediately send a follow-up command. This is more
> of an optimization, so it can probably come later.
>
> All comments welcome.
>
> Thanks,
> Chris