Re: Targeting audio

[Gerben <ge...@treora.com.INVALID>]

Hi all, this promised follow-up took a little longer than intended, as
usual..

*Bookmarking audio fragments*

We just published a proof of concept of the browser extension that
enables both linking to and bookmarking audio fragments. As Dinesh
already said, the idea is to start with enabling people to
highlight/bookmark or link to fragments. There is no concept yet of
annotation bodies (though the bookmark’s title is displayed in the
highlight).

Description, screenshot, code and all that can be found here:
https://code.treora.com/gerben/bookmark-audio-fragment

A nice aspect of audio is that there is a standard fragment identifier
to point to time ranges: https://www.w3.org/TR/media-frags/. A
bookmarked audio fragment is simply a normal browser bookmark that ends
with something like #t=15,22. Sharing such a bookmark works fine because
most browsers//will jump to the indicated position. Our extension
complements this /consuming/ support by providing a convenient interface
for /creating/ such fragment URLs (Jon Udell wrote about this asymmetry
here <https://blog.jonudell.net/2018/01/06/annotating-web-audio/>).
Hopefully, selecting audio could be made as easy as selecting text (and
ideally likewise for copying & pasting it).

One goal of the exercise was to see whether any reusable patterns or
modules could shake out of this, that might be included in Apache
Annotator. So far, this seems hardly the case; especially as anchoring a
time range is trivial compared to our more often discussed challenge of
text quote selection (unfortunately anchoring text quotes in audio
tracks may be a step too far). We could however consider creating some
more tools for dealing with fragment identifiers, and their conversion
into Web Annotation Selectors.

Some small pieces of code whose pattern may be generalisable are found
here
<https://code.treora.com/gerben/bookmark-audio-fragment/src/commit/3fee786747bfe54e38752600354e8918fc03d9c8/app/create-bookmarks/contentscript.js#L6-L28>
(time range → selector → fragment identifier → url) and here
<https://code.treora.com/gerben/bookmark-audio-fragment/src/commit/3fee786747bfe54e38752600354e8918fc03d9c8/app/display-bookmarks/contentscript.js#L6-L23>
(the reverse of that). Also I had to implement some parsing code for
media fragment identifiers (here
<https://code.treora.com/gerben/bookmark-audio-fragment/src/commit/3fee786747bfe54e38752600354e8918fc03d9c8/app/util/media-fragment-identifier.js>),
which depending on our goals might be either inside or outside
Annotator’s project scope (I suppose such code could be depended on by
Annotator, but should ideally be homed elsewhere?).

*Bookmarking text fragments*

Before creating the audio bookmarker, we quickly made an equivalent
extension for bookmarking text selections in html pages:
https://github.com/salus-sage/bookmark-selection/tree/next

For text, which still lacks a common standard for pointing at fragments,
we followed the selector-states note
<https://www.w3.org/TR/selectors-states/#frags>, for which the extension
uses @annotator/fragment-identifier. (I also played
<https://github.com/salus-sage/bookmark-selection/blob/7f5ac3a487a8ba3c201b7b38ade0d11666a3be7e/app/create-bookmarks/background.js#L16-L22>
with supporting the syntax from the Scroll to Text Fragment
<https://wicg.github.io/ScrollToTextFragment/> draft).

The extension also relies on @annotator/dom for describing/anchoring the
text quotes, specifically on the current version
<https://github.com/apache/incubator-annotator/tree/master/packages/dom/>
that is still unpublished (I use npm link to build this extension).
Apart from some anchoring failures for longer strings it seems to work
fine, so I suggest that it be published to move things forward!

— Gerben


On 18/10/2019 21:08, TB Dinesh wrote:
> Hi all,
>
> We are trying to build some annotation tools and looking forward to
> use and contribute to the Apache Annotator project. I am briefly
> mentioning the context and our need here.
>
> We have been working on Renarration Web and Alipi tools - A social
> semantic web approach to bring accessible content to the low-literates
> Internet users. This approach relies on Web Annotation as a
> fundamental need for the billions who cannot read and write but need
> to hyperlink media to help manifest a Web with audio, images, videos,
> 3D point cloud, .. .
>
> We imagine a scenario where a mesh-radio setup in a rural location can
> bring in a culture of creating, editing and sharing of audios. In this
> world, even to navigate and discover, one would need an annotation
> service to label parts of audio. Audio fragments can then be annotated
> by image or portions of an image - which can be used to render the
> audio with images. This process can help renew a culture of
> storytelling online, even among the indigenous communities who shy
> away from text.
>
> Thanks to Gerben visiting us here in Bangalore, we have been
> discussing ways to use a browser to help interconnect data (text and
> media) using annotations.
> To start with, we are creating browser extensions that enable
> selecting a fragment of an html document, of an audio file, and
> perhaps soon also of an image. Then to annotate a selected fragment,
> we will need to select another fragment elsewhere to use as the
> annotation body. But as a first step, we simply bookmark the fragments
> of interest.
>
> More details and state of progress soon by Gerben and Bhanu.
>
> -dinesh
>
> Some contextual links:
> j.mp/RuralJunctions
> j.mp/Janastu-mesh