Appscio approached the issues we identified in an earlier article with set of key objectives:

1. Promote rapid, widespread adoption
2. Leverage existing technologies
3. Simplify the development and deployment of media algorithms
4. Allow portability across the spectrum from mobile devices to server farms
5. Maximize the scalability of solutions
6. Encourage the emergence of standards

Meeting these objectives required implementing complementary technology and business strategies.  This article describes the concepts embodied in the Appscio technology and a companion article discusses the company's business strategy.

The Appscio™ Media Processing Framework (MPF) is an open software framework that is independent of media formats and data types.  To promote adoption it is distributed as open source software.  To leverage existing work it is implemented as an extension to one of the most mature and respected open source projects (GStreamer) and employs the W3C standard for expressing metadata, RDF.

Basic Terminology

At its heart the Appscio™ MPF is a tool for configuring and executing "pipelines" composed of semantic media algorithms. A Pipeline is a sequence of Algorithms (wrapped with Appscio™ MPF code) configured to perform an application-specific Service.  Services implemented in Pipelines can be as simple as transcoding a media file from one format to another or as complex as generating a transcript from the audio track, recognizing people in the video stream or both.

Algorithms perform a wide variety of elementary functions and fall into distinct categories.  CODECs compress (encode) and decompress (decode) audio and video (AV) signals.  Filters modify frames in AV signals.  Detectors identify objects (words, faces, text, etc.) and behaviors (e.g. motion) in AV channels.  Classifiers and Recognizers associate structured fragments of metadata with objects and behaviors.  Analytics distill, improve, combine and/or summarize fragments of metadata.

The Appscio™ MPF provides a mechanism for communicating semantic metadata between Algorithms.  Metadata is "published" and consumed through Pads implemented in Appscio™ MPF software Wrappers around elementary media algorithms.  Algorithms are assembled in Pipelines and metadata flows synchronously "downstream" in RDF format through Pads forming a "metadata channel."  The Appscio™ MPF also supports use of a scripting language to implement logical Elements that combine and extend the metadata flowing through the metadata channel.

Relationship to GStreamer

The Appscio™ MPF is implemented as an extension to GStreamer, an open source media handling toolkit.  Since it's inception, the GStreamer community has created large libraries of Elements, algorithms typically developed for media playback applications.  Most of these Elements are CODECs and Filters.

Appscio™ MPF Wrappers are designed to hide the complexity of the GStreamer environment and add support for transporting metadata.  Semantic algorithms can be turned into GStreamer Elements by wrapping them with Appscio™ MPF code.

Wrapping Algorithms

The Appscio™ MPF Wrapper is a block of C code which is generated by an Appscio-provided template generator. The developer defines the inputs and outputs for an Element and certain information such as name or license.  Then the template generator makes a ready-to-run autotools project; the developer simply copies the particular algorithm's code (or calls to an object code library) into one pre-defined place in the Wrapper. The developer then uses the Appscio GRDF library to add logic that generates and consumes metadata, in the Wrapper, not in the algorithm itself.  The entire process simply "wraps" the algorithm rather than modifying it.

This process is explained in more detail in the Component Developers Guide.

Constructing and Executing Pipelines

Pipelines are defined by a "launch line" which specifies a combination of native GStreamer and wrapped Appscio™ MPF Elements in a particular order.  A launch line effectively defines media processing service that applications can invoke as web services.

Because the Appscio™ MPF Wrappers are valid GStreamer Elements, you can execute a launch line from a command line using the GStreamer 'gst-launch' utility.  Appscio will be providing an extended gst-launch with metadata filtering/logging services.

Elements take parameters (at load-time, and dynamically at run-time). The Appscio Media Services package (part of the commercial offering) extends the launch line syntax with a template model to allow for improvements in launch line reuse.

Related Tools

• gst-launch
• autotools