webrtc_media_codec behaviour (webrtc v0.1.0)

View Source

A codec plugin decodes the frames a transceiver receives and encodes what the application sends on it, and reports the W3C decoder and encoder statistics. This library stays pure Erlang; a plugin is where native code (FFmpeg, libvpx, Opus) comes in, in a separate package.

Set it on a transceiver that delivers frames:

  webrtc:add_transceiver(Pc, video, #{frames => true, codec_plugin => {my_codec, Opts}})

Each frame received goes to decode/2; every output comes as {webrtc, Pc, {decoded, Mid, Out}}, after the {frame, Mid, Frame} event. webrtc:send_media/3 hands raw media to encode/2 and sends the frames it returns. The plugin runs in the transport process: keep each call short, or hand the work to a process of your own.

Stats are W3C members of the stream (framesDecoded, totalDecodeTime, keyFramesEncoded, ...): counters add up across calls, the others replace the previous value, as with webrtc:report_media_stats/4. info/1 gives decoderImplementation or encoderImplementation.

Loss indications (RFC 8834 §5.1.3, §5.1.4), when the transceiver negotiated nack sli or nack rpsi:

  • decode/2 may return {ok, Outs, Stats, State, Feedback}, with {slice_loss, [{First, Number, PictureId}]}, {reference_picture, Pt, Bits} or keyframe_request in Feedback, for the library to send to the remote encoder.
  • The optional feedback/2 gets what the remote decoder reports about the stream it encodes, {slice_loss, Entries} or {reference_picture, Pt, Bits}, to repair the slices or change its reference picture.

A plugin that raises or returns an error is turned off for that transceiver and the owner gets {codec_plugin_error, Mid, Reason}; frames keep coming.

Summary

Types

codec/0

-type codec() :: map().

frame/0

-type frame() ::
          #{data := binary(), timestamp := 0..4294967295, keyframe := boolean(), atom() => term()}.

Callbacks

decode/2

-callback decode(frame(), State :: term()) ->
                    {ok, [Out :: term()], Stats :: map(), State :: term()} |
                    {ok,
                     [Out :: term()],
                     Stats :: map(),
                     State :: term(),
                     [webrtc_media_fb:indication() | keyframe_request]} |
                    {error, term()}.

encode/2

-callback encode(Raw :: term(), State :: term()) ->
                    {ok, [frame()], Stats :: map(), State :: term()} | {error, term()}.

feedback/2

(optional)
-callback feedback(webrtc_media_fb:indication(), State :: term()) -> {ok, State :: term()} | {error, term()}.

info/1

-callback info(State :: term()) -> #{implementation => binary(), power_efficient => boolean()}.

init/3

-callback init(audio | video, codec(), Opts :: term()) -> {ok, State :: term()} | {error, term()}.