Forward media between peers
View SourceAn SFU (selective forwarding unit) receives each participant's media and sends it on to the others without decoding it. With this library you hold one peer connection per participant and move RTP packets from one to another; the library gives the forwarded stream its own SSRC and sequence numbers and handles retransmissions on each leg. You need this guide to build a conference server, a broadcast relay or a recorder that also forwards. The example examples/browser_relay relays one browser page's video to a second page.
When to use
- You relay what one peer sends to one or more others, without decoding.
- You pick one simulcast layer per receiver.
- You need to tell a sender how much it may send (TMMBR, between non-browser peers).
- To send media you produce yourself, see send media.
How
Run the example
rebar3 shell
1> cd("examples/browser_relay").
2> c(browser_relay).
3> browser_relay:start(8080).Open http://localhost:8080/?role=send and http://localhost:8080/?role=receive, in either order. The second page shows the first page's camera. browser_relay:stop() stops the server.
Receive on one peer connection, send on another
The publisher's connection In has a transceiver that receives; the viewer's connection Out has one that sends. When the viewer is a browser that offers to receive only (recvonly), the transceiver its offer creates is recvonly on your side too, as JSEP says. Make it send between set_remote_description/3 and create_answer/2:
ok = webrtc:set_remote_description(Out, offer, Offer),
[Id] = [Id || #{id := Id, kind := video} <- webrtc:transceivers(Out)],
ok = webrtc:set_direction(Out, Id, sendonly),
{ok, Answer} = webrtc:create_answer(Out, #{}),Each {rtp, Mid, Packet} from In goes to send_rtp_packet/3 on Out:
-include_lib("webrtc/include/webrtc_rtp.hrl").
relay(In, InMid, Out, OutMid) ->
receive
{webrtc, In, {rtp, InMid, #rtp{} = Pkt}} ->
_ = webrtc:send_rtp_packet(Out, OutMid, Pkt),
relay(In, InMid, Out, OutMid);
{webrtc, Out, {keyframe_request, OutMid}} ->
ok = webrtc:request_keyframe(In, InMid),
relay(In, InMid, Out, OutMid);
{webrtc, _, {connection_state, closed}} ->
ok;
{webrtc, _, _} ->
relay(In, InMid, Out, OutMid)
end.Events of both connections reach the process that owns them. To run one process per connection, hand it over with controlling_process/2 and pass packets between processes.
send_rtp_packet/3 replaces the SSRC and sequence number with those of OutMid. The payload type, timestamp, marker and payload stay as they came.
Match the payload types
send_rtp_packet/3 rewrites the SSRC and sequence number, not the payload type. Forwarding works as it is when both peers use the same payload type for the codec. When you relay between different browsers, check the payload types each answer negotiated, and when the viewer's differs, set it:
_ = webrtc:send_rtp_packet(Out, OutMid, Pkt#rtp{pt = OutPt}),Ask for a key frame when a viewer joins
A new viewer cannot decode until a key frame arrives. Ask the publisher for one (PLI, or FIR when PLI is not negotiated):
ok = webrtc:request_keyframe(In, InMid),Do the same when a viewer sends {keyframe_request, OutMid}, as in the loop above.
Forward simulcast layers
A browser publishing simulcast sends several layers on one transceiver, each with its rid. The answer accepts the layers it offers, and each packet arrives with its rid:
{webrtc, In, {rtp, InMid, Rid, Pkt}} when Rid =:= Wanted ->
_ = webrtc:send_rtp_packet(Out, OutMid, Pkt),To switch a viewer to another layer, ask for a key frame on that layer and start forwarding it from that key frame:
ok = webrtc:request_keyframe(In, InMid, #{rid => <<"h">>}),To receive layers from a peer that offers to send them, list them on your transceiver; #{rid => Rid, paused => true} starts one paused (request_resume/3 resumes it):
{ok, _} = webrtc:add_transceiver(In, video, #{direction => recvonly, recv_rids => [<<"h">>, <<"m">>, <<"l">>]}),To send layers yourself, give send_encodings and pass the layer's rid to send_rtp_packet/4:
{ok, _} = webrtc:add_transceiver(Out, video, #{send_encodings => [#{rid => <<"h">>}, #{rid => <<"l">>}]}),
_ = webrtc:send_rtp_packet(Out, OutMid, Pkt, #{rid => <<"l">>}),The negotiated layers are in send_encodings of transceivers/1. A layer the answer removed returns {error, not_sending}.
Let the library repair loss
You do nothing for retransmissions. On each leg:
- the receiving side sends NACKs for missing packets, with backoff, and asks for a key frame when too many are missing;
- the sending side keeps what it sent, forwarded packets included, and resends them, on the RTX stream when RTX is negotiated;
- retransmissions are delivered to you already decapsulated, on their layer.
Limit what a sender sends (TMMBR)
Between peers that both negotiated ccm tmmbr (browsers do not), you can ask the sender of a transceiver to stay under a rate, for example the bandwidth of a SIP leg. The rate counts RTP headers and payload; infinity lifts the limit:
ok = webrtc:request_max_bitrate(In, InMid, 300000),It returns {error, not_negotiated} when TMMBR is not negotiated. The request is repeated until the sender acknowledges it.
When a peer limits what you send, you get the payload rate it allows. The connection holds the stream to it, queuing and at worst dropping packets, so keep your encoder under it:
{webrtc, Pc, {max_bitrate, Mid, Bps}} -> set_encoder_rate(Bps),With simulcast the event is {max_bitrate, Mid, Rid, Bps}.
Notes
- Forwarded packets go through the receiving leg's pacer and congestion control, like your own. If the viewer's path is slower than the publisher's, pick a lower layer for it: watch
{bandwidth_estimate, Bps}onOut(see bandwidth and QoS). - Timestamps and codec payload headers (VP8 picture IDs, for example) are not rewritten. When you switch layers or publishers on one outgoing stream, rewrite what the receiving decoder needs to stay continuous.
{rtp, Mid, Packet}only comes on transceivers withoutframes. Withframes => trueyou get whole frames instead; forward those withsend_frame/4.request_pause/2,3andrequest_resume/2,3ask a sender to pause and resume a stream (RFC 7728), when both sides negotiatedccm pause; the answers come as{remote_paused, Mid},{remote_resumed, Mid}or{pause_refused, Mid}.- The events are in the reference; the transceiver options (
recv_rids,send_encodings) in Transceiver options.
Read next
- Send media to a browser
- Bandwidth and QoS
- Read the statistics
- Architecture: how a packet travels through the stack.