Media

Overview

The media module is responsible for establishing WebRTC sessions between the client and SFU used by OpenTalk. SFU stands for Selective Forwarding Unit and is the redistributor of the media published by each participant.

"Publishing media" means to transmit video and audio to OpenTalk via a WebRTC sendonly session. Depending on the use-case the media module allows for 2 different kinds of publishing sessions:

video: (Lower bitrate) Usually low resolution and higher framerate video used for Webcams or similar with more motion
screen: (Higher bitrate) Usually high resolution and low framerate ideal for screen share/presentations with text which must be readable

"Subscribing" to a peer in a conference means to receive video and audio via a WebRTC recvonly session. The published media of another participant is stored within the module-specific data in the Participant.

The notion of presenter is used to communicate screen share permissions.

Joining the room

JoinSuccess

When joining a room, the join_success control event contains the module-specific fields described below.

Fields

Field	Type	Always	Description
`is_presenter`	`bool`	yes	Represents if the current participant has permissions for screen share

Example

{
    "is_presenter": true
}

Joined

When joining a room, the joined control event sent to all other participants contains the module-specific fields described below.

Fields

Field	Type	Always	Description
`state`	`ParticipantMediaState`	no	An object describing the current availability of media and their current mute status
`is_presenter`	`bool`	yes	Represents if the other participant has permissions for screen share

ParticipantMediaState:

This object represents the available media sessions and encapsulates their mute state.

Field	Type	Always	Description
`video`	`MediaSessionState`	no	If this field is set, the participant is publishing a video, usually a webcam (lower resolution, higher framerate)
`screen`	`MediaSessionState`	no	If this field is set, the participant is publishing their screen (usually high resolution, low framerate)

Example

This example shows a participant's state when publishing only their screen without audio enabled.

{
    "state": {
        "screen": {
            "video": true,
            "audio": false
        }
    },
    "is_presenter": true
}

Commands

Publish

Create a WebRTC publish session by sending an SDP offer.

Response

A SdpAnswer will return an SDP response.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"publish"`
`sdp`	`string`	yes	SDP Offer as a string
`target`	`string`	yes	See `Target`
`media_session_type`	`enum`	yes	See `Target`

Example

{
    "action": "publish",
    "sdp": "v=0,....",
    "target": "07d32d3e-9510-49bf-82b7-e21fef9db120",
    "media_session_type": "video"
}

PublishComplete

Signal that the WebRTC publish process is complete and set the initial mute status. Other participants will be notified by an Update that the media is now available to be subscribed to and if the audio or video track is muted.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"publish_complete"`
`media_session_type`	`enum`	yes	Either `"video"` or `"screen"`
`media_session_state`	`MediaSessionState`	yes

Example

{
    "action": "publish_complete",
    "media_session_type": "video",
    "media_session_state": {
        "video": {
            "video": true,
            "audio": false
        }
    }
}

Unpublish

Remove/stop an existing WebRTC publish session. The WebRTC session will instantly be stopped (if not already) by the SFU when sending this command. Other participants will be notified by an Update that the published media is no longer available.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"unpublish"`
`media_session_type`	`enum`	yes	Either `"video"` or `"screen"`

Example

{
    "action": "unpublish",
    "media_session_type": "video"
}

Request an SDP offer for the specified target (a peer participant's published media).

Response

A SdpOffer will return an SDP offer to which the client must respond to with an SdpAnswer.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"subscribe"`
`target`	`string`	yes	See `Target`
`media_session_type`	`enum`	yes	See `Target`
`without_video`	`bool`	no	Can be used to opt out of the video stream for the entire WebRTC session

Example

{
    "action": "subscribe",
    "target": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video",
    "without_video": true
}

Resubscribe

Request for a WebRTC subscribe session to be restarted, this will restart the complete negotiation process.

Response

A SdpOffer will return an SDP offer to which the client must respond to with an SdpAnswer.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"resubscribe"`
`target`	`string`	yes	See `Target`
`media_session_type`	`enum`	yes	See `Target`

Example

{
    "action": "resubscribe",
    "target": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

SdpAnswer

Complete the initial SDP negotiation of a subscription by sending an SDP answer. The SFU will begin the WebRTC/ICE handshake and notify later via an WebrtcUp if the establishment was successful from it's side.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"sdp_answer"`
`sdp`	`string`	yes	SDP Answer as a string
`target`	`string`	yes	See `Target`
`media_session_type`	`enum`	yes	See `Target`

Example

{
    "action": "sdp_answer",
    "sdp": "v=0,...",
    "target": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

SdpCandidate

Exchange a ICE trickle candidate with a establishing WebRTC session. This is part of the WebRTC establishment and is forwarded to the SFU.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"sdp_candidate"`
`candidate`	`TrickleCandidate`	yes	Candidate to send
`target`	`string`	yes	See `Target`
`media_session_type`	`enum`	yes	See `Target`

Example

{
    "action": "sdp_candidate",
    "candidate": {
        "sdpMLineIndex": 0,
        "candidate": "candidate:..."
    },
    "target": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

SdpEndOfCandidates

Signal the establishing WebRTC session that there are no more ICE trickle candidates. This is part of the WebRTC establishment and is forwarded to the SFU.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"sdp_end_of_candidates"`
`target`	`string`	yes	See `Target`
`media_session_type`	`enum`	yes	See `Target`

Example

{
    "action": "sdp_end_of_candidates",
    "target": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

UpdateMediaSession

Update the mute state for a WebRTC publish session. This information is forwarded to all other participants using inside an Update.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"update_media_session"`
`media_session_type`	`enum`	yes	Either `"video"` or `"screen"`
`media_session_state`	`MediaSessionState`	yes	The new state for the session

Example

{
    "action": "update_media_session",
    "media_session_type": "video",
    "media_session_state": {
        "video": {
            "audio": true,
            "video": true,
        },
        "screen": null
    }
}

Configure

Configure a WebRTC subscribe session. The request is forwarded to the SFU which configured the media streams appropriately.

This is used to adjust the quality of a video stream to save on bandwidth.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"configure"`
`configuration`	`SubscriberConfiguration`	yes
`target`	`string`	yes	See `Target`
`media_session_type`	`enum`	yes	See `Target`

SubscriberConfiguration:

Field	Type	Required	Description
`video`	`bool`	no	If video transmission should be enabled. This option should be used to turn off videos of participants which are out of view
`substream`	`int`	no	Select a different substream if any are available. Substreams are usually the same video with difference in bitrate/quality, where 0 is usually the lowest quality. This options should be used to lower the quality of participants that are displayed in a smaller window.

Example

{
    "action": "configure",
    "configuration": {
        "video": true,
        "substream": 0
    },
    "target": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video",
}

GrantPresenterRole

Grant one or more participants the right to show their screen

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"grant_presenter_role"`
`participant_ids`	`string`	yes	List of participant ids to grant the presenter role

Example

{
    "action": "grant_presenter_role",
    "participant_ids": [
        "84a2c872-94fb-4b41-aca7-13d784c92a72",
        "2375602f-c74c-4935-9933-bfd67d4e8ae5"
    ]
}

RevokePresenterRole

Revoke one or more participants the right to show their screen

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"revoke_presenter_role"`
`participant_ids`	`string`	yes	List of participant ids to revoke their presenter role

Example

{
    "action": "revoke_presenter_role",
    "participant_ids": [
        "84a2c872-94fb-4b41-aca7-13d784c92a72",
        "2375602f-c74c-4935-9933-bfd67d4e8ae5"
    ]
}

ModeratorMute

Request another participant to mute their microphone.

Fields

Field	Type	Required	Description
`action`	`enum`	yes	Must be `"moderator_mute"`
`targets`	`string`	yes	List of participant ids to request to be muted
`force`	`bool`	yes	We're not asking you to mute, we're telling!

Example

{
    "action": "moderator_mute",
    "targets": [
        "84a2c872-94fb-4b41-aca7-13d784c92a72",
        "2375602f-c74c-4935-9933-bfd67d4e8ae5"
    ],
    "force": true
}

Events

SdpAnswer

Provides an SDP answer string in response to a Publish command.

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"sdp_answer"`
`sdp`	`string`	yes	SDP Answer in a string
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "sdp_answer",
    "sdp": "v=0,...",
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

SdpOffer

Returns an SDP offer string in response to a Subscribe command.

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"sdp_offer"`
`sdp`	`string`	yes	SDP Answer in a string
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "sdp_offer",
    "sdp": "v=0,...",
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

SdpCandidate

Receive an SDP candidate for the specified WebRTC session.

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"sdp_candidate"`
`candidate`	`TrickleCandidate`	yes
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "sdp_candidate",
    "candidate": {
        "sdpMLineIndex": 0,
        "candidate": "candidate:..."
    },
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

SdpEndOfCandidates

Receive an SDP end-of-candidates signal for the specified WebRTC session.

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"sdp_end_of_candidates"`
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "sdp_end_of_candidates",
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

WebrtcUp

The SFU reports that the specified WebRTC session has been established.

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"webrtc_up"`
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "webrtc_up",
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

WebrtcDown

The SFU reports that the specified WebRTC session has broken down.

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"webrtc_down"`
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "webrtc_down",
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

WebrtcSlow

The SFU reports problems sending/receiving data for the specified WebRTC session

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"webrtc_slow"`
`direction`	`enum`	yes	Either `upstream` or `downstream`
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "webrtc_slow",
    "direction": "upstream",
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

MediaStatus

The SFU reports an update in the current status of the specified WebRTC session

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"media_status"`
`kind`	`string`	yes	The kind of media, so usually either `video`, `audio` but might also be application data if the WebRTC sessions are used for that
`receiving`	`bool`	yes	Reports the status of the media being received, if false the SFU reports that it no longer receives any data.
`source`	`string`	yes	See `Source`
`media_session_type`	`enum`	yes	See `Source`

Example

{
    "message": "media_status",
    "kind": "audio",
    "receiving": true,
    "source": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "media_session_type": "video"
}

FocusUpdate

A new participant has been selection to be focused/highlighted.

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"focus_update"`
`focus`	`string`	no	Id of the participant to focus. If null or missing no one should focussed/highlighted

Example

{
    "message": "focus_update",
    "focus": "84a2c872-94fb-4b41-aca7-13d784c92a72"
}

PresenterGranted

Presenter rights have been granted

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"presenter_granted"`

Example

{
    "message": "presenter_granted",
}

PresenterRevoked

Presenter rights have been revoked

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"presenter_revoked"`

Example

{
    "message": "presenter_revoked",
}

RequestMute

You are being asked to mute yourself

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"request_mute"`
`issuer`	`string`	yes	Id of the participant which asked you to mute yourself
`force`	`bool`	yes	Clients should automatically mute when this is set to `true`

Example

{
    "message": "request_mute",
    "issuer": "84a2c872-94fb-4b41-aca7-13d784c92a72",
    "force": true
}

Error

Something went wrong

Fields

Field	Type	Always	Description
`message`	`enum`	yes	Is `"error"`
`error`	`enum`	yes	Any of the codes specified below
`source`	`string`	if `error` is `"invalid_request_offer"` or `"invalid_configure_request"`	See `Source`
`media_session_type`	`enum`	if `error` is `"invalid_request_offer"` or `"invalid_configure_request"`	See `Source`

Error codes:

"invalid_sdp_offer"
"handle_sdp_answer"
"invalid_candidate"
"invalid_end_of_candidates"
"invalid_request_offer"
"invalid_configure_request"
"permission_denied": The requester didn't meet the required permissions for the request

Example

{
    "message": "error",
    "error": "invalid_sdp_offer"
}

Common Types

Source

Field	Type	Always	Description
`source`	`string`	yes	ParticipantID describing the source WebRTC session of the event. If the WebRTC session is publishing media, the participants own id is used
`media_session_type`	`enum`	yes	Either `"video"` or `"screen"`

Target

Field	Type	Required	Description
`target`	`string`	yes	ParticipantID describing the target WebRTC session of the command. If the WebRTC session is publishing media, the participants own id must be used
`media_session_type`	`enum`	yes	Either `"video"` or `"screen"`

MediaSessionState

This object represent the current mute status of a media session.

Field	Type	Always	Description
`video`	`bool`	yes	Video is enabled (unmuted)
`audio`	`bool`	yes	Audio is enabled (unmuted)

TrickleCandidate

Field	Type	Required	Description
`sdpMLineIndex`	`int`	yes	MDN web docs reference
`candidate`	`string`	yes	MDN web docs reference

Media

Overview​

Joining the room​

JoinSuccess​

Fields​

Example​

Joined​

Fields​

Example​

Commands​

Publish​

Response​

Fields​

Example​

PublishComplete​

Fields​

Example​

Unpublish​

Fields​

Example​

Subscribe​

Response​

Fields​

Example​

Resubscribe​

Response​

Fields​

Example​

SdpAnswer​

Fields​

Example​

SdpCandidate​

Fields​

Example​

SdpEndOfCandidates​

Fields​

Example​

UpdateMediaSession​

Fields​

Example​

Configure​

Fields​

Example​

GrantPresenterRole​

Fields​

Example​

RevokePresenterRole​

Fields​

Example​

ModeratorMute​

Fields​

Example​

Events​

SdpAnswer​

Fields​

Example​

SdpOffer​

Fields​

Example​

SdpCandidate​

Fields​

Example​

SdpEndOfCandidates​

Fields​

Example​

WebrtcUp​

Fields​

Example​

WebrtcDown​

Fields​

Example​

WebrtcSlow​

Fields​

Example​

MediaStatus​

Fields​

Example​

FocusUpdate​

Fields​

Example​

Overview

Joining the room

JoinSuccess

Fields

Example

Joined

Fields

Example

Commands

Publish

Response

Fields

Example

PublishComplete

Fields

Example

Unpublish

Fields

Example

Subscribe

Response

Fields

Example

Resubscribe

Response

Fields

Example

SdpAnswer

Fields

Example

SdpCandidate

Fields

Example

SdpEndOfCandidates

Fields

Example

UpdateMediaSession

Fields

Example

Configure

Fields

Example

GrantPresenterRole

Fields

Example

RevokePresenterRole

Fields

Example

ModeratorMute

Fields

Example

Events

SdpAnswer

Fields

Example

SdpOffer

Fields

Example

SdpCandidate

Fields

Example

SdpEndOfCandidates

Fields

Example

WebrtcUp

Fields

Example

WebrtcDown

Fields

Example

WebrtcSlow

Fields

Example

MediaStatus

Fields

Example

FocusUpdate

Fields

Example

PresenterGranted