iOS Reference Application
Basic Overview
The iOS Meeting SDK Reference Application shows how you would use the Visionable MeetingSDK to build an application for iOS to join a meeting, process incoming audio and video data/metadata as well as capture and send audio and video into the meeting for others to see and hear.
The Reference Application does demonstrate best practices when using the MeetingSDK but it does not necessarily reflect best practices in iOS UI design and implementation. Instead, it gives you an "idea" on how you can pair iOS UI design with the SDK.
We'll take a walk through the iOS Meeting SDK Reference app and show you how it uses the SDK to implement participation in a meeting. To launch the application, simply double-click on the application icon. You will be greeted with the following screen:
Here you will see a Table View with place to enter the server name you wish to connect to, a Meeting UUID as well as the name you'd like to join as. When using the reference app, you can only join as a "guest" (there is no notion of "logging in" to a remote server).
Retrieving Audio and Video Options
After filling in the server name, Meeting UUID and your name, you'll need to choose the audio and video options you'd like to join meeting with. Tapping on either the audio input or audio output table view cell will bring up an action sheet containing the corresponding options for audio input and output:
Currently, on an iOS device you will only see "Default device" for both audio input and output. If you would like to give the user the option to switch between the speaker (as in a speaker phone), requiring them to hold the device up to their ear as they would with a phone call OR the ability to use wired or wireless headphones, you will need to implement code that changes the default input/output device separately from the SDK.
If you tap on the "Video Devices" table view cell, you will be presented with a list of available video devices on this device:
We retrieve the available audio input, audio output and video devices using API calls from the MeetingSDK.
You should always use the getAudioInputDevices() and getAudioOutputDevices() API call to retrieve the list of available devices even if (for now) you only get "Default device" back for each. To retrieve a list of available video devices, use the getVideoDevices call:
You'll notice that when viewing the list of available video devices you can tap on an entry to see all available resolutions you can capture and send into the meeting. It is expected that your will provide the user with a list of available resolutions for each device available.
Obtaining this list of resolutions is done with the getSupportedVideoSendResolutions() API call.
Initializing and Joining a Meeting
Find a valid Meeting UUID to use for this meeting. Once you have it, enter it into the proper location in the "Settings" tab and enter a name to use when joining the meeting, like this:
Then, click "Join Meeting" to join the meeting!
Joining a meeting involves a two step process of calling the initializeMeeting and joinMeeting API calls. These must be "chained" as you cannot join a meeting until the initialization process completes. This can be achieved with the following code:
In the iOS SDK Reference App, we make the chained calls to initializeMeeting() and joinMeeting() right in the SettingsTableViewController() class. Once we know the calls are successful, we'll switch to the "Videos" tab in the application where we'll continue to process the meeting.
Enabling Local Audio and Video
Part of the "Join Meeting" process should be to automatically enable outgoing video as well as incoming/outgoing audio for your device (once you have successfully joined the meeting). You can do this with the following code:
Setting the Delegate
Once a meeting is successfully joined, the SDK will attempt to notify you of any "meeting events" through an instance of the MeetingSDKDelegate. This delegate class should be implemented by one of the classes in your application, and an instance of that class needs to be assigned to the delegate property in the MeetingSDK singleton. This usually happens just before you start the "join meeting" process. First, you need to have a class that implements the delegate protocol. In the iOS Reference Application, that is the VideoGalleryViewController class:
// Manages the "Videos" view controller which will allow the user to drag and resize
// a number of ParticipantView objects on a 5000x5000 scroll view
class VideoGalleryViewController: UIViewController, MeetingSDKDelegate {
// Rest of implementation here
...
}
Then, you need to make sure you set the delegate in the MeetingSDK singleton. In the case of our VideoGalleryViewController class, this happens right at the beginning of the viewDidLoad() method:
override func viewDidLoad() {
super.viewDidLoad()
// Setting the delegate of the MeetingSDK
MeetingSDK.shared.delegate = self
Once the delegate is set, you will begin to see delegate methods being called when meeting events occur. Such events include, but are not limited to, participants joining the meeting, video streams being added and removed from the meeting, and participants leaving the meeting. We'll cover how to respond to such events in the "Responding to Meeting Events" section below.
Enabling/Disabling Local Audio and Video, Exiting the Meeting
Once the user has joined a meeting, it is common for your user interface to provide controls to allow the user to mute/unmute their audio, disable/enable their outgoing video and to exit the meeting. While there is an "Exit Meeting" button in the Settings tab, The reference app also provides the following "convenience" buttons in the Videos tab:
The "Microphone" and "Video" buttons are "Toggle" buttons that, when tapped, will be shown with a slash through them to indicate that either video or audio is not being sent up into the meeting. This is implemented through the SDK as follows:
Mute/Unmute Local Audio
To mute or unmute video, use either the disableAudioInput API call (to mute) or the enableAudioInput API call (to unmute). You must pass the name of the audio device as returned by the getAudioInputDevices call.
Enable/Disable Local Video
To disable local video, you will use the disableVideoCapture API call on all video devices to be disabled:
Keep in mind that we've already covered how to enable local video in the "Enable Audio/Video for Meeting" recipe above.
Exit the Meeting
You will most likely have an "Exit Meeting" type button in your user interface. The iOS MeetingSDK Reference app does, and uses the following code to exit the meeting when the button is tapped (either the text button in the Settings panel or the "phone" icon in the Video Gallery view:
Controlling Input Volume, Overall Output Volume
You will notice that at the bottom the "Settings" tab in the reference app there are two sliders. One is to control the "input level" of the audio input device (independent of what the operating system settings may be) and the other is to control the "output level" of the audio output device (again, independent of what the operating system setting may be). The reference app handles this by implementing action routines that are fired whenever the value of either slider is changed (defined in SettingsTableViewController.swift):
// If the inputLevel slider is changed this will be called. Again, we'll make the right MeetingSDK call
// to respond, but right now it will do nothing
@IBAction func inputLevelSliderChanged(_ sender: Any) {
let slider = sender as? UISlider
if let volume = slider?.value {
print("setting input level to \(volume)")
MeetingSDK.shared.setAudioInputVolume(Int32(volume))
}
}
// If the outputLevel slider is changed this will be called. Again, we'll make the right MeetingSDK call
// to respond, but right now it will do nothing
@IBAction func outputLevelSliderChanged(_ sender: Any) {
let slider = sender as? UISlider
if let volume = slider?.value {
print("setting output level to \(volume)")
MeetingSDK.shared.setAudioOutputVolume(Int32(volume))
}
}
As the comments in the source code above indicate, right now the code shown will have no effect. The SDK defines (for both MacOS and iOS) API calls named setAudioInputVolume() and setAudioOutputVolume() which work as expected in MacOS but do not currently work in iOS. This is due to issues with third party libraries used to manipulate audio in iOS. We hope to have this working in a future release of the Meeting SDK for iOS.
Responding to Meeting Events
The iOS SDK Reference app uses an "Attendees" table view to show all participants in the active meeting (in the "Attendees" tab). While the meeting is active, all "meeting events" are reflected through this table view and any video views that appear in the Video Gallery for the participants in the meeting. The reference app responds to meeting events by using an instance of the MeetingSDKDelegate.
We'll take a look at how the iOS SDK Reference App implements various delegate methods. We can inspect the implementation of each delegate method and will provide further recipes for any implementation that involves making additional SDK calls!
participantAdded
We receive a participantAdded delegate method call when we see a new participant for the first time. At the low level of the SDK, we might have been notified of an audio or video stream for this user (they will always have an audio stream, but if they are joining with audio and video we might be notified of the video stream first). Depending on how you code your user interface, you may use this opportunity to adjust the displayed list of meeting attendees. In the iOS Reference App, we don't implement participantAdded() because we rely on the Attendee View Controller's viewWillAppear method to query the SDK for a list of participants in the meeting to display in that view controller's table view.
participantVideoAdded
The participantVideoAdded delegate method will be called for each video stream added to the meeting by each Participant. Remember that a participant may have multiple video streams associated with them. The action to take when this method is called is to ask the SDK to enable the video stream. That will cause the SDK to create an UIView object in which the video will be rendered. The reference app uses this code:
// Whenever we receive notification that there's a new video stream in the meeting, we immediately
// enable it so that we'll end up showing the video in our user interface
func participantVideoAdded(participant: Participant, streamId: String) {
print("iOSReferenceApp::participantVideoAdded")
MeetingSDK.shared.enableVideoStream(participant: participant, streamId: streamId)
}
The two parameters you need to pass to enableVideoStream() are provided as the arguments to the delegate method:
participantVideoUpdated
The participantVideoUpdated method is called when any of the "metadata" associated with a given video stream is updated for a video stream that we've already received a participantVideoAdded delegate method callback for. Consult the documentation for the VideoInfo structure (in the SDK Reference) to see what fields could have been changed. The reference app (being a simple application) does not take any action in this method. This is because it may have already responded to events that caused some of the values in the VideoInfo structure to change and/or it doesn't need to respond to them. Here are some of the scenarios where you might care about these values changing:
- The 'site' field might change if the user changes the name they are sending for themselves in the middle of the meeting. You could conceivably update any name you are displaying for the user by utilizing the 'displayName' property of the 'Participant' object when this callback occurs.
- The 'deviceName' field could conceivably change but it is not likely. This would be the name of the remote user's camera device OR the name of a window/desktop they are sharing into the meeting.
- The 'active' field may change to "false" if the local display of a video stream in the meeting is disabled. However you would already know that is happening as you would have to make a disableVideoStream SDK call to trigger this (covered later)
- The 'codec_name' field can change if a meeting participant decides to change the resolution of their camera. This won't necessarily require you to take any action unless you would like to attempt to resize the VideoView being used to render the corresponding window (otherwise the SDK will still render the video in the VideoView as it is currently sized)
- The 'layout' string may change. This is a value (usually an integer) that can be used by some clients to dictate how to display/tile windows. The SDK does not (currently) provide a way to set this value, so it is of limited use.
- The 'width' and 'height' values of the source video may change if the device is rotated. In this scenario you need to be prepared to "re-enable" the video stream so that you begin looking for video frames in the new size. The reference app uses the following code:
func participantVideoUpdated(participant: Participant, streamId: String, videoView: VideoView) {
print("iOSReferenceApp::participantVideoUpdated")
for videoInfo in participant.videoInfo {
if videoInfo.active != "false" {
// If active is not false we remove the current video view and re enable the video stream
for (index, view) in self.participantViews.enumerated() {
if view.videoView == videoView {
participantViews.remove(at: index)
DispatchQueue.main.async {
// Remove the ParticipantView from the scroll view
view.removeFromSuperview()
}
}
}
MeetingSDK.shared.enableVideoStream(participant: participant, streamId: streamId)
} else {
print("iOSReferenceApp::participantVideoUpdated -- videoInfo.active was FALSE")
}
}
}
What happens above is we look through all the video streams associated with the participant. For those that are active, (active field not equal to "false") we check to see if the VideoView associated with the stream is the one passed to the delegate method (and hence the one that was updated). If this is true, we remove it from the view hierarchy and then call enableVideoStream() on that video stream's streamId again. This will trigger events that will cause the reference app to re-add the VideoView to the view hierarchy with the updated parameters.
participantVideoViewCreated
Called when you have enabled a new video stream and the SDK is ready to hand you a new VideoView object to add to your user interface. The VideoView object is derived from UIView and can be added with a standard addSubview UIView Framework call. The reference app handles it this way:
// This is a private utility function used to create a new ParticipantView and add it to
// the user interface
private func makeAndAddNewParticipantView(participant: Participant, videoView: VideoView) {
// Adding video view to the scroll view
let genericFrame = CGRect(x: 0, y: 0, width: 200, height: 200)
let participantView = ParticipantView(frame:genericFrame)
participantView.displayName = participant.displayName
participantView.videoView = videoView
participantView.videoView?.frameView?.contentMode = .scaleAspectFit
// Locate the stream id
for videoInfo in participant.videoInfo {
if (videoInfo.videoView == videoView) {
participantView.streamId = videoInfo.streamId
}
}
participantViews.append(participantView)
// Adding pan gesture to make video view draggable
let panGesture = UIPanGestureRecognizer(target: self, action: #selector(panView))
participantView.addGestureRecognizer(panGesture)
// Adding pinch gesutre to make video view pinchable to resize it
let pinchGestureRecognizer = UIPinchGestureRecognizer(target: self, action: #selector(self.pinchToResize(_:)))
participantView.addGestureRecognizer(pinchGestureRecognizer)
self.scrollView.addSubview(participantView)
}
// This delegate method is received when the SDK has created a new VideoView for us in response
// to an enableVideoStream call. We utilize the private makeAndAddNewParticipantView function
// (above) to add the new view to our user interface
func participantVideoViewCreated(participant: Participant, videoView: VideoView, local: Bool) {
print("iOSReferenceApp::videoViewCreated")
DispatchQueue.main.async {
self.makeAndAddNewParticipantView(participant: participant, videoView: videoView)
}
}
The 'makeAndAddNewParticipantView' function is defined in the reference app's VideoGalleryViewController.swift file. It instantiates a new instance of ParticipantView using the VideoView passed and then will make sure the ParticipantView is added as a subview to the Video Gallery's scrolling view of videos.
participantVideoViewRetrieved
Called as a result of calling the enableVideoStream() SDK call for a video stream that a VideoView had already been created for once. This is a rare occurrence, but can happen if you were to call disableVideoStream for a given stream id directory followed by calling enableVideoStream with the same stream id later. This serves as an indicator to you that the VideoView passed to the delegate method is about to become active again, and you may need to take measures in your user interface to allow that. The reference app will remove a user's ParticipantView if the corresponding video stream is disabled, so this delegate method will check to see if we already know about the ParticipantView. If we do, we don't need to do anything as it is already being displayed. Otherwise, we'll take the necessary steps to create a new ParticipantView for the re-enabled video stream and add it back into the UI.
// This delegate method is received when enableVideoStream is called on a VideoView that already
// exists. This can happen if an existing video stream is disabled and then re-enabled
func participantVideoViewRetrieved(participant: Participant, videoView: VideoView) {
print("iOSReferenceApp::participantVideoViewRetrieved")
// If we don't have this videoView in our list of ParticipantViews, create and add
for participantView in self.participantViews {
if participantView.videoView == videoView {
// we already have it, so just return
return
}
}
// If we get here, we don't have this VideoView in our participant views.
// Go ahead and add it. The active flag is likely still "false" as we will get
// a participantVideoViewRetrieved event when the enableVideoStream API is called.
// The active flag is set when we receive the corresponding video_stream_buffer_ready event.
for videoInfo in participant.videoInfo {
// Find corresponding videoInfo structure
if videoInfo.videoView == videoView {
// add the view
DispatchQueue.main.async {
videoInfo.videoView?.frameView?.image = nil
self.makeAndAddNewParticipantView(participant: participant, videoView: videoView)
}
}
}
}
participantVideoRemoteLayoutChanged
As mentioned as part of the 'participantVideoUpdated' callback documentation, there is a field in the VideoInfo structure named 'layout' which can be used to give hints to a meeting application on how to lay out video views in the client application. This method is triggered if the value associated with the 'layout' field is changed. The reference app does not attempt to do anything with this field so, in the reference app, this delegate method implementation takes no action.
participantVideoRemoved
Called when a video stream is removed from the meeting. Your user interface should remove any UI elements associated with this video stream. The reference app handles it this way:
// Called when a given video stream is removed from a meeting
func participantVideoRemoved(participant: Participant, streamId: String, videoView: VideoView?) {
for participantView in participantViews {
if participantView.streamId == streamId {
DispatchQueue.main.async {
participantView.removeFromSuperview()
guard let index = self.participantViews.firstIndex(of: participantView) else
{
print("iOSReferenceApp::participantVideoRemoved: Could not find index of view to be removed")
return
}
self.participantViews.remove(at: index)
print("iOSReferenceApp::participantViews now: \(self.participantViews)")
}
}
}
print("iOSReferenceApp::participantVideoRemoved")
}
participantRemoved
This delegate method is called when the participant passed as the first argument has exited the meeting. The implementation of this delegate method will clean up all model data stored in the user interface as well as removing any video windows associated with this participant. It will also update the list of participants shown in the table view by calling reloadData() on the main thread.
// When a participant is removed, we pro-actively remove all video views associated with it
func participantRemoved(participant: Participant) {
print("iOSReferenceApp::participantRemoved")
// Remove all video streams associated with this participant
for videoInfo in participant.videoInfo {
participantVideoRemoved(participant: participant, streamId: videoInfo.streamId, videoView: videoInfo.videoView)
}
}
Here we simply iterate through all VideoView records associated with this participant and call the participantVideoRemoved callback for each.
participantDidMute
Called when a remote participant mutes their audio. The reference app does not implement this method but it could be used to provide a visual indicator that this particular participant's audio is muted
participantDidUnmute
Called when a remote participant unmutes their audio. The reference app does not implement this method but it could be used to provide a visual indicator that this particular participant's audio is no longer muted
inputMeetingChanged
Called whenever the sdk detects that the audio input of the local user has changed. If the local user is speaking, there will be many of these calls! The reference app does not implement this method but it could be used to provide a visual indicator of the audio input level resulting from the local user speaking. While the argument is a string, the value can be interpreted as an integer value.
outputMeterChanged
Called whenever the sdk detects that the audio level coming from the meeting has changed. This will be called frequently as people in the meeting talk! The reference app does not implement this method but it could be used to provide a visual indicator of the audio output level coming from the remote users. While the argument is a string, the value can be interpreted as an integer value.
amplitude
Called whenever the sdk detects that the audio level coming from an individual user in the meeting has changed. This will be called frequently as people in the meeting talk! The reference app does not implement this method but it could be used to provide a visual indicator of the audio output level of a particular user in the meeting. While the argument is a string, the value can be interpreted as an integer value.
Miscellaneous Tasks
As a reminder, the reference app has an "Attendees" tab which contains a table view with all the participants who are currently in the meeting:
You will notice that there is a "See Video Info" button that shows for every participant in the meeting. There will also be a Volume slider to control that user's individual volume level in the meeting. If the participant is the local participant, there will be no volume control. All entries on this screen should have a "See Video Info" button to bring up information about the video streams associated with participant. Tapping on the "See Video Info" button will bring up a screen similar to this:
Enabling/Disabling Video
Earlier, we discussed how to enable/disable local video. In that context, we were talking about how to turn local video sources on/off and send/stop sending the video captured up into the meeting. In this context, enabling/disabling video refers to whether or not we want the SDK to actively render video streams associated with a given video stream id. It has nothing to do with whether or not the video is present in the meeting, just whether or not the local client will render it. We can chose to enable/disable both remote/local video sources -- we just need to know the stream id.
For each video stream associated with this participant, you will see a big table view cell with all sorts of information about the stream. Along with this you should see a "Disable Video Stream" button or an "Enable Video Stream" button. If you see "Disable Video Stream" that means that the associated video stream is showing in the user interface and you can tap on this button to disable it. If it is already disabled, this button will read "Enable Video Stream" and it will provide a way to re-enable the video in the user interface.
The code to handle enabling/disabling video using this interface is contained in the VideoInfoViewController.swift file:
@objc func enableDisableVideoStream(_ sender: UIButton) {
let buttonTag = sender.tag
let videoInfo = videoInfo[buttonTag]
if videoInfo.active == "true" {
disableVideoStream(streamId: videoInfo.streamId)
NotificationCenter.default.post(name: NSNotification.Name("videoStreamDisabled"), object: videoInfo.videoView)
} else {
enableVideoStream(streamId: videoInfo.streamId)
}
self.navigationController?.popViewController(animated: true)
}
As you can see, we determine if we need to enable or disable the video by looking at the "active" field of the associated VideoInfo record. If we are going to disable the video stream we will call disableVideoStream() as shown above. Once we've called disableVideoStream() we won't get any frame updates for that window. We send an NSNotification that will be observed in the VideoGalleryViewController.swift file which will remove the appropriate ParticipantView from the user interface when the video stream is disabled:
// The selector called when a videoStreamDisabled NSNotification is received. We'll
// get the associated VideoView as the notification object; we just need to map it to
// the appropriate ParticipantView and then remove that participant view from our array
// of objects and from the view hierarchy
@objc func videoStreamDisabled(notification: NSNotification) {
guard let videoView = notification.object as? VideoView else { return }
for (index, view) in participantViews.enumerated() {
if view.videoView == videoView {
participantViews.remove(at: index)
DispatchQueue.main.async {
view.removeFromSuperview()
}
}
}
}
If we need to enable the stream, we call enableVideoStream() instead.
Changing the Audio Level of a Remote Participant
Changing the audio level for a particular remote participant is handled with the setAudioStreamVolume() SDK API call. It takes a Int32 value between 0 - 100. The reference app implements a slider control in the Attendees view for each attendee that is set to have a min value of 0 and a max of 100. Then all that needs to happen is to implement an action routine that responds when the slider value changes:
@objc func sliderValueChanged(sender: UISlider, event: UIEvent) {
var audioInfo: [AudioInfo] = []
for participant in MeetingSDK.shared.participants {
if let info = participant.audioInfo {
audioInfo.append(info)
}
}
// When receiving slider events, wait until we get the last one to set the volume stream
// This is not absolutely necessary (we could keep calling the APIs to change the stream's
// volume on every minute change of the slider) but this cuts down on unnecessary intermediate
// calls to the SDK.
if let touchEvent = event.allTouches?.first {
let currentValue = Int(sender.value)
// We've stored the streamId in the slider's tag field. If we didn't have a stream id
// this will be 0 which we'll deal with below
let streamId = String(sender.tag)
switch touchEvent.phase {
case .ended:
// Make sure we have a valid stream id
if sender.tag > 0 {
audioStreams[streamId] = currentValue
MeetingSDK.shared.setAudioStreamVolume(streamId: streamId, volume: Int32(currentValue))
}
default:
break
}
}
}
We wait until the we receive the last touch event when interacting with the slider to call setAudioStreamVolume().
Summary
This has been a general overview of the iOS SDK Reference application and how it interacts with the Visionable Meeting SDK (high level).
Updated over 1 year ago