Background Removal for Video Conferences

API Interface of the iOS Framework

Define selfie segmentation model variable

private var selfieSegmentationModel: SelfieSegmentationModel?

Configure and Instantiate the model with SelfieSegmentationModelBuilder

@interface SelfieSegmentationModelBuilder : NSObject

- (instancetype _Nonnull)init;

/**
 * \brief Creates a new instance of \a SelfieSegmentationModel.
 *
 * \param error Object containing error information if model instantiation fails.
 *
 * \returns Pointer to the new instance of \a SelfieSegmentationModel if instantiation
 * is successful, \a nil otherwise.
 *
 * \note Model instantiation is a blocking call which can take some time, therefore
 * this should be done on a separate serial dispatch queue.
 * That won't block the main queue which keeps the UI responsive.
 */
- (SelfieSegmentationModel* _Nullable)build:(NSError* _Nullable* _Nonnull)error;

@end

Example:

do {
    self.selfieSegmentationModel = try SelfieSegmentationModelBuilder()
        .build()
} catch {
    fatalError(
        "Failed to instantiate selfie segmentation model: \(error.localizedDescription)"
    )
}

Model instantiation is a blocking call that can take some time, therefore this should be done on a separate serial dispatch queue. That won't block the main queue which keeps the UI responsive.

Schedule the task with SelfieSegmentationModel.segment method when the model is instantiated

func captureOutput(_ output: AVCaptureOutput,
                   didOutput sampleBuffer: CMSampleBuffer,
                   from connection: AVCaptureConnection) {
    guard let imageBuffer = sampleBuffer.imageBuffer else {
        return
    }

    do {
        try selfieSegmentationModel!.segment(on: imageBuffer,
                                             at: sampleBuffer.outputPresentationTimeStamp)
    } catch {
        NSLog("Failed to submit selfie segmentation task: \(error.localizedDescription)")
    }
}

SelfieSegmentationModel returns its results through the SelfieSegmentationDelegate

@protocol SelfieSegmentationDelegate <NSObject>

/**
 * \brief Callback triggered whenever the \a SelfieSegmentationModel completes the
 * processing of the passed frame.
 *
 * \param model The \a SelfieSegmentationModel that acquired provided mask
 * \param mask The MaskData instance with representation of mask values
 */
@optional
- (void)selfieSegmentationModel:(SelfieSegmentationModel* _Nonnull)model
       didOutputMask:(MaskData* _Nonnull)mask;

@end

Example:

func selfieSegmentationModel(_ model: SelfieSegmentationModel,
                             didOutputMask mask: MaskData) {
    cameraPreviewView.draw(frame: lastCapturedFrame, mask: mask)
}

Each MaskData instance is represented with the following class

@interface MaskData : NSObject

 /**
 * \brief Width of the mask.
 */
@property (readonly, nonatomic) int width;

 /**
 * \brief Height of the mask.
 */
@property (readonly, nonatomic) int height;

 /**
 * \brief Creates a non-owning NSData view to the raw mask bytes.
 *
 * \returns Pointer to the instance of \a NSData, which represents the
 * raw binary mask values.
 */
- (NSData* _Nonnull) getDataView;

 /**
 * \brief Converts the mask to UIImage format.
 *
 * \returns Pointer to the instance of \a UIImage, where the foreground has zero values
 * and background has non-zero alpha values.
 *
 * \note The mask should be overlaid on the image. The background will be replaced
 * by black color in the final image.
 */
- (UIImage*) asUIImage;

 /**
 * \brief Applies blur filter to the image background using the mask.
 *
 * \param image The image in UIImage format, for which the mask was created.
 *
 * \returns Pointer to the instance of \a UIImage containing the submitted image
 * with the blurred background.
 */
- (UIImage*) applyBlur:(UIImage*)image;

@end