BBMetalImage

A high performance Swift library for GPU-accelerated image/video processing based on Metal.

This library is highly inspired by GPUImage.

Performance

Test libraries are BBMetalImage (0.1.1) and GPUImage (0.1.7). Test device is iPhone 7 with iOS 12.1. The code can be found in CompareImageLib project and the test result data can be found in CompareImageLib.numbers.

• BBMetalImage has low memory usage for processing image.

• BBMetalImage has low CPU usage and high speed for camera capturing, processing and rendering.

Features

• [x] 80+ built-in filters
• [x] Filter chain supported
• [x] Customized filter
• [x] Camera capturing video and audio
• [x] Depth camera supported
• [x] Multiple camera supported
• [x] Video source processing video file
• [x] Image source providing image texture
• [x] UI source recording view animation
• [x] Metal view displaying Metal texture
• [x] Video writer writting video
• [x] High performance

Requirements

• iOS 10.0+
• Swift 5

Installation

Install with CocoaPods:

• Add use_frameworks! and pod 'BBMetalImage' to your Podfile.
• Run pod install or pod update.
• Add import BBMetalImage to the Swift source file.

How to Use

Demo

View sample code in demo project.

Single Filter

Call filteredImage(with:) function of a filter is the simplest way to get filtered image synchronously.

let filteredImage = BBMetalContrastFilter(contrast: 3).filteredImage(with: image)

Filter Chain

Capture, Preview and Recording

The code below:

• Captures image and audio with a camera
• The image captured by the camera is processed by 3 filters
• The processed image is rendered to the metal view
• The processed image and audio are written to a video file
• Do something after writing the video file

// Hold camera and video writer
var camera: BBMetalCamera!
var videoWriter: BBMetalVideoWriter!
func setup() {     // Set up camera to capture image     camera = BBMetalCamera(sessionPreset: .hd1920x1080)
// Set up 3 filters to process image     let contrastFilter = BBMetalContrastFilter(contrast: 3)     let lookupFilter = BBMetalLookupFilter(lookupTable: UIImage(named: "testlookup")!.bbmetalTexture!)     let sharpenFilter = BBMetalSharpenFilter(sharpeness: 1)
// Set up metal view to display image     let metalView = BBMetalView(frame: frame)     view.addSubview(metalView)
// Set up video writer     let filePath = NSTemporaryDirectory() + "test.mp4"     let url = URL(fileURLWithPath: filePath)     videoWriter = BBMetalVideoWriter(url: url, frameSize: camera.textureSize)
// Set camera audio consumer to record audio     camera.audioConsumer = videoWriter
// Set up filter chain     camera.add(consumer: contrastFilter)         .add(consumer: lookupFilter)         .add(consumer: sharpenFilter)         .add(consumer: metalView)
sharpenFilter.add(consumer: videoWriter)
// Start capturing     camera.start()
// Start writing video file     videoWriter.start() }
func finishRecording() {     videoWriter.finish {         // Do something after recording the video file     } }

Capture Image

There are two ways to capture image with camera. Use capturePhoto(completion:) function or takePhoto() function.

The capturePhoto(completion:) function runs faster and provides original frame texture. If the filter chain contains filter, we can get filtered frame texture with addCompletedHandler(_:) function.

// Hold camera
var camera: BBMetalCamera!
func setup() {     // Set up camera to capture image     camera = BBMetalCamera(sessionPreset: .hd1920x1080)
// Set up metal view to display image     let metalView = BBMetalView(frame: frame)     view.addSubview(metalView)
// Set up filter to process image     let filter = BBMetalContrastFilter(contrast: 3)          // Add completed handler to get filtered image     filter.addCompletedHandler { [weak self] info in         // Check whether is camera photo         guard info.isCameraPhoto else { return }         switch info.result {         case let .success(texture):             // Convert filtered texture to image             let image = texture.bb_image             DispatchQueue.main.async {                 guard let self = self else { return }                 // Display filtered image             }         case let .failure(error):             // Handle error         }     }          // Set up filter chain     camera.add(consumer: filter)         .add(consumer: metalView)
// Start capturing     camera.start() }
func takePhoto() {     camera.capturePhoto { [weak self] info in         // No need to check whether is camera photo         switch info.result {         case let .success(texture):             // Convert filtered texture to image             let image = texture.bb_image             DispatchQueue.main.async {                 guard let self = self else { return }                 // Display filtered image             }         case let .failure(error):             // Handle error         }     } }

The takePhoto() functions runs slower and provides original frame texture.

// Hold camera
var camera: BBMetalCamera!
func setup() {     // Set up camera to capture image     // Set canTakePhoto to true and set photoDelegate to nonnull     camera = BBMetalCamera(sessionPreset: .hd1920x1080)     camera.canTakePhoto = true     camera.photoDelegate = self
// Set up metal view to display image     let metalView = BBMetalView(frame: frame)     view.addSubview(metalView)
// Set up filter chain     camera.add(consumer: metalView)
// Start capturing     camera.start() }
func takePhoto() {     camera.takePhoto() }
// BBMetalCameraPhotoDelegate func camera(_ camera: BBMetalCamera, didOutput texture: MTLTexture) {     // Do something to the photo texture     // Note: the texture is the original photo which is not filtered even though there are filters in the filter chain }

Process Video File

// Hold video source and writer
var videoSource: BBMetalVideoSource!
var videoWriter: BBMetalVideoWriter!
func setup() {     // Set up video writer     let filePath = NSTemporaryDirectory() + "test.mp4"     let outputUrl = URL(fileURLWithPath: filePath)     videoWriter = BBMetalVideoWriter(url: outputUrl, frameSize: BBMetalIntSize(width: 1080, height: 1920))
// Set up video source     let sourceURL = Bundle.main.url(forResource: "testvideo2", withExtension: "mov")!     videoSource = BBMetalVideoSource(url: sourceURL)
// Set video source audio consumer to write audio data     videoSource.audioConsumer = videoWriter
// Set up 3 filters to process image     let contrastFilter = BBMetalContrastFilter(contrast: 3)     let lookupFilter = BBMetalLookupFilter(lookupTable: UIImage(named: "testlookup")!.bbmetalTexture!)     let sharpenFilter = BBMetalSharpenFilter(sharpeness: 1)
// Set up filter chain     videoSource.add(consumer: contrastFilter)         .add(consumer: lookupFilter)         .add(consumer: sharpenFilter)         .add(consumer: videoWriter)
// Start receiving Metal texture and writing video file     videoWriter.start()
// Start reading and processing video frame and auido data     videoSource.start { [weak self] (_) in         // All video data is processed         guard let self = self else { return }
// Finish writing video file         self.videoWriter.finish {             // Do something after writing the video file         }     } }

Process Image Synchronously

// Set up image source
let imageSource = BBMetalStaticImageSource(image: image)
// Set up 3 filters to process image let contrastFilter = BBMetalContrastFilter(contrast: 3) let lookupFilter = BBMetalLookupFilter(lookupTable: UIImage(named: "testlookup")!.bbmetalTexture!) let sharpenFilter = BBMetalSharpenFilter(sharpeness: 1)
// Set up filter chain // Make last filter run synchronously imageSource.add(consumer: contrastFilter)     .add(consumer: lookupFilter)     .add(consumer: sharpenFilter)     .runSynchronously = true
// Start processing imageSource.transmitTexture()
// Get filtered image let filteredImage = sharpenFilter.outputTexture?.bb_image

Process Image Asynchronously

// Hold image source
var imageSource: BBMetalStaticImageSource!
func process() {     // Set up image source     imageSource = BBMetalStaticImageSource(image: image)          // Set up 3 filters to process image     let contrastFilter = BBMetalContrastFilter(contrast: 3)     let lookupFilter = BBMetalLookupFilter(lookupTable: UIImage(named: "testlookup")!.bbmetalTexture!)     let sharpenFilter = BBMetalSharpenFilter(sharpeness: 1)          // Set up filter chain     // Add complete handler to last filter     weak var wLastFilter = sharpenFilter     imageSource.add(consumer: contrastFilter)         .add(consumer: lookupFilter)         .add(consumer: sharpenFilter)         .addCompletedHandler { [weak self] _ in             if let filteredImage = wLastFilter?.outputTexture?.bb_image {                 DispatchQueue.main.async {                     guard let self = self else { return }                     // Display filtered image                 }             }     }          // Start processing     imageSource.transmitTexture() }

Record a view animation

Use BBMetalUISource to capture UIView snapshot and transmit texture.

// Hold UI source and video writer
var uiSource: BBMetalUISource!
var videoWriter: BBMetalVideoWriter!
func setup() {     // Set up UI source with a view     uiSource = BBMetalUISource(view: animationView)          // Set up filter     let filter = BBMetalContrastFilter(contrast: 3)          // Set up video writer     let filePath = NSTemporaryDirectory() + "test.mp4"     let outputUrl = URL(fileURLWithPath: filePath)     let frameSize = uiSource.renderPixelSize!     videoWriter = BBMetalVideoWriter(url: outputUrl, frameSize: BBMetalIntSize(width: Int(frameSize.width), height: Int(frameSize.height)))          // Set up filter chain     uiSource.add(consumer: filter)         .add(consumer: videoWriter)          // Start recording     videoWriter.start() }
@objc func refreshDisplayLink(_ link: CADisplayLink) {     // Update UI     // ...          // Transmit texture and video frame sample time     // Repeat this step for each video frame     // If CADisplayLink is used for the view animation, repeat this step in the target selector     uiSource.transmitTexture(with: sampleTime) }
func finishRecording() {     videoWriter.finish {         // Do something after recording the video file     } }

Built-in Filters

• Brightness
• Exposure
• Contrast
• Saturation
• Gamma
• Levels
• Color Matrix
• RGBA
• Hue
• Vibrance
• White Balance
• Highlight Shadow
• Highlight Shadow Tint
• Lookup
• Color Inversion
• Monochrome
• False Color
• Haze
• Luminance
• Luminance Threshold
• Depth Luminance
• Erosion
• RGBA Erosion
• Dilation
• RGBA Dilation
• Chroma Key
• Crop
• Resize
• Rotate
• Flip
• Transform
• Sharpen
• Unsharp Mask
• Gaussian Blur
• Box Blur
• Zoom Blur
• Motion Blur
• Tilt Shift
• Blend Modes

- Normal - Chroma Key - Dissolve - Add - Subtract - Multiply - Divide - Overlay - Darken - Lighten - Color - Color Burn - Color Dodge - Screen - Exclusion - Difference - Hard Light - Soft Light - Alpha - Source Over - Hue - Saturation - Luminosity - Linear Burn - Mask

• Pixellate
• Polar Pixellate
• Polka Dot
• Halftone
• Crosshatch
• Sketch
• Threshold Sketch
• Toon
• Posterize
• Vignette
• Kuwahara
• Swirl
• Bulge
• Pinch
• Convolution 3x3
• Emboss
• Sobel Edge Detection
• Bilateral Blur
• Beauty

License

BBMetalImage is released under the MIT license. See LICENSE for details.