ML Kit adds face contours to create smarter visual apps

November 6, 2018

If you’re building or looking to build a visual app, you’ll love ML Kit’s new face contour detection. With ML Kit, you can take advantage of many common Machine Learning (ML) use-cases, such as detecting faces using computer vision. Need to know where to put a hat on a head in a photo? Want to place a pair of glasses over the eyes? Or maybe just a monocle over the left eye. It’s all possible with ML Kit’s face detection. In this post we’ll cover the new face contour feature that allows you to build better visual apps on both Android or iOS.

Features

Detect facial contours

With just a few configuration options you can now detect detailed contours of a face. Contours are a set of over 100 points that outline the face and common features such as the eyes, nose and mouth. You can see them in the image below. Note that as the subject raises his eyebrows, the contour dots move to match it. These points are how advanced camera apps set creative filters and artistic lenses over a user’s face.

Setting up the face detector to detect these points only takes a few lines of code.

lazy var vision = Vision.vision()
let options = VisionFaceDetectorOptions()
options.contourMode = .all
let faceDetector = vision.faceDetector(options: options) 

The contour points can update in realtime as well. To achieve an ideal frame rate the face detector is configured with the fast mode by default.

When you’re ready to detect points in a face, send an image or a buffer to ML Kit for processing.

faceDetector.process(visionImage) { faces, error in
  guard error == nil, let faces = faces, !faces.isEmpty else  return 
  for face in faces 
    if let faceContour = face.contour(ofType: .face) 
      for point in faceContour.points 
        print(point.x) 
        print(point.y) 
      
   
      

ML Kit will then give you an array of points that are the x and y coordinates of the contours in the same scale as the image.

Detect the location of facial features

The face detector can also detect landmarks within faces. A landmark is just an umbrella term for facial features like your nose, eyes, ears, and mouth. We’ve dramatically improved its performance since launching ML Kit at I/O!

To detect landmarks configure the face detector with the landmarkMode option:

lazy var vision = Vision.vision()
let options = VisionFaceDetectorOptions()
options.landmarkMode = .all
let faceDetector = vision.faceDetector(options: options)

Then pass an image into the detector to receive and process the coordinates of the detected landmarks.

faceDetector.process(visionImage)  faces, error in
  guard error == nil, let faces = faces, !faces.isEmpty else  return 
  for face in faces 
    
    if let leftEye = face.landmark(ofType: .leftEye) 
      
      print(leftEye.position.x) 
      print(leftEye.position.y) 
    
  

We can’t wait to see what you’ll build with ML Kit

Hopefully these new features can empower you to easily build smarter features into your visual apps. Check out our docs for iOS or Android to learn all about face detection with ML Kit. Happy building!