Build your own FaceID with Amazon ReKognition in Ruby

Amazon launched an amazing tool about one year go : Rekognition. It is a deep learning-based image and video analysis that provides highly accurate facial analysis and facial recognition.

Today, we are going to use Amazon Rekognition in Ruby in order to be able to recognize someone’s face through a camera. We could then create funny things like an app that plays a specific song around the office when a specific person enters or add your own FaceID system to your app :smile:

But first, how Amazon Rekognition works ?

Amazon Rekognition stores information about detected faces in server-side containers called collections. We can use this information to recognize faces in an image, a video and even a streaming video. So basically, in order to recognize someone in an image, we need 2 setup steps :

• Create a collection.
• Index faces on this collection.

Then, we can search a face in any image and Amazon Rekognition will detect the person, as long as we added this person to the collection.

Ok , it might be confusing at first, but it will be clearer after an example.

AWS Setup

The first thing to do is to create a bucket on AWS S3 and populate it with pictures of the people you want to be able to recognize later on. Go to you AWS console, and create a bucket :

Then, add pictures to your bucket :

For the purpose of this example, name your pictures name.jpg, other_name.jpg so later we can retrieve the name of the person. You can even add multiple pictures of the same person, it will increase the probability of AWS recognizing him/her. As you can see, I added 2 pictures : samir and samir2.

Files on our bucket are private by default, so now, we need to go to our Identity & Access Management (IAM) service and create a user linked to our bucket so we can grant the right permissions to our app. This step is crucial for security reasons.

Go to your IAM console, create a new user with the same name as your bucket, choose Programmatic access. Then, choose Add user to a group > Create group and create a group attached with AmazonRekognitionFullAccess policy. Create the user.

You should land on your credential page, so download them and store them in a safe place because you won’t be able to access it in the future. We will need those credentials in a few minutes.

The last thing to do is to add persmissions to our IAM user to access our bucket. To do so, go to your user on IAM, and click on the bottom-right button Add inline policy :

Choose Custom Policy > Select and enter the following JSON as the Policy Document :

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::rekognition-example",
                "arn:aws:s3:::rekognition-example/*"
            ]
        }
    ]
}

This will allow our app to list our bucket and access files stored in the rekognition-examplebucket. Don’t forget to replace rekognition-example with your bucket name. Ok, that’s it for the S3/IAM setup, now let’s go to our application.

Application Setup

Our application will be a really basic Sinatra app but you can use whatever framework you want. I tried to put as much code as possible so you can have a global view of what our app is doing.

Start by creating a rekognition_app folder, with main.rb and a Gemfile :

$ mkdir rekognition_app
$ cd rekognition_app
$ touch main.rb Gemfile

Next, add the required gems in Gemfile :

source 'https://rubygems.org'

gem 'sinatra'
gem 'dotenv'
gem 'aws-sdk'

And run :

$ bundle install

Ok, our application is setup, now we can go to our main.rb file and start coding !

The first thing we wan’t to do is to have a way of capturing a picture from the user webcam in order to use our recognition system. We will use the Javascript library JpegCamera in order to achieve this.

Create a views folder with an empty index.erb file and update main.rb :

require 'sinatra'

get '/' do
  erb :index
end

If you run $ ruby main.rb and go to localhost:4567 you should see an empty white page.

Ok, now let’s update our index.erb :

<html>
  <head>
    <title>Rekognition App</title>
    <link href="css/main.css" rel="stylesheet" type="text/css" />
  </head>
  <body>
    <h1>Face reKognition</h1>
    <div id='camera'>
      <div id='placeholder'>
        <p>Your browser does not support a camera!</p>
      </div>
    </div>
    <br>
    <p>
      <button id='rekognize-face'>Identify</button>
    </p>
    <div id='rekognition-status'></div>
    <div id='rekognition-result'></div>
    <script src='https://code.jquery.com/jquery-3.1.1.min.js'></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.17.1/moment.min.js"></script>
    <script src="js/jpeg_camera/swfobject.min.js" type="text/javascript"></script>
    <script src="js/jpeg_camera/canvas-to-blob.min.js" type="text/javascript"></script>
    <script src="js/jpeg_camera/jpeg_camera_with_dependencies.min.js" type="text/javascript"></script>
    <script src="js/main.js" type="text/javascript"></script>
  </body>
</html>

As you can see, we load a bunch of js files. They correspond to the JpegCamera library we are using. As we are not using Rails, we can go to JpegCamera’s github repository and copy all the files from dist into jpeg_camera directory under our server’s root. Here is all the files we should have :

All the files in public come from JpegCamera library, except from main.css and main.js.

We can kick off our camera as follow (in main.js) :

$(document).ready(function() {
  if (window.JpegCamera) {
    var options = {
      shutter_ogg_url: "js/jpeg_camera/shutter.ogg",
      shutter_mp3_url: "js/jpeg_camera/shutter.mp3",
      swf_url: "js/jpeg_camera/jpeg_camera.swf"
    }
    camera = new JpegCamera("#camera", options);
  }
});

And add some style in main.css :

#camera {
  display: inline-block;
  background-color: #eee;
  width: 250px;
  height: 300px;
  margin: 0.5em;
}

If you go to localhost:4567, everything should be working :

But if you click on Identify, of course nothing will happen because we haven’t implemented nothing yet. Let’s update our main.js file :

$(document).ready(function() {

  if (window.JpegCamera) {

    var options = {
      shutter_ogg_url: "js/jpeg_camera/shutter.ogg",
      shutter_mp3_url: "js/jpeg_camera/shutter.mp3",
      swf_url: "js/jpeg_camera/jpeg_camera.swf"
    }

    camera = new JpegCamera("#camera", options);

    var rekognify_face = function() {
      var snapshot = camera.capture();
      snapshot.upload({api_url: "/rekognify"}).done(function(response) {
        var data = JSON.parse(response);
        if (data.id !== undefined) {
          var result = data.message + ": " + data.id + ", Confidence: " + data.confidence
          $("#rekognition-result").html(result);
        } else {
          $("#rekognition-result").html(data.message);
        }
        this.discard();
      }).fail(function(status_code, error_message, response) {
        alert("Upload failed with status " + status_code);
      });
    };

    $("#rekognify-face").click(rekognify_face);
  }
});

The most part of this comes from JpegCamera library documentation, but basically, when we click the Identify button, we are taking a snapshot, sending it to /rekognify URL and then displaying the response.

But as you can see, we haven’t any rekognify URL yet… so let’s add it to our main.rb file :

require 'sinatra'

get '/' do
  erb :index
end

post '/rekognify' do
  content_type :json
end

Now we want to take the request (i.e the picture) and send it to Amazon Rekognition in order to recognize the person in this picture. We can update main.rb accordingly :

require 'sinatra'
require 'aws-sdk'
require 'dotenv'
Dotenv.load

require_relative 'lib/bucket'
require_relative 'lib/recognition'

get '/' do
  erb :index
end

post '/rekognify' do
  content_type :json
  response = Recognition.new(bucket: Bucket.new(name: 'rekognition-example')).recognize(request.body.read.to_s)

  if response.face_matches.count == 0
    {
      :message => "No faces were recognized..."
    }.to_json
  else
    {
      id: response.face_matches[0].face.external_image_id,
      confidence: response.face_matches[0].face.confidence,
      message: "Rekognition worked"
    }.to_json
  end
end

Retrieve objects from S3 bucket

Before indexing faces to our collection, we need to retrieve pictures that we previsouly stored on our S3 bucket. Let’s create a lib/bucket.rb file :

class Bucket
  attr_reader :name, :client

  def initialize(args = {})
    @client = Aws::S3::Client.new(region: ENV['AWS_REGION'])
    @name   = args[:name]
  end

  def get_objects
    client.list_objects_v2({
      bucket: name,
    })
  end
end

The get_objects method returns all the objects in the specified bucket. It uses the list_objects_v2 from AWS API which returns an object that responds to contents and bucket_name, beyond many other methods. Those 2 methods will be useful to us in the next steps.

Let’s create a .env file and add our credentials as follow :

AWS_REGION=eu-west-1
AWS_ACCESS_KEY_ID=EXAMPLE_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=EXAMPLE_SECRET_ACCESS_KEY

Implement our Recognition class

Now we need to implement the Recognition class with its public method recognize. Let’s create a file called recognition.rb and add :

require_relative 'recognition_processor'

class Recognition
  attr_reader :bucket, :recognition_processor

  def initialize(args = {})
    @bucket                = args[:bucket]
    @recognition_processor = RecognitionProcessor.new(collection_id: 'collection-faces')
  end

  def recognize(image_name)
    recognition_processor.recognize(image_name, bucket_objects)
  end

  private

  def bucket_objects
    bucket.get_objects
  end
end

As you can guess, our next step is to create the RecognitionProcessor.

Create a collection

In order to be able to recognize someone, we need to first create a collection in which we will “store” faces so AWS can run our subject against this collection. In order to do so, we will use AWS API and the method create_collection (pretty straightforward, isn’t it ?).

class RecognitionProcessor
  attr_reader :client, :collection_id

  def initialize(args = {})
    @collection_id = args[:collection_id]
    @client        = Aws::Rekognition::Client.new
  end

  def create_collection
    client.create_collection({collection_id: collection_id}) unless collection_exists?
  end

  def collection_exists?
    client.list_collections.collection_ids.include? collection_id
  end
end

Index faces to our collection

Now that we have created a collection and we have methods to retrieve objects from our bucket, we can go ahead and implement our index_faces_from method. This method takes an argument bucket_objects and iterate over each object in order to index faces of each object.

class RecognitionProcessor
  attr_reader :client, :collection_id

  def initialize(args = {})
    @collection_id = args[:collection_id]
    @client        = Aws::Rekognition::Client.new
  end

  def index_faces_from(bucket_objects)
    bucket_objects.contents.each do |object|
      index_faces(object_file_name: object.key, bucket_name: bucket_objects.name)
    end
  end

  private

  def index_faces(args = {})
    client.index_faces({
      collection_id: collection_id,
      detection_attributes: [
      ],
      external_image_id: get_name_from(args[:object_file_name]),
      image: {
        s3_object: {
          bucket: args[:bucket_name],
          name: args[:object_file_name],
        },
      },
    })
  end

  def get_name_from(file_name)
    file_name.gsub(/\.(.*)/, '')
  end
end

# We can use this method as follows
bucket_objects = Bucket.new(name: "bucket_name").get_objects
RecognitionProcessor.new(collection_id: "your_collection").index_faces_from(bucket_objects)

Again, we used AWS API with the method index_faces which takes an hash of parameters with, in particular, an external_image_id key.

Remember that Amazon does not actually store the picture when we index it, instead, it stores a JSON representation of this picture.

Note : we added a get_name_from method that removes the extension from a file name so we can get the name of the people we indexed.

Let’s reKognize !

Ok, so now we have everything ready to do the fun part : recognize faces. The idea is that we need a method able to take a picture as argument and return the result of the comparaison between this picture and all the pictures in our collection.

class RecognitionProcessor
  attr_reader :client, :collection_id

  def initialize(args = {})
    @collection_id = args[:collection_id]
    @client        = Aws::Rekognition::Client.new
  end

  def compare_faces_with_file(file)
    client.search_faces_by_image({
      collection_id: collection_id,
      face_match_threshold: 85,
      image: { bytes: file },
      max_faces: 3,
    })
  end
end

Again, AWS documentation is pretty straightforward for AWS Rekognition. We used the search_faces_by_image method that takes an hash as argument with:

• collection_id : the unique ID referencing our collection when we created it.
• face_match_treshold : which is the threshold in percentage of similarity by which AWS declares a match.
• image : here a file, but it could be a picture stored on S3 meaning that we would use the definition of our index_faces method.
• max_faces : limiting the number of matching faces AWS will return.

And here it is!

Now, if you go back to your browser and click on the Identify button, you should have this output under the button : Rekognition worked: samir-face, Confidence: 0.99 with samir being the name you gave your file on your S3 bucket.

Now you can use this in any way you want!

Resources

• Building a face recognition web app in under an hour
• How to detect image contents from Ruby with Amazon Rekognition