In this post, you’ll learn in-depth about the five of the most easiest and effective face detection options available in python, along with the pros and cons of each one of them. You will become capable of obtaining the required balance in accuracy, speed, and efficiency in any given scenario.
The face detection methods we will be covering are:
Face Detection is one of the most common and simplest vision techniques out there, as the name implies, it detects (i.e., locates) the faces in the images and is the first and essential step for almost every face application like Face Recognition, Facial Landmarks Detection, Face Gesture Recognition, and Augmented Reality (AR) Filters, etc.
Other than these, one of its most common applications, that you must have used, is your mobile camera which detects your face and adjusts the camera focus automatically in real-time.
Also, for what it’s worth Tony Stark’s EDITH (Even Dead I’m The Hero) glasses, inherited by Peter Parker in the Spider-Man Far From Home movie, also uses Face Detection as an initial step to perform its functionalities. Cool 😊 … right?
Yeah I know .. I know, I needed to add a marvel reference into it, the whole post get’s cooler.
Face detection also serves as a ground for a lot of exciting face applications for e.g. You can even appoint Mr. Beans as the President 😂 using Deepfake.
But for now, let’s just go back to Face Detection.
The idea behind face detection is to make the computer capable of identifying what human face exactly is and detecting the features that are associated with the faces in images/videos which might not always be easy because of changing facial expression, orientation, lighting conditions, and occlusions due to face masks, glasses, etc.
But with enough training data covering all the possible scenarios, you can create a very robust face detector.
And people throughout the years have done just that, they have designed various algorithms for facial detection and in this post, we’ll explore 5 such algorithms.
As this is the most common and widely used technique, there are a lot of face detectors out there.
But which Algorithm is the best?
If you’re looking for a single solution then it’s a hard answer as each of the algorithms that we’re going to cover has its own pros and cons, take a look at the demos at the end for some comparison, and make sure to read the summary for the final verdict.
Alright, so without further ado, let’s dive in.
[optin-monster slug=”rhkojx1lcwd45akbz8u8″]
Import the Libraries
We will first import the required libraries.
import os
import cv2
import dlib
from time import time
import mediapipe as mp
import matplotlib.pyplot as plt
Algorithm 1: OpenCV Haar Cascade Face Detection
This face detector was introduced in 2001 and remained the state-of-the-art face detection algorithm for many years. Other than just this face detector, OpenCV provides some other detectors (like eye, and smile, etc) too, which use the same haar cascade technique.
Load the OpenCV Haar Cascade Face Detector
To perform the face detection using this algorithm, first, we will have to load the pre-trained Haar cascade face detection model around 900 KBs from the disk, stored in a .xml file format, using the function CascadeClassifier().
# Load the pre-trained Haar cascade face detection model.
cascade_face_detector = cv2.CascadeClassifier("models/haarcascade_frontalface_default.xml")
cascade_face_detector
Create a Haar Cascade Face Detection Function
Now we will create a function haarCascadeDetectFaces() that will perform haar cascade face detection using the function cv2.CascadeClassifier.detectMultiScale() on an image/frame and will visualize the resultant image along with the original image (when working with images) or return the resultant image along with the output of the model (when working with videos) depending upon the passed arguments.
image – It is the input grayscale image containing the faces.
scaleFactor (optional) – It is the image size that is reduced at each image scale. Its default value is 1.1 which means a decrease of 10%.
minNeighbors (optional) – It is the number of minimum neighbors each predicted face should have, to retain. Otherwise, the prediction is ignored. Its default value is 3.
minSize (optional) – It is the minimum possible face size, the faces smaller than that size are ignored.
maxSize (optional) – It is the maximum possible face size, the faces larger than that are ignored. If maxSize == minSize then only the faces of a particular size are detected.
Returns:
results – It is an array of bounding boxes coordinates (i.e., x1, y1, bbox_width, bbox_height) where each bounding box encloses the detected face, the boxes may be partially outside the original image.
Note:When the value of the minNeighbors parameter is decreased, false positives are increased, and when the value of scaleFactor is decreased the large faces in the image become smaller and detectable by the algorithm at the cost of speed.
So the algorithm can detection very large and very small faces too by appropriately utilizing the scaleFactor argument.
def haarCascadeDetectFaces(image, cascade_face_detector, display = True):
'''
This function performs face(s) detection on an image using opencv haar cascade face detector.
Args:
image: The input image of the person(s) whose face needs to be detected.
cascade_face_detector: The pre-trained Haar cascade face detection model loaded from the disk required to
perform the detection.
display: A boolean value that is if set to true the function displays the original input image,
and the output image with the bounding boxes drawn and time taken written and returns
nothing.
Returns:
output_image: A copy of input image with the bounding boxes drawn.
results: The output of the face detection process on the input image.
'''
# Get the height and width of the input image.
image_height, image_width, _ = image.shape
# Create a copy of the input image to draw bounding boxes on.
output_image = image.copy()
# Convert the input image to grayscale.
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
# Get the current time before performing face detection.
start = time()
# Perform the face detection on the image.
results = cascade_face_detector.detectMultiScale(image=gray, scaleFactor=1.2, minNeighbors=3)
# Get the current time after performing face detection.
end = time()
# Loop through each face detected in the image and retireve the bounding box cordinates.
for (x1, y1, bbox_width, bbox_height) in results:
# Draw bounding box around the face on the copy of the input image using the retrieved coordinates.
cv2.rectangle(output_image, pt1=(x1, y1), pt2=(x1 + bbox_width, y1 + bbox_height), color=(0, 255, 0),
thickness=image_width//200)
# Check if the original input image and the output image are specified to be displayed.
if display:
# Write the time take by face detection process on the output image.
cv2.putText(output_image, text='Time taken: '+str(round(end - start, 2))+' Seconds.', org=(10, 65),
fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=image_width//700, color=(0,0,255),
thickness=image_width//500)
# Display the original input image and the output image.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title("Output");plt.axis('off');
# Otherwise
else:
# Return the output image and results of face detection.
return output_image, results
Now we will utilize the function haarCascadeDetectFaces() created above to perform face detection on a few sample images and display the results.
# Read a sample image and perform haar cascade face detection on it.
image = cv2.imread('media/sample4.jpg')
haarCascadeDetectFaces(image, cascade_face_detector, display=True)
The time taken by the algorithm to perform detection is pretty impressive, so yeah, it can work in real-time on a CPU.
# Read another sample image and perform haar cascade face detection on it.
image = cv2.imread('media/sample5.jpg')
haarCascadeDetectFaces(image, cascade_face_detector, display=True)
A major drawback of this algorithm is that it does not work on non-frontal and occluded faces.
# Read another sample image and perform haar cascade face detection on it.
image = cv2.imread('media/sample1.jpg')
haarCascadeDetectFaces(image, cascade_face_detector, display=True)
And also gives a lot of false positives But that can be controlled by increasing the value of the minNeighbors argument in the function cv2.CascadeClassifier.detectMultiScale().
Algorithm 2: Dlib HoG Face Detection
This face detector is based on HoG (Histogram of Oriented Gradients), and SVM (Support Vector Machine) and is significantly more accurate than the previous one. The technique used in this one is not invariant to changes in face angle, so it uses five different HOG filters that are for:
Frontal face
Right side turned face
Left side turned face
Frontal face but rotated right
Frontal face but rotated left
So it can work on slightly non-frontal and rotated faces as well.
Load the Dlib HoG Face Detector
Now we will use the dlib.get_frontal_face_detector() function to load the pre-trained HoG face detector and we will not need to pass the path of the model file for this one as the model is included in the dlib library.
# Get the HoG face detection model.
hog_face_detector = dlib.get_frontal_face_detector()
hog_face_detector
Create a HoG Face Detection Function
Now we will create a function hogDetectFaces() that will perform HoG face detection by inputting the image/frame into the loaded hog_face_detector and will visualize the resultant image along with the original image or return the resultant image along with the output of HoG face detector depending upon the passed arguments.
Function Syntax:
results = hog_face_detector(image, upsample)
Parameters:
image – It is the input image containing the faces in RGB format.
upsample (optional) – It is the number of times to upsample an image before performing face detection.
Returns:
results – It is an array of rectangle objects containing the (x, y) coordinates of the corners of the bounding boxes enclosing the faces in the input image.
Note:The model is trained to detect a minimum face size of 80×80, so to detect small faces in the images, you will have to upsample the images that increase the resolution of the input images, thus increases the face size at the cost of computation speed of the detection process.
def hogDetectFaces(image, hog_face_detector, display = True):
'''
This function performs face(s) detection on an image using dlib hog face detector.
Args:
image: The input image of the person(s) whose face needs to be detected.
hog_face_detector: The hog face detection model required to perform the detection on the input image.
display: A boolean value that is if set to true the function displays the original input image,
and the output image with the bounding boxes drawn and time taken written and returns nothing.
Returns:
output_image: A copy of input image with the bounding boxes drawn.
results: The output of the face detection process on the input image.
'''
# Get the height and width of the input image.
height, width, _ = image.shape
# Create a copy of the input image to draw bounding boxes on.
output_image = image.copy()
# Convert the image from BGR into RGB format.
imgRGB = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# Get the current time before performing face detection.
start = time()
# Perform the face detection on the image.
results = hog_face_detector(imgRGB, 0)
# Get the current time after performing face detection.
end = time()
# Loop through the bounding boxes of each face detected in the image.
for bbox in results:
# Retrieve the left most x-coordinate of the bounding box.
x1 = bbox.left()
# Retrieve the top most y-coordinate of the bounding box.
y1 = bbox.top()
# Retrieve the right most x-coordinate of the bounding box.
x2 = bbox.right()
# Retrieve the bottom most y-coordinate of the bounding box.
y2 = bbox.bottom()
# Draw a rectangle around a face on the copy of the image using the retrieved coordinates.
cv2.rectangle(output_image, pt1=(x1, y1), pt2=(x2, y2), color=(0, 255, 0), thickness=width//200)
# Check if the original input image and the output image are specified to be displayed.
if display:
# Write the time take by face detection process on the output image.
cv2.putText(output_image, text='Time taken: '+str(round(end - start, 2))+' Seconds.', org=(10, 65),
fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=width//700, color=(0,0,255), thickness=width//500)
# Display the original input image and the output image.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title("Output");plt.axis('off');
# Otherwise
else:
# Return the output image and results of face detection.
return output_image, results
Now we will utilize the function hogDetectFaces() created above to perform HoG face detection on a few sample images and display the results.
# Read a sample image and perform hog face detection on it.
image = cv2.imread('media/sample4.jpg')
hogDetectFaces(image, hog_face_detector, display=True)
So this too can work in real-time on a CPU. You can also resize the images before passing them to the model, as the smaller the images are, the faster the detection process will be. But this also increases the probability of faces smaller than 80×80 in the images.
# Read another sample image and perform hog face detection on it.
image = cv2.imread('media/sample3.jpg')
hogDetectFaces(image, hog_face_detector, display=True)
As you can see, it works on slightly rotated faces but will fail on extremely rotated and non-frontal ones and the bounding box often excludes some parts of the face like the chin and forehead.
# Read another sample image and perform hog face detection on it.
image = cv2.imread('media/sample7.jpg')
hogDetectFaces(image, hog_face_detector, display=True)
And also works on small occlusions but will fail on massive ones.
# Read another sample image and perform hog face detection on it.
image = cv2.imread('media/sample6.jpg')
hogDetectFaces(image, hog_face_detector, display=True)
As mentioned above, it cannot detect faces smaller than 80x80. Now, if you want, you can increase the upsample argument value of the loaded hog_face_detector in the function hogDetectFaces() created above, to detect the face in the image above, but that will also tremendously increase the time taken by the face detection process.
Algorithm 3: OpenCV Deep Learning based Face Detection
This one is based on a deep learning approach and uses ResNet-10 Architecture to detect multiple faces in a single pass (Single Shot Detector SSD) of the image through the network (model). It has been included in OpenCV since August 2017, with the official release of version 3.3, still, it is not as popular as the OpenCV Haar Cascade Face Detector but surely is highly more accurate.
Load the OpenCV Deep Learning based Face Detector
Now to load the face detector, OpenCV provides us with two options, one of them is in the Caffe framework’s format and takes around 5.10 MBs in memory and the other one is in the TensorFlow framework’s format and acquires only 2.7 MBs in memory.
To load the first one from the disk, we can use the cv2.dnn.readNetFromCaffe() function and to load the other one we will have to use the cv2.dnn.readNetFromTensorflow() function with appropriate arguments.
# Select the framework you want to use.
########################################################################################################################
# Load a model stored in Caffe framework's format using the architecture and the layers weights file stored in the disk.
opencv_dnn_model = cv2.dnn.readNetFromCaffe(prototxt="models/deploy.prototxt",
caffeModel="models/res10_300x300_ssd_iter_140000_fp16.caffemodel")
########################################################## OR ##########################################################
# Load a model stored in TensorFlow framework's format using the architecture and the layers weights file stored in the disk
# opencv_dnn_model = cv2.dnn.readNetFromTensorflow(model="models/opencv_face_detector_uint8.pb",
# config="models/opencv_face_detector.pbtxt")
########################################################################################################################
opencv_dnn_model
Create an OpenCV Deep Learning based Face Detection Function
Now we will create a function cvDnnDetectFaces() that will perform Deep Learning-based face detection using OpenCV. First, we will pre-process the image/frame using the cv2.dnn.blobFromImage() function and then we will set the pre-processed image as an input to the network by using the function opencv_dnn_model.setInput().
And after that, pass the input image into the network by using the opencv_dnn_model.forward() function to get an array containing the bounding boxes coordinates normalized to ([0.0, 1.0]) and the detection confidence of each faces in the image.
After performing the detection, the function will also visualize the resultant image along with the original image or return the resultant image along with the output of the dnn face detector depending upon the passed arguments.
Note:Higher the face detection confidence score is, the more certain the model is about the detection.
def cvDnnDetectFaces(image, opencv_dnn_model, min_confidence=0.5, display = True):
'''
This function performs face(s) detection on an image using opencv deep learning based face detector.
Args:
image: The input image of the person(s) whose face needs to be detected.
opencv_dnn_model: The pre-trained opencv deep learning based face detection model loaded from the disk
required to perform the detection.
min_confidence: The minimum detection confidence required to consider the face detection model's
prediction correct.
display: A boolean value that is if set to true the function displays the original input image,
and the output image with the bounding boxes drawn, confidence scores, and time taken
written and returns nothing.
Returns:
output_image: A copy of input image with the bounding boxes drawn and confidence scores written.
results: The output of the face detection process on the input image.
'''
# Get the height and width of the input image.
image_height, image_width, _ = image.shape
# Create a copy of the input image to draw bounding boxes and write confidence scores.
output_image = image.copy()
# Perform the required pre-processings on the image and create a 4D blob from image.
# Resize the image and apply mean subtraction to its channels
# Also convert from BGR to RGB format by swapping Blue and Red channels.
preprocessed_image = cv2.dnn.blobFromImage(image, scalefactor=1.0, size=(300, 300),
mean=(104.0, 117.0, 123.0), swapRB=False, crop=False)
# Set the input value for the model.
opencv_dnn_model.setInput(preprocessed_image)
# Get the current time before performing face detection.
start = time()
# Perform the face detection on the image.
results = opencv_dnn_model.forward()
# Get the current time after performing face detection.
end = time()
# Loop through each face detected in the image.
for face in results[0][0]:
# Retrieve the face detection confidence score.
face_confidence = face[2]
# Check if the face detection confidence score is greater than the thresold.
if face_confidence > min_confidence:
# Retrieve the bounding box of the face.
bbox = face[3:]
# Retrieve the bounding box coordinates of the face and scale them according to the original size of the image.
x1 = int(bbox[0] * image_width)
y1 = int(bbox[1] * image_height)
x2 = int(bbox[2] * image_width)
y2 = int(bbox[3] * image_height)
# Draw a bounding box around a face on the copy of the image using the retrieved coordinates.
cv2.rectangle(output_image, pt1=(x1, y1), pt2=(x2, y2), color=(0, 255, 0), thickness=image_width//200)
# Draw a filled rectangle near the bounding box of the face.
# We are doing it to change the background of the confidence score to make it easily visible.
cv2.rectangle(output_image, pt1=(x1, y1-image_width//20), pt2=(x1+image_width//16, y1),
color=(0, 255, 0), thickness=-1)
# Write the confidence score of the face near the bounding box and on the filled rectangle.
cv2.putText(output_image, text=str(round(face_confidence, 1)), org=(x1, y1-25),
fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=image_width//700,
color=(255,255,255), thickness=image_width//200)
# Check if the original input image and the output image are specified to be displayed.
if display:
# Write the time take by face detection process on the output image.
cv2.putText(output_image, text='Time taken: '+str(round(end - start, 2))+' Seconds.', org=(10, 65),
fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=image_width//700,
color=(0,0,255), thickness=image_width//500)
# Display the original input image and the output image.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title("Output");plt.axis('off');
# Otherwise
else:
# Return the output image and results of face detection.
return output_image, results
Now we will utilize the function cvDnnDetectFaces() created above to perform OpenCV deep learning-based face detection on a few sample images and display the results.
# Read a sample image and perform OpenCV dnn face detection on it.
image = cv2.imread('media/sample5.jpg')
cvDnnDetectFaces(image, opencv_dnn_model, display=True)
So it is highly more accurate than both of the above and works great even under massive occlusions and on non-frontal faces. And the reason for its significantly higher speed is that it can detect faces across various scales, allowing us to resize the images to a smaller size which decreases computations.
# Read another sample image and perform OpenCV dnn face detection on it.
image = cv2.imread('media/sample3.jpg')
cvDnnDetectFaces(image, opencv_dnn_model, display=True)
Also, the bounding box encloses the whole face, unlike the HoG Face Detector, making it easier to crop regions of interest (i.e., faces) from the images.CodeText
Also, the bounding box encloses the whole face, unlike the HoG Face Detector, making it easier to crop regions of interest (i.e., faces) from the images.CodeText
# Read another sample image and perform OpenCV dnn face detection on it.
image = cv2.imread('media/sample8.jpg')
cvDnnDetectFaces(image, opencv_dnn_model, display=True)
So even the faces with masks are detectable with this one.
Algorithm 4: Dlib Deep Learning based Face Detection
This detector is also based on a Deep learning (Convolution Neural Network) approach and uses Maximum-Margin Object Detection (MMOD) method to detect faces in images. This one is also trained for a minimum face size of 80×80 and provides the option of upsampling the images. This one is very slow on a CPU but can be used on an NVIDIA GPU and outperforms the other detectors in speed on the GPU.
Load the Dlib Deep Learning based Face Detector
Now first, we will use the dlib.cnn_face_detection_model_v1() function to load the pre-trained maximum-margin cnn face detector around 700 KBs from the disk, stored in a .dat file format.
# Load the dlib dnn face detection model from the file stored in the disk.
cnn_face_detector = dlib.cnn_face_detection_model_v1("models/mmod_human_face_detector.dat")
cnn_face_detector
Create a Dlib Deep Learning based Face Detection Function
Now we will create a function dlibDnnDetectFaces() in which we will perform deep Learning-based face detection using dlib by inputting the image/frame and the number of times to upsample the image to the loaded cnn_face_detector as we had done for the HoG face detection.
The only difference is that we are loading a different model, and it will return a list of objects, where each object will be a wrapper around a rectangle object (containing the bounding box coordinates) and a detection confidence score. As our every other function, this one will also visualize the results or return them depending upon the passed arguments.
def dlibDnnDetectFaces(image, cnn_face_detector, new_width = 600, display = True):
'''
This function performs face(s) detection on an image using dlib deep learning based face detector.
Args:
image: The input image of the person(s) whose face needs to be detected.
cnn_face_detector: The pre-trained dlib deep learning based (CNN) face detection model loaded from
the disk required to perform the detection.
new_width: The new width of the input image to which it will be resized before passing it to the model.
display: A boolean value that is if set to true the function displays the original input image,
and the output image with the bounding boxes drawn, confidence scores, and time taken
written and returns nothing.
Returns:
output_image: A copy of input image with the bounding boxes drawn and confidence scores written.
results: The output of the face detection process on the input image.
'''
# Get the height and width of the input image.
height, width, _ = image.shape
# Calculate the new height of the input image while keeping the aspect ratio constant.
new_height = int((new_width / width) * height)
# Resize a copy of input image while keeping the aspect ratio constant.
resized_image = cv2.resize(image.copy(), (new_width, new_height))
# Convert the resized image from BGR into RGB format.
imgRGB = cv2.cvtColor(resized_image, cv2.COLOR_BGR2RGB)
# Create a copy of the input image to draw bounding boxes and write confidence scores.
output_image = image.copy()
# Get the current time before performing face detection.
start = time()
# Perform the face detection on the image.
results = cnn_face_detector(imgRGB, 0)
# Get the current time after performing face detection.
end = time()
# Loop through each face detected in the image.
for face in results:
# Retriece the bounding box of the face.
bbox = face.rect
# Retrieve the bounding box coordinates and scale them according to the size of original input image.
x1 = int(bbox.left() * (width/new_width))
y1 = int(bbox.top() * (height/new_height))
x2 = int(bbox.right() * (width/new_width))
y2 = int(bbox.bottom() * (height/new_height))
# Draw bounding box around the face on the copy of the image using the retrieved coordinates.
cv2.rectangle(output_image, pt1=(x1, y1), pt2=(x2, y2), color=(0, 255, 0), thickness=width//200)
# Draw a filled rectangle near the bounding box of the face.
# We are doing it to change the background of the confidence score to make it easily visible.
cv2.rectangle(output_image, pt1=(x1, y1-width//20), pt2=(x1+width//16, y1), color=(0, 255, 0), thickness=-1)
# Write the confidence score of the face near the bounding box and on the filled rectangle.
cv2.putText(output_image, text=str(round(face.confidence, 1)), org=(x1, y1-25), fontFace=cv2.FONT_HERSHEY_COMPLEX,
fontScale=width//700, color=(255,255,255), thickness=width//200)
# Check if the original input image and the output image are specified to be displayed.
if display:
# Write the time take by face detection process on the output image.
cv2.putText(output_image, text='Time taken: '+str(round(end - start, 2))+' Seconds.', org=(10, 65),
fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=width//700, color=(0,0,255), thickness=width//500)
# Display the original input image and the output image.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title("Output");plt.axis('off');
# Otherwise
else:
# Return the output image and results of face detection.
return output_image, results
Now we will utilize the function dlibDnnDetectFaces() created above to perform dlib deep learning-based face detection on a few sample images and display the results.
# Read a sample image and perform dlib dnn face detection on it.
image = cv2.imread('media/sample8.jpg')
dlibDnnDetectFaces(image, cnn_face_detector, display=True)
Interesting! this one is also far more accurate and robust than the first two and is also capable of detecting faces under occlusion. But as you can see, the time taken by the detection process is very high, so this detector cannot work in real-time on a CPU.
# Read another sample image and perform dlib dnn face detection on it.
image = cv2.imread('media/sample9.jpg')
dlibDnnDetectFaces(image, cnn_face_detector, display=True)
Also, the varying face orientations and lighting do not stop it from detecting faces accurately.
# Read another sample image and perform dlib dnn face detection on it.
image = cv2.imread('media/sample3.jpg')
dlibDnnDetectFaces(image, cnn_face_detector, display=True)
Similar to the HoG face detector, the bounding box for this one is also small and does not enclose the whole face.
Algorithm 5: Mediapipe Deep Learning based Face Detection
The last one is also based on Deep learning approach and uses BlazeFace that is a very lightweight and highly accurate face detector inspired and modified from Single Shot MultiBox Detector (SSD) & MobileNetv2. The detector provided by Mediapipe is capable of running at a speed of 200-1000+ FPS on flagship devices.
Load the Mediapipe Face Detector
To load the model, we first have to initialize the face detection class using the mp.solutions.face_detection syntax and then we will have to call the function mp.solutions.face_detection.FaceDetection() with the arguments explained below:
model_selection – It is an integer index ( i.e., 0 or 1 ). When set to 0, a short-range model is selected that works best for faces within 2 meters from the camera, and when set to 1, a full-range model is selected that works best for faces within 5 meters. Its default value is 0.
min_detection_confidence – It is the minimum detection confidence between ([0.0, 1.0]) required to consider the face-detection model’s prediction successful. Its default value is 0.5 ( i.e., 50% ) which means that all the detections with prediction confidence less than 0.5 are ignored by default.
We will also have to initialize the mp.solutions.drawing_utils class which is used to visualize the detection results on the images/frames.
# Initialize the mediapipe drawing class.
mp_drawing = mp.solutions.drawing_utils
# Initialize the mediapipe face detection class.
mp_face_detection = mp.solutions.face_detection
# Set up the face detection function by selecting the full-range model.
mp_face_detector = mp_face_detection.FaceDetection(min_detection_confidence=0.4)
mp_face_detector
Create a Mediapipe Deep Learning based Face Detection Function
Now we will create a function mpDnnDetectFaces() in which we will use the mediapipe face detector to perform the detection on an image/frame by passing it into the loaded model by using the function mp_face_detector.process() and get a list of a bounding box and six key points for each face in the image. The six key points are on the:
Right Eye
Left Eye
Nose Tip
Mouth Center
Right Ear Tragion
Left Ear Tragion
The bounding boxes are composed of xmin and width (both normalized to [0.0, 1.0] by the image width) and ymin and height (both normalized to [0.0, 1.0] by the image height). Each key point is composed of x and y, which are normalized to [0.0, 1.0] by the image width and height respectively. The function will work on images and videos as well as this one will also display or return the results depending upon passed arguments.
def mpDnnDetectFaces(image, mp_face_detector, display = True):
'''
This function performs face(s) detection on an image using mediapipe deep learning based face detector.
Args:
image: The input image with person(s) whose face needs to be detected.
mp_face_detector: The mediapipe's face detection function required to perform the detection.
display: A boolean value that is if set to true the function displays the original input image,
and the output image with the bounding boxes, and key points drawn, and also confidence
scores, and time taken written and returns nothing.
Returns:
output_image: A copy of input image with the bounding box and key points drawn and also confidence scores written.
results: The output of the face detection process on the input image.
'''
# Get the height and width of the input image.
image_height, image_width, _ = image.shape
# Create a copy of the input image to draw bounding box and key points.
output_image = image.copy()
# Convert the image from BGR into RGB format.
imgRGB = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# Get the current time before performing face detection.
start = time()
# Perform the face detection on the image.
results = mp_face_detector.process(imgRGB)
# Get the current time after performing face detection.
end = time()
# Check if the face(s) in the image are found.
if results.detections:
# Iterate over the found faces.
for face_no, face in enumerate(results.detections):
# Draw the face bounding box and key points on the copy of the input image.
mp_drawing.draw_detection(image=output_image, detection=face,
keypoint_drawing_spec=mp_drawing.DrawingSpec(color=(0,255,0),
thickness=-1,
circle_radius=image_width//115),
bbox_drawing_spec=mp_drawing.DrawingSpec(color=(0,255,0),thickness=image_width//180))
# Retrieve the bounding box of the face.
face_bbox = face.location_data.relative_bounding_box
# Retrieve the required bounding box coordinates and scale them according to the size of original input image.
x1 = int(face_bbox.xmin*image_width)
y1 = int(face_bbox.ymin*image_height)
# Draw a filled rectangle near the bounding box of the face.
# We are doing it to change the background of the confidence score to make it easily visible
cv2.rectangle(output_image, pt1=(x1, y1-image_width//20), pt2=(x1+image_width//16, y1) ,
color=(0, 255, 0), thickness=-1)
# Write the confidence score of the face near the bounding box and on the filled rectangle.
cv2.putText(output_image, text=str(round(face.score[0], 1)), org=(x1, y1-25),
fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=image_width//700, color=(255,255,255),
thickness=image_width//200)
# Check if the original input image and the output image are specified to be displayed.
if display:
# Write the time take by face detection process on the output image.
cv2.putText(output_image, text='Time taken: '+str(round(end - start, 2))+' Seconds.', org=(10, 65),
fontFace=cv2.FONT_HERSHEY_COMPLEX, fontScale=image_width//700, color=(0,0,255),
thickness=image_width//500)
# Display the original input image and the output image.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title("Output");plt.axis('off');
# Otherwise
else:
# Return the output image and results of face detection.
return output_image, results
Now we will utilize the function mpDnnDetectFaces() created above to perform face detection using Mediapipe’s detector on a few sample images and display the results.
# Read a sample image and perform Mediapipe's face detection on it.
image = cv2.imread('media/sample11.jpg')
mpDnnDetectFaces(image, mp_face_detector, display=True)
You can get an idea of its super-realtime performance from the time taken by the detection process. After all, this is what differentiates this detector from all the others.
# Read another sample image and perform Mediapipe's face detection on it.
image = cv2.imread('media/sample7.jpg')
mpDnnDetectFaces(image, mp_face_detector, display=True)
It can detect the non-frontal and occluded faces but fails to accurately detect the key points in such scenerios.
# Read another sample image and perform Mediapipe's face detection on it.
image = cv2.imread('media/sample2.jpg')
mpDnnDetectFaces(image, mp_face_detector, display=True)
The size of the bounding box returned by this detector is also quite appropriate.
# Set up the face detection function by selecting the short-range model.
mp_face_detector = mp_face_detection.FaceDetection( min_detection_confidence=0.5)
# Read another sample image and perform Mediapipe's face detection on it.
image = cv2.imread('media/sample10.jpg')
mpDnnDetectFaces(image, mp_face_detector, display=True)
By using the short-range model, one can easily ignore the faces in the background, which is normally required in most of the applications out there, like face gesture recognition.
Face Detection on Real-Time Webcam Feed
We have compared the face detection algorithms on the images and discussed the pros and cons of each of them, but now the real test begins, as we will test the algorithms on a real-time webcam feed. First, we will select the algorithm we want to use as one of them will be used at a time. We have designed the code below to switch between different face detection algorithms in real-time, by pressing the key s.
We will utilize the functions created above to perform face detection on the real-time webcam feed using the selected algorithm and will also calculate and display the number of frames being updated in one second to get an idea of whether the algorithms can work in real-time on a CPU or not.
# Initialize the VideoCapture object to read from the webcam.
camera_video = cv2.VideoCapture(1)
camera_video.set(3,1280)
camera_video.set(4,960)
# Create named window for resizing purposes.
cv2.namedWindow('Face Detection', cv2.WINDOW_NORMAL)
# Initialize a list to store the face detection algorithms.
algoirthms = ['Mediapipe', 'OpenCV DNN', 'HOG', 'Haar Cascade', 'Dlib DNN']
# Initialize a variable to store the index of the selected face detection algorithm
algo_index = 0
# Initialize a variable to store the time of the previous frame.
time1 = 0
# Iterate until the webcam is accessed successfully.
while camera_video.isOpened():
# Read a frame.
ok, frame = camera_video.read()
# Check if frame is not read properly then continue to the next iteration to read the next frame.
if not ok:
continue
# Flip the frame horizontally for natural (selfie-view) visualization.
frame = cv2.flip(frame, 1)
# Get the height and width of the frame.
frame_height, frame_width, _ = frame.shape
# Retrive the currently selected face detection algorithm.
algoirthm = algoirthms[algo_index % len(algoirthms)]
# Check if the Haar Cascade algorithm is selected.
if algoirthm == 'Haar Cascade':
# Perform face detection using the Haar Cascade algorithm.
frame, _ = haarCascadeDetectFaces(frame, cascade_face_detector, display=False)
# Check if the HOG algorithm is selected.
elif algoirthm == 'HOG':
# Perform face detection using the HOG algorithm.
frame, _ = hogDetectFaces(frame, hog_face_detector, display=False)
# Check if the OpenCV DNN algorithm is selected.
elif algoirthm == 'OpenCV DNN':
# Perform face detection using the OpenCV DNN algorithm.
frame, _ = cvDnnDetectFaces(frame, opencv_dnn_model, display=False)
# Check if the 'Dlib DNN algorithm is selected.
elif algoirthm == 'Dlib DNN':
# Perform face detection using the Dlib DNN algorithm.
frame, _ = dlibDnnDetectFaces(frame, cnn_face_detector, display=False)
# Check if the Mediapipe algorithm is selected.
elif algoirthm == 'Mediapipe':
# Perform face detection using the Mediapipe algorithm.
frame, _ = mpDnnDetectFaces(frame, mp_face_detector, display=False)
# Write the currently selected method on the frame.
cv2.putText(frame, algoirthm, (frame_width//3, frame_height//8),cv2.FONT_HERSHEY_PLAIN, 4, (255, 155, 0), 3)
# Set the time for this frame to the current time.
time2 = time()
# Check if the difference between the previous and this frame time > 0 to avoid division by zero.
if (time2 - time1) > 0:
# Calculate the number of frames per second.
frames_per_second = 1.0 / (time2 - time1)
# Write the calculated number of frames per second on the frame.
cv2.putText(frame, 'FPS: {}'.format(int(frames_per_second)), (10, 30),cv2.FONT_HERSHEY_PLAIN, 2, (0, 255, 0), 3)
# Update the previous frame time to this frame time.
# As this frame will become previous frame in next iteration.
time1 = time2
# Display the frame.
cv2.imshow('Face Detection', frame)
# Wait for 1ms. If a a key is pressed, retreive the ASCII code of the key.
k = cv2.waitKey(1) & 0xFF
# Check if 'ESC' is pressed and break the loop.
if(k == 27):
break
# Check if 's' is pressed then increment the algorithm index.
elif (k == ord('s')):
algo_index += 1
# Release the VideoCapture Object and close the windows.
camera_video.release()
cv2.destroyAllWindows()
Output
As expected! all of them can work in real-time on a CPU except for the Dlib Deep Learning-based Face Detector.
Join My Course Computer Vision For Building Cutting Edge Applications Course
The only course out there that goes beyond basic AI Applications and teaches you how to create next-level apps that utilize physics, deep learning, classical image processing, hand and body gestures. Don’t miss your chance to level up and take your career to new heights
You’ll Learn about:
Creating GUI interfaces for python AI scripts.
Creating .exe DL applications
Using a Physics library in Python & integrating it with AI
Advance Image Processing Skills
Advance Gesture Recognition with Mediapipe
Task Automation with AI & CV
Training an SVM machine Learning Model.
Creating & Cleaning an ML dataset from scratch.
Training DL models & how to use CNN’s & LSTMS.
Creating 10 Advance AI/CV Applications
& More
Whether you’re a seasoned AI professional or someone just looking to start out in AI, this is the course that will teach you, how to Architect & Build complex, real world and thrilling AI applications
Bleed Face Detector – It is a python package that allows using 4 different face detectors (OpenCV Haar Cascade, Dlib HoG, OpenCV Deep Learning-based, and Dlib Deep Learning-based) by just changing a single line of code.
Summary:
In this tutorial, you have learned about the five most popular and effective face detectors along with the best tips, and suggestions. You have become capable of acquiring the required balance in accuracy, speed, and efficiency in any given scenario. Now to summarize;
If you have a low-end device or an embedded device like the Raspberry Pi and are expecting faces under substantial occlusion and with various sizes, orientations, and angles then I will recommend you to go for the Mediapipe Face Detector, as it is the fastest one and also pretty accurate. In fact, this one has the best trade-off between speed and accuracy and also gives a few facial landmarks (key points).
Otherwise, if you have some environmental restrictions and cannot use the Mediapipe face detector, then the next best option will be OpenCV DNN Face Detector as this one is also pretty accurate but has higher latency.
For applications in which the face size can be controlled (> 80×80), and you want to skip the people (small faces) that are far away from the camera, the Dlib HoG Face Detector can be used but surely is not the best option and for flag-ship devices with NVIDIA GPU in the same scenario, Dlib DNN Face Detector can be a good alternative to the HoG Face Detector, but try to use it on a CPU.
And If you are only working with frontal faces and want to skip all the non-frontal and rotated faces, then the Haar Cascade detector can be an option but remember you will have to manually tune the parameters to get rid of false positives.
So generally, you should just go with the Mediapipe Face Detector for super real-time speed and high accuracy.
You can reach out to me personally for a 1 on 1 consultation session in AI/computer vision regarding your project. Our talented team of vision engineers will help you every step of the way. Get on a call with me directlyhere.
Ready to seriously dive into State of the Art AI & Computer Vision? Then Sign up for these premium Courses by Bleed AI
Convolutional Neural Networks (CNN) are great for image data and Long-Short Term Memory (LSTM) networks are great when working with sequence data but when you combine both of them, you get the best of both worlds and you solve difficult computer vision problems like video classification.
In this tutorial, we’ll learn to implement human action recognition on videos using a Convolutional Neural Network combined with a Long-Short Term Memory Network. We’ll actually be using two different architectures and approaches in TensorFlow to do this. In the end, we’ll take the best-performing model and perform predictions with it on youtube videos.
Before I start with the code, let me cover some theories on video classification and different approaches that are available for it.
Image Classification
You may already be familiar with an image classification problem, where, you simply pass an image to the classifier (either a trained Deep Neural Network (CNN or an MLP) or a classical classifier) and get the class predictions out of it.
But what if you have a video? What will happen then?
Before we talk about how to go about dealing with videos, let’s just discuss what videos are exactly.
But First What Exactly Videos are?
Well, so it’s no secret that a video is just a sequence of multiple still images (aka. frames) that are updated really fast creating the appearance of a motion. Consider the video (converted into .gif format) below of a cat jumping on a bookshelf, it is just a combination of 15 different still images that are being updated one after the other.
Now that we understand what videos are, let’s take a look at a number of approaches that we can use to do video classification.
Approach 1: Single-Frame Classification
The simplest and most basic way of classifying actions in a video can be using an image classifier on each frame of the video and classify action in each frame independently. So if we implement this approach for a video of a person doing a backflip, we will get the following results.
The classifier predicts Falling in some frames instead of Backflipping because this approach ignores the temporal relation of the frames sequence. And even if a person looks at those frames independently he may think that the person is Falling.
Now a simple way to get a final prediction for the video is to consider the most frequent one which can work in simple scenarios but is Falling in our case and is not correct. So another way to go about this is to take an average of the probabilities of predictions and get a more robust final prediction.
You should also check another Video Classification and Human Activity Recognition tutorial I had published a while back, in which I had discussed a number of other approaches too and implemented this one using a single-frame CNN with moving averages and it had worked fine for a relatively simpler problem.
But as mentioned before, this approach is not effective, because it does not take into account the temporal aspect of the data.
Approach 2: Late Fusion
Another slightly different approach is late fusion, in which after performing predictions on each frame independently, the classification results are passed to a fusion layer that merges all the information and makes the prediction. This approach also leverages the temporal information of the data.
This approach does give decent results but is still not powerful enough. Now before moving to the next approach let’s discuss what Convolutional Neural Networks are. So that you get an idea of what that black box named image classifier was, that I was using in the images.
Convolutional Neural Network (CNN)
A Convolutional Neural Network (CNN or ConvNet) is a type of deep neural network that is specifically designed to work with image data and excels when it comes to analyzing the images and making predictions on them.
It works with kernels (called filters) that go over the image and generates feature maps (that represent whether a certain feature is present at a location in the image or not) and initially it generates few feature maps and as we go deeper in the network the number of feature maps is increased and the size of maps is decreased using pooling operations without losing critical information.
Each layer of a ConvNet learns features of increasing complexity which means, for example, the first layer may learn to detect edges and corners, while the last layer may learn to recognize humans in different postures.
Now let’s get back to discussing other approaches for video classification.
Approach 3: Early Fusion
Another approach of video classification is early fusion, in which all the information is merged at the beginning of the network, unlike late fusion which merges the information in the end. This is a powerful approach but still has its own limitations.
Approach 4: Using 3D CNN’s (aka. Slow Fusion)
Another option is to use a 3D Convolutional Network, where the temporal and spatial information are merged slowly throughout the whole network that is why it’s called Slow Fusion. But a disadvantage of this approach is that it is computationally really expensive so it is pretty slow.
Approach 5: Using Pose Detection and LSTM
Another method is to use a pose detection network on the video to get the landmark coordinates of the person for each frame in the video. And then feed the landmarks to an LSTM Network to predict the activity of the person.
There are already a lot of efficient pose detectors out there that can be used for this approach. But a disadvantage of using this approach is that you discard all the information other than the landmarks, like the environment information can be very useful, for example for playing football action category the stadium and uniform info can help the model a lot in predicting the action accurately.
Before going to the approach that we will implement in this tutorial, let’s briefly discuss what are Long Short Term Memory (LSTM) networks, as we will be using them in the approach.
Long Short Term Memory (LSTM)
An LSTM network is specifically designed to work with a data sequence as it takes into consideration all of the previous inputs while generating an output. LSTMs are actually a type of neural network called Recurrent Neural Network, but RNNs are not known to be effective for dealing with the Long term dependencies in the input sequence because of a problem called the Vanishing gradient problem.
LSTMs were developed to overcome the vanishing gradient and so an LSTM cell can remember context for long input sequences.
Many-to-one LSTM network
This makes an LSTM more capable of solving problems involving sequential data such as time series prediction, speech recognition, language translation, or music composition. But for now, we will only explore the role of LSTMs in developing better action recognition models.
Now let’s move on towards the approach we will implement in this tutorial to build an Action Recognizer. We will use a Convolution Neural Network (CNN) + Long Short Term Memory (LSTM) Network to perform Action Recognition while utilizing the Spatial-temporal aspect of the videos.
Approach 6: CNN + LSTM
We will be using a CNN to extract spatial features at a given time step in the input sequence (video) and then an LSTM to identify temporal relations between frames.
The two architectures that we will be using to use CNN along with LSTM are:
ConvLSTM
LRCN
Both of these approaches can be used using TensorFlow. This tutorial also has a video version as well, that you can go and watch for a more detailed overview of the code.
Alright, so without further ado, let’s get started.
Import the Libraries
We will start by installing and importing the required libraries.
# Install the required libraries.
!pip install pafy youtube-dl moviepy
# Import the required libraries.
import os
import cv2
import pafy
import math
import random
import numpy as np
import datetime as dt
import tensorflow as tf
from collections import deque
import matplotlib.pyplot as plt
from moviepy.editor import *
%matplotlib inline
from sklearn.model_selection import train_test_split
from tensorflow.keras.layers import *
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.utils import plot_model
And will set Numpy, Python, and Tensorflow seeds to get consistent results on every execution.
Step 1: Download and Visualize the Data with its Labels
In the first step, we will download and visualize the data along with labels to get an idea about what we will be dealing with. We will be using the UCF50 – Action Recognition Dataset, consisting of realistic videos taken from youtube which differentiates this data set from most of the other available action recognition data sets as they are not realistic and are staged by actors. The Dataset contains:
50 Action Categories
25 Groups of Videos per Action Category
133 Average Videos per Action Category
199 Average Number of Frames per Video
320 Average Frames Width per Video
240 Average Frames Height per Video
26 Average Frames Per Seconds per Video
Let’s download and extract the dataset.
# Discard the output of this cell.
%%capture
# Downlaod the UCF50 Dataset
!wget --no-check-certificate https://www.crcv.ucf.edu/data/UCF50.rar
#Extract the Dataset
!unrar x UCF50.rar
For visualization, we will pick 20 random categories from the dataset and a random video from each selected category and will visualize the first frame of the selected videos with their associated labels written. This way we’ll be able to visualize a subset ( 20 random videos ) of the dataset.CodeText
# Create a Matplotlib figure and specify the size of the figure.
plt.figure(figsize = (20, 20))
# Get the names of all classes/categories in UCF50.
all_classes_names = os.listdir('UCF50')
# Generate a list of 20 random values. The values will be between 0-50,
# where 50 is the total number of class in the dataset.
random_range = random.sample(range(len(all_classes_names)), 20)
# Iterating through all the generated random values.
for counter, random_index in enumerate(random_range, 1):
# Retrieve a Class Name using the Random Index.
selected_class_Name = all_classes_names[random_index]
# Retrieve the list of all the video files present in the randomly selected Class Directory.
video_files_names_list = os.listdir(f'UCF50/{selected_class_Name}')
# Randomly select a video file from the list retrieved from the randomly selected Class Directory.
selected_video_file_name = random.choice(video_files_names_list)
# Initialize a VideoCapture object to read from the video File.
video_reader = cv2.VideoCapture(f'UCF50/{selected_class_Name}/{selected_video_file_name}')
# Read the first frame of the video file.
_, bgr_frame = video_reader.read()
# Release the VideoCapture object.
video_reader.release()
# Convert the frame from BGR into RGB format.
rgb_frame = cv2.cvtColor(bgr_frame, cv2.COLOR_BGR2RGB)
# Write the class name on the video frame.
cv2.putText(rgb_frame, selected_class_Name, (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2)
# Display the frame.
plt.subplot(5, 4, counter);plt.imshow(rgb_frame);plt.axis('off')
Step 2: Preprocess the Dataset
Next, we will perform some preprocessing on the dataset. First, we will read the video files from the dataset and resize the frames of the videos to a fixed width and height, to reduce the computations and normalized the data to range [0-1] by dividing the pixel values with 255, which makes convergence faster while training the network.
But first, let’s initialize some constants.
# Specify the height and width to which each video frame will be resized in our dataset.
IMAGE_HEIGHT , IMAGE_WIDTH = 64, 64
# Specify the number of frames of a video that will be fed to the model as one sequence.
SEQUENCE_LENGTH = 20
# Specify the directory containing the UCF50 dataset.
DATASET_DIR = "UCF50"
# Specify the list containing the names of the classes used for training. Feel free to choose any set of classes.
CLASSES_LIST = ["WalkingWithDog", "TaiChi", "Swing", "HorseRace"]
Note:The IMAGE_HEIGHT, IMAGE_WIDTH and SEQUENCE_LENGTH constants can be increased for better results, although increasing the sequence length is only effective to a certain point, and increasing the values will result in the process being more computationally expensive.
Create a Function to Extract, Resize & Normalize Frames
We will create a function frames_extraction() that will create a list containing the resized and normalized frames of a video whose path is passed to it as an argument. The function will read the video file frame by frame, although not all frames are added to the list as we will only need an evenly distributed sequence length of frames.
def frames_extraction(video_path):
'''
This function will extract the required frames from a video after resizing and normalizing them.
Args:
video_path: The path of the video in the disk, whose frames are to be extracted.
Returns:
frames_list: A list containing the resized and normalized frames of the video.
'''
# Declare a list to store video frames.
frames_list = []
# Read the Video File using the VideoCapture object.
video_reader = cv2.VideoCapture(video_path)
# Get the total number of frames in the video.
video_frames_count = int(video_reader.get(cv2.CAP_PROP_FRAME_COUNT))
# Calculate the the interval after which frames will be added to the list.
skip_frames_window = max(int(video_frames_count/SEQUENCE_LENGTH), 1)
# Iterate through the Video Frames.
for frame_counter in range(SEQUENCE_LENGTH):
# Set the current frame position of the video.
video_reader.set(cv2.CAP_PROP_POS_FRAMES, frame_counter * skip_frames_window)
# Reading the frame from the video.
success, frame = video_reader.read()
# Check if Video frame is not successfully read then break the loop
if not success:
break
# Resize the Frame to fixed height and width.
resized_frame = cv2.resize(frame, (IMAGE_HEIGHT, IMAGE_WIDTH))
# Normalize the resized frame by dividing it with 255 so that each pixel value then lies between 0 and 1
normalized_frame = resized_frame / 255
# Append the normalized frame into the frames list
frames_list.append(normalized_frame)
# Release the VideoCapture object.
video_reader.release()
# Return the frames list.
return frames_list
Create a Function for Dataset Creation
Now we will create a function create_dataset() that will iterate through all the classes specified in the CLASSES_LIST constant and will call the function frame_extraction() on every video file of the selected classes and return the frames (features), class index ( labels), and video file path (video_files_paths).
def create_dataset():
'''
This function will extract the data of the selected classes and create the required dataset.
Returns:
features: A list containing the extracted frames of the videos.
labels: A list containing the indexes of the classes associated with the videos.
video_files_paths: A list containing the paths of the videos in the disk.
'''
# Declared Empty Lists to store the features, labels and video file path values.
features = []
labels = []
video_files_paths = []
# Iterating through all the classes mentioned in the classes list
for class_index, class_name in enumerate(CLASSES_LIST):
# Display the name of the class whose data is being extracted.
print(f'Extracting Data of Class: {class_name}')
# Get the list of video files present in the specific class name directory.
files_list = os.listdir(os.path.join(DATASET_DIR, class_name))
# Iterate through all the files present in the files list.
for file_name in files_list:
# Get the complete video path.
video_file_path = os.path.join(DATASET_DIR, class_name, file_name)
# Extract the frames of the video file.
frames = frames_extraction(video_file_path)
# Check if the extracted frames are equal to the SEQUENCE_LENGTH specified above.
# So ignore the vides having frames less than the SEQUENCE_LENGTH.
if len(frames) == SEQUENCE_LENGTH:
# Append the data to their repective lists.
features.append(frames)
labels.append(class_index)
video_files_paths.append(video_file_path)
# Converting the list to numpy arrays
features = np.asarray(features)
labels = np.array(labels)
# Return the frames, class index, and video file path.
return features, labels, video_files_paths
Now we will utilize the function create_dataset() created above to extract the data of the selected classes and create the required dataset.
# Create the dataset.
features, labels, video_files_paths = create_dataset()
Extracting Data of Class: WalkingWithDog
Extracting Data of Class: TaiChi
Extracting Data of Class: Swing
Extracting Data of Class: HorseRace
Now we will convert labels (class indexes) into one-hot encoded vectors.
# Using Keras's to_categorical method to convert labels into one-hot-encoded vectors
one_hot_encoded_labels = to_categorical(labels)
Step 3: Split the Data into Train and Test Set
As of now, we have the required features (a NumPy array containing all the extracted frames of the videos) and one_hot_encoded_labels (also a Numpy array containing all class labels in one hot encoded format). So now, we will split our data to create training and testing sets. We will also shuffle the dataset before the split to avoid any bias and get splits representing the overall distribution of the data.
# Split the Data into Train ( 75% ) and Test Set ( 25% ).
features_train, features_test, labels_train, labels_test = train_test_split(features, one_hot_encoded_labels, test_size = 0.25, shuffle = True, random_state = seed_constant)
Step 4: Implement the ConvLSTM Approach
In this step, we will implement the first approach by using a combination of ConvLSTM cells. A ConvLSTM cell is a variant of an LSTM network that contains convolutions operations in the network. it is an LSTM with convolution embedded in the architecture, which makes it capable of identifying spatial features of the data while keeping into account the temporal relation.
For video classification, this approach effectively captures the spatial relation in the individual frames and the temporal relation across the different frames. As a result of this convolution structure, the ConvLSTM is capable of taking in 3-dimensional input (width, height, num_of_channels) whereas a simple LSTM only takes in 1-dimensional input hence an LSTM is incompatible for modeling Spatio-temporal data on its own.
To construct the model, we will use Keras ConvLSTM2D recurrent layers. The ConvLSTM2D layer also takes in the number of filters and kernel size required for applying the convolutional operations. The output of the layers is flattened in the end and is fed to the Dense layer with softmax activation which outputs the probability of each action category.
We will also use MaxPooling3D layers to reduce the dimensions of the frames and avoid unnecessary computations and Dropout layers to prevent overfitting the model on the data. The architecture is a simple one and has a small number of trainable parameters. This is because we are only dealing with a small subset of the dataset which does not require a large-scale model.
def create_convlstm_model():
'''
This function will construct the required convlstm model.
Returns:
model: It is the required constructed convlstm model.
'''
# We will use a Sequential model for model construction
model = Sequential()
# Define the Model Architecture.
########################################################################################################################
model.add(ConvLSTM2D(filters = 4, kernel_size = (3, 3), activation = 'tanh',data_format = "channels_last",
recurrent_dropout=0.2, return_sequences=True, input_shape = (SEQUENCE_LENGTH,
IMAGE_HEIGHT, IMAGE_WIDTH, 3)))
model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))
model.add(TimeDistributed(Dropout(0.2)))
model.add(ConvLSTM2D(filters = 8, kernel_size = (3, 3), activation = 'tanh', data_format = "channels_last",
recurrent_dropout=0.2, return_sequences=True))
model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))
model.add(TimeDistributed(Dropout(0.2)))
model.add(ConvLSTM2D(filters = 14, kernel_size = (3, 3), activation = 'tanh', data_format = "channels_last",
recurrent_dropout=0.2, return_sequences=True))
model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))
model.add(TimeDistributed(Dropout(0.2)))
model.add(ConvLSTM2D(filters = 16, kernel_size = (3, 3), activation = 'tanh', data_format = "channels_last",
recurrent_dropout=0.2, return_sequences=True))
model.add(MaxPooling3D(pool_size=(1, 2, 2), padding='same', data_format='channels_last'))
#model.add(TimeDistributed(Dropout(0.2)))
model.add(Flatten())
model.add(Dense(len(CLASSES_LIST), activation = "softmax"))
########################################################################################################################
# Display the models summary.
model.summary()
# Return the constructed convlstm model.
return model
Now we will utilize the function create_convlstm_model() created above, to construct the required convlstm model.
# Construct the required convlstm model.
convlstm_model = create_convlstm_model()
# Display the success message.
print("Model Created Successfully!")
Check Model’s Structure:
Now we will use the plot_model() function, to check the structure of the constructed model, this is helpful while constructing a complex network and making that the network is created correctly.
# Plot the structure of the contructed model.
plot_model(convlstm_model, to_file = 'convlstm_model_structure_plot.png', show_shapes = True, show_layer_names = True)
Step 4.2: Compile & Train the Model
Next, we will add an early stopping callback to prevent overfitting and start the training after compiling the model.
# Create an Instance of Early Stopping Callback
early_stopping_callback = EarlyStopping(monitor = 'val_loss', patience = 10, mode = 'min', restore_best_weights = True)
# Compile the model and specify loss function, optimizer and metrics values to the model
convlstm_model.compile(loss = 'categorical_crossentropy', optimizer = 'Adam', metrics = ["accuracy"])
# Start training the model.
convlstm_model_training_history = convlstm_model.fit(x = features_train, y = labels_train, epochs = 50, batch_size = 4,shuffle = True, validation_split = 0.2, callbacks = [early_stopping_callback])
Evaluate the Trained Model
After training, we will evaluate the model on the test set.
# Evaluate the trained model.
model_evaluation_history = convlstm_model.evaluate(features_test, labels_test)
Now we will save the model to avoid training it from scratch every time we need the model.
# Get the loss and accuracy from model_evaluation_history.
model_evaluation_loss, model_evaluation_accuracy = model_evaluation_history
# Define the string date format.
# Get the current Date and Time in a DateTime Object.
# Convert the DateTime object to string according to the style mentioned in date_time_format string.
date_time_format = '%Y_%m_%d__%H_%M_%S'
current_date_time_dt = dt.datetime.now()
current_date_time_string = dt.datetime.strftime(current_date_time_dt, date_time_format)
# Define a useful name for our model to make it easy for us while navigating through multiple saved models.
model_file_name = f'convlstm_model___Date_Time_{current_date_time_string}___Loss_{model_evaluation_loss}___Accuracy_{model_evaluation_accuracy}.h5'
# Save your Model.
convlstm_model.save(model_file_name)
Step 4.3: Plot Model’s Loss & Accuracy Curves
Now we will create a function plot_metric() to visualize the training and validation metrics. We already have separate metrics from our training and validation steps so now we just have to visualize them.
def plot_metric(model_training_history, metric_name_1, metric_name_2, plot_name):
'''
This function will plot the metrics passed to it in a graph.
Args:
model_training_history: A history object containing a record of training and validation
loss values and metrics values at successive epochs
metric_name_1: The name of the first metric that needs to be plotted in the graph.
metric_name_2: The name of the second metric that needs to be plotted in the graph.
plot_name: The title of the graph.
'''
# Get metric values using metric names as identifiers.
metric_value_1 = model_training_history.history[metric_name_1]
metric_value_2 = model_training_history.history[metric_name_2]
# Construct a range object which will be used as x-axis (horizontal plane) of the graph.
epochs = range(len(metric_value_1))
# Plot the Graph.
plt.plot(epochs, metric_value_1, 'blue', label = metric_name_1)
plt.plot(epochs, metric_value_2, 'red', label = metric_name_2)
# Add title to the plot.
plt.title(str(plot_name))
# Add legend to the plot.
plt.legend()
Now we will utilize the function plot_metric() created above, to visualize and understand the metrics.
# Visualize the training and validation loss metrices.
plot_metric(convlstm_model_training_history, 'loss', 'val_loss', 'Total Loss vs Total Validation Loss')
# Visualize the training and validation accuracy metrices.
plot_metric(convlstm_model_training_history, 'accuracy', 'val_accuracy', 'Total Accuracy vs Total Validation Accuracy')
Step 5: Implement the LRCN Approach
In this step, we will implement the LRCN Approach by combining Convolution and LSTM layers in a single model. Another similar approach can be to use a CNN model and LSTM model trained separately. The CNN model can be used to extract spatial features from the frames in the video, and for this purpose, a pre-trained model can be used, that can be fine-tuned for the problem. And the LSTM model can then use the features extracted by CNN, to predict the action being performed in the video.
But here, we will implement another approach known as the Long-term Recurrent Convolutional Network (LRCN), which combines CNN and LSTM layers in a single model. The Convolutional layers are used for spatial feature extraction from the frames, and the extracted spatial features are fed to LSTM layer(s) at each time-steps for temporal sequence modeling. This way the network learns spatiotemporal features directly in an end-to-end training, resulting in a robust model.
You can read the paper Long-term Recurrent Convolutional Networks for Visual Recognition and Description by Jeff Donahue (CVPR 2015), to learn more about this architecture. We will also use TimeDistributed wrapper layer, which allows applying the same layer to every frame of the video independently. So it makes a layer (around which it is wrapped) capable of taking input of shape (no_of_frames, width, height, num_of_channels) if originally the layer’s input shape was (width, height, num_of_channels) which is very beneficial as it allows to input the whole video into the model in a single shot.
Step 5.1: Construct the Model
To implement our LRCN architecture, we will use time-distributed Conv2D layers which will be followed by MaxPooling2D and Dropout layers. The feature extracted from the Conv2D layers will be then flattened using the Flatten layer and will be fed to a LSTM layer. The Dense layer with softmax activation will then use the output from the LSTM layer to predict the action being performed.
def create_LRCN_model():
'''
This function will construct the required LRCN model.
Returns:
model: It is the required constructed LRCN model.
'''
# We will use a Sequential model for model construction.
model = Sequential()
# Define the Model Architecture.
########################################################################################################################
model.add(TimeDistributed(Conv2D(16, (3, 3), padding='same',activation = 'relu'),
input_shape = (SEQUENCE_LENGTH, IMAGE_HEIGHT, IMAGE_WIDTH, 3)))
model.add(TimeDistributed(MaxPooling2D((4, 4))))
model.add(TimeDistributed(Dropout(0.25)))
model.add(TimeDistributed(Conv2D(32, (3, 3), padding='same',activation = 'relu')))
model.add(TimeDistributed(MaxPooling2D((4, 4))))
model.add(TimeDistributed(Dropout(0.25)))
model.add(TimeDistributed(Conv2D(64, (3, 3), padding='same',activation = 'relu')))
model.add(TimeDistributed(MaxPooling2D((2, 2))))
model.add(TimeDistributed(Dropout(0.25)))
model.add(TimeDistributed(Conv2D(64, (3, 3), padding='same',activation = 'relu')))
model.add(TimeDistributed(MaxPooling2D((2, 2))))
#model.add(TimeDistributed(Dropout(0.25)))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(32))
model.add(Dense(len(CLASSES_LIST), activation = 'softmax'))
########################################################################################################################
# Display the models summary.
model.summary()
# Return the constructed LRCN model.
return model
Now we will utilize the function create_LRCN_model() created above to construct the required LRCN model.
# Construct the required LRCN model.
LRCN_model = create_LRCN_model()
# Display the success message.
print("Model Created Successfully!")
Check Model’s Structure:
Now we will use the plot_model() function to check the structure of the constructed LRCN model. As we had checked for the previous model.
# Plot the structure of the contructed LRCN model.
plot_model(LRCN_model, to_file = 'LRCN_model_structure_plot.png', show_shapes = True, show_layer_names = True)
Step 5.2: Compile & Train the Model
After checking the structure, we will compile and start training the model.
# Create an Instance of Early Stopping Callback.
early_stopping_callback = EarlyStopping(monitor = 'val_loss', patience = 15, mode = 'min', restore_best_weights = True)
# Compile the model and specify loss function, optimizer and metrics to the model.
LRCN_model.compile(loss = 'categorical_crossentropy', optimizer = 'Adam', metrics = ["accuracy"])
# Start training the model.
LRCN_model_training_history = LRCN_model.fit(x = features_train, y = labels_train, epochs = 70, batch_size = 4 , shuffle = True, validation_split = 0.2, callbacks = [early_stopping_callback])
Evaluating the trained Model
As done for the previous one, we will evaluate the LRCN model on the test set.
# Evaluate the trained model.
model_evaluation_history = LRCN_model.evaluate(features_test, labels_test)
After that, we will save the model for future uses using the same technique we had used for the previous model.
# Get the loss and accuracy from model_evaluation_history.
model_evaluation_loss, model_evaluation_accuracy = model_evaluation_history
# Define the string date format.
# Get the current Date and Time in a DateTime Object.
# Convert the DateTime object to string according to the style mentioned in date_time_format string.
date_time_format = '%Y_%m_%d__%H_%M_%S'
current_date_time_dt = dt.datetime.now()
current_date_time_string = dt.datetime.strftime(current_date_time_dt, date_time_format)
# Define a useful name for our model to make it easy for us while navigating through multiple saved models.
model_file_name = f'LRCN_model___Date_Time_{current_date_time_string}___Loss_{model_evaluation_loss}___Accuracy_{model_evaluation_accuracy}.h5'
# Save the Model.
LRCN_model.save(model_file_name)
Step 5.3: Plot Model’s Loss & Accuracy Curves
Now we will utilize the function plot_metric() we had created above to visualize the training and validation metrics of this model.
# Visualize the training and validation loss metrices.
plot_metric(LRCN_model_training_history, 'loss', 'val_loss', 'Total Loss vs Total Validation Loss')
# Visualize the training and validation accuracy metrices.
plot_metric(LRCN_model_training_history, 'accuracy', 'val_accuracy', 'Total Accuracy vs Total Validation Accuracy')
Step 6: Test the Best Performing Model on YouTube videos
From the results, it seems that the LRCN model performed significantly well for a small number of classes. so in this step, we will put the LRCN model to test on some youtube videos.
Create a Function to Download YouTube Videos:
We will create a function download_youtube_videos() to download the YouTube videos first using pafy library. The library only requires a URL to a video to download it along with its associated metadata like the title of the video.
def download_youtube_videos(youtube_video_url, output_directory):
'''
This function downloads the youtube video whose URL is passed to it as an argument.
Args:
youtube_video_url: URL of the video that is required to be downloaded.
output_directory: The directory path to which the video needs to be stored after downloading.
Returns:
title: The title of the downloaded youtube video.
'''
# Create a video object which contains useful information about the video.
video = pafy.new(youtube_video_url)
# Retrieve the title of the video.
title = video.title
# Get the best available quality object for the video.
video_best = video.getbest()
# Construct the output file path.
output_file_path = f'{output_directory}/{title}.mp4'
# Download the youtube video at the best available quality and store it to the contructed path.
video_best.download(filepath = output_file_path, quiet = True)
# Return the video title.
return title
Download a Test Video:
Now we will utilize the function download_youtube_videos() created above to download a youtube video on which the LRCN model will be tested.
# Make the Output directory if it does not exist
test_videos_directory = 'test_videos'
os.makedirs(test_videos_directory, exist_ok = True)
# Download a YouTube Video.
video_title = download_youtube_videos('https://www.youtube.com/watch?v=8u0qjmHIOcE', test_videos_directory)
# Get the YouTube Video's path we just downloaded.
input_video_file_path = f'{test_videos_directory}/{video_title}.mp4'
Create a Function To Perform Action Recognition on Videos
Next, we will create a function predict_on_video() that will simply read a video frame by frame from the path passed in as an argument and will perform action recognition on video and save the results.
def predict_on_video(video_file_path, output_file_path, SEQUENCE_LENGTH):
'''
This function will perform action recognition on a video using the LRCN model.
Args:
video_file_path: The path of the video stored in the disk on which the action recognition is to be performed.
output_file_path: The path where the ouput video with the predicted action being performed overlayed will be stored.
SEQUENCE_LENGTH: The fixed number of frames of a video that can be passed to the model as one sequence.
'''
# Initialize the VideoCapture object to read from the video file.
video_reader = cv2.VideoCapture(video_file_path)
# Get the width and height of the video.
original_video_width = int(video_reader.get(cv2.CAP_PROP_FRAME_WIDTH))
original_video_height = int(video_reader.get(cv2.CAP_PROP_FRAME_HEIGHT))
# Initialize the VideoWriter Object to store the output video in the disk.
video_writer = cv2.VideoWriter(output_file_path, cv2.VideoWriter_fourcc('M', 'P', '4', 'V'),
video_reader.get(cv2.CAP_PROP_FPS), (original_video_width, original_video_height))
# Declare a queue to store video frames.
frames_queue = deque(maxlen = SEQUENCE_LENGTH)
# Initialize a variable to store the predicted action being performed in the video.
predicted_class_name = ''
# Iterate until the video is accessed successfully.
while video_reader.isOpened():
# Read the frame.
ok, frame = video_reader.read()
# Check if frame is not read properly then break the loop.
if not ok:
break
# Resize the Frame to fixed Dimensions.
resized_frame = cv2.resize(frame, (IMAGE_HEIGHT, IMAGE_WIDTH))
# Normalize the resized frame by dividing it with 255 so that each pixel value then lies between 0 and 1.
normalized_frame = resized_frame / 255
# Appending the pre-processed frame into the frames list.
frames_queue.append(normalized_frame)
# Check if the number of frames in the queue are equal to the fixed sequence length.
if len(frames_queue) == SEQUENCE_LENGTH:
# Pass the normalized frames to the model and get the predicted probabilities.
predicted_labels_probabilities = LRCN_model.predict(np.expand_dims(frames_queue, axis = 0))[0]
# Get the index of class with highest probability.
predicted_label = np.argmax(predicted_labels_probabilities)
# Get the class name using the retrieved index.
predicted_class_name = CLASSES_LIST[predicted_label]
# Write predicted class name on top of the frame.
cv2.putText(frame, predicted_class_name, (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
# Write The frame into the disk using the VideoWriter Object.
video_writer.write(frame)
# Release the VideoCapture and VideoWriter objects.
video_reader.release()
video_writer.release()
Perform Action Recognition on the Test Video
Now we will utilize the function predict_on_video() created above to perform action recognition on the test video we had downloaded using the function download_youtube_videos() and display the output video with the predicted action overlayed on it.
# Construct the output video path.
output_video_file_path = f'{test_videos_directory}/{video_title}-Output-SeqLen{SEQUENCE_LENGTH}.mp4'
# Perform Action Recognition on the Test Video.
predict_on_video(input_video_file_path, output_video_file_path, SEQUENCE_LENGTH)
# Display the output video.
VideoFileClip(output_video_file_path, audio=False, target_resolution=(300,None)).ipython_display()
Create a Function To Perform a Single Prediction on Videos
Now let’s create a function that will perform a single prediction for the complete videos. We will extract evenly distributed N(SEQUENCE_LENGTH) frames from the entire video and pass them to the LRCN model. This approach is really useful when you are working with videos containing only one activity as it saves unnecessary computations and time in that scenario.
def predict_single_action(video_file_path, SEQUENCE_LENGTH):
'''
This function will perform single action recognition prediction on a video using the LRCN model.
Args:
video_file_path: The path of the video stored in the disk on which the action recognition is to be performed.
SEQUENCE_LENGTH: The fixed number of frames of a video that can be passed to the model as one sequence.
'''
# Initialize the VideoCapture object to read from the video file.
video_reader = cv2.VideoCapture(video_file_path)
# Get the width and height of the video.
original_video_width = int(video_reader.get(cv2.CAP_PROP_FRAME_WIDTH))
original_video_height = int(video_reader.get(cv2.CAP_PROP_FRAME_HEIGHT))
# Declare a list to store video frames we will extract.
frames_list = []
# Initialize a variable to store the predicted action being performed in the video.
predicted_class_name = ''
# Get the number of frames in the video.
video_frames_count = int(video_reader.get(cv2.CAP_PROP_FRAME_COUNT))
# Calculate the interval after which frames will be added to the list.
skip_frames_window = max(int(video_frames_count/SEQUENCE_LENGTH),1)
# Iterating the number of times equal to the fixed length of sequence.
for frame_counter in range(SEQUENCE_LENGTH):
# Set the current frame position of the video.
video_reader.set(cv2.CAP_PROP_POS_FRAMES, frame_counter * skip_frames_window)
# Read a frame.
success, frame = video_reader.read()
# Check if frame is not read properly then break the loop.
if not success:
break
# Resize the Frame to fixed Dimensions.
resized_frame = cv2.resize(frame, (IMAGE_HEIGHT, IMAGE_WIDTH))
# Normalize the resized frame by dividing it with 255 so that each pixel value then lies between 0 and 1.
normalized_frame = resized_frame / 255
# Appending the pre-processed frame into the frames list
frames_list.append(normalized_frame)
# Passing the pre-processed frames to the model and get the predicted probabilities.
predicted_labels_probabilities = LRCN_model.predict(np.expand_dims(frames_list, axis = 0))[0]
# Get the index of class with highest probability.
predicted_label = np.argmax(predicted_labels_probabilities)
# Get the class name using the retrieved index.
predicted_class_name = CLASSES_LIST[predicted_label]
# Display the predicted action along with the prediction confidence.
print(f'Action Predicted: {predicted_class_name}\nConfidence: {predicted_labels_probabilities[predicted_label]}')
# Release the VideoCapture object.
video_reader.release()
Perform Single Prediction on a Test Video
Now we will utilize the function predict_single_action() created above to perform a single prediction on a complete youtube test video that we will download using the function download_youtube_videos(), we had created above.
# Download the youtube video.
video_title = download_youtube_videos('https://youtu.be/fc3w827kwyA', test_videos_directory)
# Construct tihe nput youtube video path
input_video_file_path = f'{test_videos_directory}/{video_title}.mp4'
# Perform Single Prediction on the Test Video.
predict_single_action(input_video_file_path, SEQUENCE_LENGTH)
# Display the input video.
VideoFileClip(input_video_file_path, audio=False, target_resolution=(300,None)).ipython_display()
Action Predicted: TaiChi
Confidence: 0.94
Join My Course Computer Vision For Building Cutting Edge Applications Course
The only course out there that goes beyond basic AI Applications and teaches you how to create next-level apps that utilize physics, deep learning, classical image processing, hand and body gestures. Don’t miss your chance to level up and take your career to new heights
You’ll Learn about:
Creating GUI interfaces for python AI scripts.
Creating .exe DL applications
Using a Physics library in Python & integrating it with AI
Advance Image Processing Skills
Advance Gesture Recognition with Mediapipe
Task Automation with AI & CV
Training an SVM machine Learning Model.
Creating & Cleaning an ML dataset from scratch.
Training DL models & how to use CNN’s & LSTMS.
Creating 10 Advance AI/CV Applications
& More
Whether you’re a seasoned AI professional or someone just looking to start out in AI, this is the course that will teach you, how to Architect & Build complex, real world and thrilling AI applications
In this tutorial, we discussed a number of approaches to perform video classification and learned about the importance of the temporal aspect of data to gain higher accuracy in video classification and implemented two CNN + LSTM architectures in TensorFlow to perform Human Action Recognition on videos by utilizing the temporal as well as spatial information of the data.
We also learned to perform pre-processing on videos using the OpenCV library to create an image dataset, we also looked into getting youtube videos using just their URLs with the help of the Pafy library for testing our model.
Now let’s discuss a limitation in our application that you should know about, our action recognizer cannot work on multiple people performing different activities. There should be only one person in the frame to correctly recognize the activity of that person by our action recognizer, this is because the data was in this manner, on which we had trained our model.
You can use some different dataset that has been annotated for more than one person’s activity and also provides the bounding box coordinates of the person along with the activity he is performing, to overcome this limitation.
Or a hacky way is to crop out each person and perform activity recognition separately on each person but this will be computationally very expensive.
That is all for this lesson, if you enjoyed this tutorial, let me know in the comments, you can also reach out to me personally for a 1 on 1 Coaching/consultation session in AI/computer vision regarding your project or your career.
Hire Us
Let our team of expert engineers and managers build your next big project using Bleeding Edge AI Tools & Technologies
In this tutorial, we will learn how to manipulate facial expressions and create a DeepFake video out of a static image using the famous First-Order Motion Model. Yes, you heard that right, we just need a single 2D image of a person to create the DeepFake video.
Excited yet? … not that much ? .. well what if I tell you, the whole tutorial is actually on Google Colab, so you don’t need to worry about installation or GPUs to run, everything is configured.
And you know what the best part is?
Utilizing the colab that you will get in this tutorial, you can generate deepfakes in a matter of seconds, yes seconds, not weeks, not days, not hours but seconds.
What is a DeepFake?
The term DeepFake is a combination of two words; Deep refers to the technology responsible for generating DeepFake content, known as Deep learning, and Fake refers to the falsified content. The technology generates synthetic media, to create falsified content, which can be done by either replacing or synthesizing the new content (can be a video or even audio).
Below you can see the results on a few sample images:
This feels like putting your own words in a person’s mouth but on a whole new level.
Also, you may have noticed, in the results above, that we are generating the output video utilizing the whole frame/image, not just on the face ROI that people normally do.
First-Order Motion Model
We will be using the aforementioned First-Order Motion Model, so let’s start by understanding what it is and how it works?
The term First-Order Motion refers to a change in luminance over space and time, and the first-order motion model utilizes this change to capture motion in the source video (also known as the driving video).
The framework is composed of two main components: motion estimation (which predicts a dense motion field) and image generation (which predicts the resultant video). You don’t have to worry about the technical details of these modules to use this model. If you are not a computer vision practitioner, you should skip the paragraph below.
The Motion Extractor module uses the unsupervised key point detector to get the relevant key points from the source image and a driving video frame. The local affine transformation is calculated concerning the frame from the driving video. A Dense Motion Network then generates an occlusion map and a dense optical flow, which is fed into the Generator Module alongside the source image. The Generator Module generates the output frame, which is a replica of the relevant motion from the driving video’s frame onto the source image.
This approach can also be used to manipulate faces, human bodies, and even animated characters, given that the model is trained on a set of videos of similar object categories.
Now that we have gone through the prerequisite theory and implementation details of the approach we will be using, let’s dive into the code.
# Discard the output of this cell.
%%capture
# Clone the First Order Motion Model Github Repository.
!git clone https://github.com/AliaksandrSiarohin/first-order-model
# Change Current Working Directory to "first-order-model".
%cd first-order-model
# Clone the Face Alignment Repository.
!git clone https://github.com/1adrianb/face-alignment
# Change Current Working Directory to "face-alignment".
%cd face-alignment
Step 1.2: Install the required Modules
Install helper modules that are required to perform the necessary pre- and post-processing.
# Discard the output of this cell.
%%capture
# Install the modules required to use the Face Alignment module.
!pip install -r requirements.txt
# Install the Face Alignment module.
!python setup.py install
# Install the mediapipe library.
!pip install mediapipe
# Move one Directory back, i.e., to first-order-model Directory.
%cd ..
Import the required libraries.
import os
import cv2
import mediapipe as mp
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import demo
import imageio
import warnings
warnings.filterwarnings("ignore")
import requests
from skimage.transform import resize
from skimage import img_as_ubyte
from google.colab import files
from IPython.display import display, Javascript,HTML
from google.colab.output import eval_js
from base64 import b64encode, b64decode
Step 2: Prepare a driving video
In this step, we will create a driving video and will make it ready to be passed into the model.
Step 2.1: Record a video from the webcam
Create a function record_video() that can access the webcam utilizing JavaScript.
Remember that Colab is a web IDE that runs entirely on the cloud, so that’s why JavaScript is needed to access the system Webcam.
def record_video(filename = 'Video.mp4'):
'''
This function will record a video, by accessing the Webcam using the javascript and store it into a Video file.
Args:
filename: It is the name by which recorded video will be saved. Its default value is 'Video.mp4'.
'''
# Java Script Code for accessing the Webcam and Recording the Video.
js=Javascript("""
async function recordVideo() {
// Create a div. It is a division or a section in an HTML document.
// This div will contain the buttons and the video.
const div = document.createElement('div');
// Create a start recording button.
const capture = document.createElement('button');
// Create a stop recording button.
const stopCapture = document.createElement("button");
// Set the text content, background color and foreground color of the button.
capture.textContent = "Start Recording";
capture.style.background = "orange";
capture.style.color = "white";
// Set the text content, background color and foreground color of the button.
stopCapture.textContent = "Recording";
stopCapture.style.background = "red";
stopCapture.style.color = "white";
// Append the start recording button into the div.
div.appendChild(capture);
// Create a video element.
const video = document.createElement('video');
video.style.display = 'block';
// Prompt the user for permission to use a media input.
const stream = await navigator.mediaDevices.getUserMedia({audio:true, video: true});
// Create a MediaRecorder Object.
let recorder = new MediaRecorder(stream, { mimeType: "video/webm" });
// Append the div into the document.
document.body.appendChild(div);
// Append the video into the div.
div.appendChild(video);
// Set the video source.
video.srcObject = stream;
// Mute the video.
video.muted = true;
// Play the video.
await video.play();
// Set height of the output.
google.colab.output.setIframeHeight(document.documentElement.scrollHeight, true);
// Wait until the video recording button is pressed.
await new Promise((resolve) => {
capture.onclick = resolve;
});
// Start recording the video.
recorder.start();
// Replace the start recording button with the stop recording button.
capture.replaceWith(stopCapture);
// Stop recording automatically after 11 seconds.
setTimeout(()=>{recorder.stop();}, 11000);
// Get the recording.
let recData = await new Promise((resolve) => recorder.ondataavailable = resolve);
let arrBuff = await recData.data.arrayBuffer();
// Stop the stream.
stream.getVideoTracks()[0].stop();
// Remove the div.
div.remove();
// Convert the recording into a binaryString.
let binaryString = "";
let bytes = new Uint8Array(arrBuff);
bytes.forEach((byte) => {
binaryString += String.fromCharCode(byte);
})
// Return the results.
return btoa(binaryString);
}
""")
# Create a try block.
try:
# Execute the javascript code and display the webcam results.
display(js)
data=eval_js('recordVideo({})')
# Decode the recorded data.
binary=b64decode(data)
# Write the video file on the disk.
with open(filename,"wb") as video_file:
video_file.write(binary)
# Display the success message.
print(f"Saved recorded video at: {filename}")
# Handle the exceptions.
except Exception as err:
print(str(err))
Now utilize the record_video() function created above, to record a video. Click the recording button, and then the browser will ask for user permission to access the webcam and microphone (if you have not allowed these by default) after allowing, the video will start recording and will be saved into the disk after a few seconds. Please make sure to have neutral facial expressions at the start of the video to get the best Deep Fake results.
You can also use a pre-recorded video if you want, by skipping this step and saving that pre-recorded video at the video_path.
# Specify the width at which the video will be displayed.
video_width = 300
# Specify the path of the video.
video_path = 'Video.mp4'
# Record the video.
record_video(video_path)
# Read the Video file.
video_file = open(video_path, "r+b").read()
# Display the Recorded Video, using HTML.
video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
HTML(f"""<video width={video_width} controls><source src="{video_url}"></video>""")
The video is saved, but the issue is that the video is just a set of frames with no FPS and Duration information, and this can cause issues later on, so now, before proceeding further, resolve the issue by utilizing the FFMPEG command.
# Discard the output of this cell.
%%capture
# Check if the source video already exists.
if os.path.exists('source_video.mp4'):
# Remove the video.
os.remove('source_video.mp4')
# Set the FPS=23 of the Video.mp4 and save it with the name source_video.mp4.
!ffmpeg -i Video.mp4 -filter:v fps=23 source_video.mp4
Step 2.2: Crop the face from the recorded video
Crop the face from the video by utilizing the crop-video.py script provided in the First-Order-Model repository.
The Script will generate a FFMPEG Command that we can use to align and crop the face region of interest after resizing it to 256x256. Note that it does not print any FFMPEG Command if it fails to detect the face in the video.
# Generate the `FFMPEG` to crop the face from the video.
!python crop-video.py --inp source_video.mp4
Utilize the FFMPEG command generated by the crop-video.py script to create the desired video.
# Discard the output of this cell.
%%capture
# Check if the face video already exists.
if os.path.exists('crop.mp4'):
# Remove the video.
os.remove('crop.mp4')
# Crop the face from the video and resize it to 256x256.
!ffmpeg -i source_video.mp4 -ss 0.0 -t 6.913043478260869 -filter:v "crop=866:866:595:166, scale=256:256" crop.mp4
Now that the cropped face video is stored in the disk, display it to make sure that we have extracted exactly what we desired.
# Read the Cropped Video file.
video_file = open('crop.mp4', "r+b").read()
# Display the Cropped Video, using HTML.
video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
HTML(f"""<video width={video_width} controls><source src="{video_url}"></video>""")
Perfect! The driving video looks good. Now we can start working on a source image.
Step 3: Prepare a source Image
In this step, we will make the source Image ready to be passed into the model.
Download the Image
Download the image that we want to pass to the First-Order Motion Model utilizing the wget command.
# Discard the output of this cell.
%%capture
# Specify the path of the images directory.
IMAGES_DIR = 'media'
# Check if the images directory does not already exist.
if not os.path.exists(os.getcwd()+"/"+IMAGES_DIR):
# Download the images directory.
!wget -O {IMAGES_DIR + '.zip'} 'https://drive.google.com/uc?export=download&id=18t14YLm0nDc7USp550pIjslcZ3g5ZJ0t'
# Extract the compressed directory.
!unzip {os.getcwd() + "/" + IMAGES_DIR + '.zip'}
Load the Image
Read the image using the function cv2.imread() and display it utilizing the matplotlib library.
Note: In case you want to use a different source image, make sure to use an image of a person with neutral expressions to get the best results.
%matplotlib inline
# Specify the source image name.
image_name = 'imran.jpeg'
# Read the source image.
source_image = cv2.imread(os.path.join(os.getcwd(), IMAGES_DIR , image_name))
# Resize the image to make its width 720, while keeping its aspect ratio constant.
source_image = cv2.resize(source_image, dsize=(720, int((720/source_image.shape[1])*source_image.shape[0])))
# Display the image.
plt.imshow(source_image[:,:,::-1]);plt.title("Source Image");plt.axis("off");plt.show()
Step 3.1: Detect the face
Similar to the driving video, we can’t pass the whole source image into the First-Order Motion Model, we have to crop the face from the image and then pass the face image into the model. For this we will need a Face Detector to get the Face Bounding Box coordinates and we will utilize the Mediapipe’s Face Detection Solution.
Initialize the Mediapipe Face Detection Model
To use the Mediapipe’s Face Detection solution, initialize the face detection class using the syntax mp.solutions.face_detection, and then call the function mp.solutions.face_detection.FaceDetection() with the arguments explained below:
model_selection – It is an integer index ( i.e., 0 or 1 ). When set to 0, a short-range model is selected that works best for faces within 2 meters from the camera, and when set to 1, a full-range model is selected that works best for faces within 5 meters. Its default value is 0.
min_detection_confidence – It is the minimum detection confidence between ([0.0, 1.0]) required to consider the face-detection model’s prediction successful. Its default value is 0.5 ( i.e., 50% ) which means that all the detections with prediction confidence less than 0.5 are ignored by default.
# Initialize the mediapipe face detection class.
mp_face_detection = mp.solutions.face_detection
# Setup the face detection function.
face_detection = mp_face_detection.FaceDetection(model_selection=0, min_detection_confidence=0.5)
Create a function to detect face
Create a function detect_face() that will utilize the Mediapipe’s Face Detection Solution to detect a face in an image and will return the bounding box coordinates of the detected face.
To perform the face detection, pass the image (in RGB format) into the loaded face detection model by using the function mp.solutions.face_detection.FaceDetection().process(). The output object returned will have an attribute detections that contains a list of a bounding box and six key points for each face in the image.
Note that the bounding boxes are composed of xmin and width (both normalized to [0.0, 1.0] by the image width) and ymin and height (both normalized to [0.0, 1.0] by the image height). Ignore the face key points for now as we are only interested in the bounding box coordinates.
After performing the detection, convert the bounding box coordinates back to their original scale utilizing the image width and height. Also draw the bounding box on a copy of the source image using the function cv2.rectangle().
def detect_face(image, face_detection, draw=False, display=True):
'''
This function performs face detection, converts the bounding box coordinates back to their original scale,
and returns the coordinates.
Args:
image: The input image of the person's face whose face needs to be detected.
face_detection: The Mediapipe's face detection function required to perform the face detection.
draw: A boolean value that is if set to true the function draws the face bounding box on the output image.
display: A boolean value that is if set to true the function displays the output image with
the face bounding box drawn and returns nothing.
Returns:
face_bbox: A tuple (xmin, ymin, box_width, box_height) containing the face bounding box coordinates.
'''
# Get the height and width of the input image.
image_height, image_width, _ = image.shape
# Create a copy of the input image to draw a face bounding box.
output_image = image.copy()
# Convert the image from BGR into RGB format.
imgRGB = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# Perform the face detection on the image.
face_detection_results = face_detection.process(imgRGB)
# Initialize a tuple to store the face bounding box coordinates.
face_bbox = ()
# Check if the face(s) in the image are found.
if face_detection_results.detections:
# Iterate over the found faces.
for face_no, face in enumerate(face_detection_results.detections):
# Get the bounding box coordinates and convert them back to their original scale.
xmin = int(face.location_data.relative_bounding_box.xmin * image_width)
ymin = int(face.location_data.relative_bounding_box.ymin * image_height)
box_width = int(face.location_data.relative_bounding_box.width * image_width)
box_height = int(face.location_data.relative_bounding_box.height * image_height)
# Update the bounding box tuple values.
face_bbox = (xmin, ymin, box_width, box_height)
# Check if the face bounding box is specified to be drawn.
if draw:
# Draw the face bounding box on the output image.
cv2.rectangle(output_image, (xmin, ymin), (xmin+box_width, ymin+box_height), (0, 0, 255), 2)
# Check if the output image is specified to be displayed.
if display:
# Display the output image.
plt.figure(figsize=[15,15])
plt.imshow(output_image[:,:,::-1]);plt.title("Output Image");plt.axis('off');
# Otherwise.
else:
# Return the face bounding box coordinates.
return face_bbox
Utilize the detect_face() function created above to detect the face in the source image and display the results.
# Perform face detection on the image.
detect_face(source_image, face_detection, draw=True, display=True)
Nice! face detection is working perfectly.
Step 3.2: Align and crop the face
Another very important preprocessing step is the Face Alignment on the source image. Make sure that the face is properly aligned in the source image otherwise the model can generate weird/funny output results.
To align the face in the source image, first detect the 468 facial landmarks using Mediapipe’s Face Mesh Solution, then extract the eyes center and nose tip landmarks to calculate the face orientation and then finally rotate the image accordingly to align the face.
Initialize the Face Landmarks Detection Model
To use the Mediapipe’s Face Mesh solution, initialize the face mesh class using the syntax mp.solutions.face_mesh and call the function mp.solutions.face_mesh.FaceMesh() with the arguments explained below:
static_image_mode – It is a boolean value that is if set to False, the solution treats the input images as a video stream. It will try to detect faces in the first input images, and upon a successful detection further localizes the face landmarks. In subsequent images, once all max_num_faces faces are detected and the corresponding face landmarks are localized, it simply tracks those landmarks without invoking another detection until it loses track of any of the faces. This reduces latency and is ideal for processing video frames. If set to True, face detection runs on every input image, ideal for processing a batch of static, possibly unrelated, images. Its default value is False.
max_num_faces – It is the maximum number of faces to detect. Its default value is 1.
refine_landmarks – It is a boolean value that is if set to True, the solution further refines the landmark coordinates around the eyes and lips, and outputs additional landmarks around the irises by applying the Attention Mesh Model. Its default value is False.
min_detection_confidence – It is the minimum detection confidence ([0.0, 1.0]) required to consider the face-detection model’s prediction correct. Its default value is 0.5 which means that all the detections with prediction confidence less than 50% are ignored by default.
min_tracking_confidence – It is the minimum tracking confidence ([0.0, 1.0]) from the landmark-tracking model for the face landmarks to be considered tracked successfully, or otherwise face detection will be invoked automatically on the next input image, so increasing its value increases the robustness, but also increases the latency. It is ignored if static_image_mode is True, where face detection simply runs on every image. Its default value is 0.5.
We will be working with images only, so we will have to set the static_image_mode to True. We will also define the eyes and nose landmarks indexes that are required to extract the eyes and nose landmarks.
# Initialize the mediapipe face mesh class.
mp_face_mesh = mp.solutions.face_mesh
# Set up the face landmarks function for images.
face_mesh = mp_face_mesh.FaceMesh(static_image_mode=True, max_num_faces=2,
refine_landmarks=True, min_detection_confidence=0.5)
# Specify the nose and eyes indexes.
NOSE = 2
LEFT_EYE = [362, 263] # [right_landmark left_landmark]
RIGHT_EYE = [33, 133] # [right_landmark left_landmark]
Create a function to extract eyes and nose landmarks
Create a function extract_landmarks() that will utilize the Mediapipe’s Face Mesh Solution to detect the 468 Facial Landmarks and then extract the left and right eyes corner landmarks and the nose tip landmark.
To perform the Face(s) landmarks detection, pass the image to the face’s landmarks detection machine learning pipeline by using the function mp.solutions.face_mesh.FaceMesh().process(). But first, convert the image from BGR to RGB format using the function cv2.cvtColor() as OpenCV reads images in BGR format and the ml pipeline expects the input images to be in RGB color format.
The machine learning pipeline outputs an object that has an attribute multi_face_landmarks that contains the 468 3D facial landmarks for each detected face in the image. Each landmark has:
x – It is the landmark x-coordinate normalized to [0.0, 1.0] by the image width.
y – It is the landmark y-coordinate normalized to [0.0, 1.0] by the image height.
z – It is the landmark z-coordinate normalized to roughly the same scale as x. It represents the landmark depth with the center of the head being the origin, and the smaller the value is, the closer the landmark is to the camera.
After performing face landmarks detection on the image, convert the landmarks’ x and y coordinates back to their original scale utilizing the image width and height and then extract the required landmarks utilizing the indexes we had specified earlier. Also draw the extracted landmarks on a copy of the source image using the function cv2.circle(), just for visualization purposes.
def extract_landmarks(image, face_mesh, draw=False, display=True):
'''
This function performs face landmarks detection, converts the landmarks x and y coordinates back to their original scale,
and extracts left and right eyes corner landmarks and the nose tip landmark.
Args:
image: The input image of the person's face whose facial landmarks needs to be extracted.
face_mesh: The Mediapipe's face landmarks detection function required to perform the landmarks detection.
draw: A boolean value that is if set to true the function draws the extracted landmarks on the output image.
display: A boolean value that is if set to true the function displays the output image with
the extracted landmarks drawn and returns nothing.
Returns:
extracted_landmarks: A list containing the left and right eyes corner landmarks and the nose tip landmark.
'''
# Get the height and width of the input image.
height, width, _ = image.shape
# Initialize an array to store the face landmarks.
face_landmarks = np.array([])
# Create a copy of the input image to draw facial landmarks.
output_image = image.copy()
# Convert the image from BGR into RGB format.
imgRGB = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# Perform the facial landmarks detection on the image.
results = face_mesh.process(imgRGB)
# Check if facial landmarks are found.
if results.multi_face_landmarks:
# Iterate over the found faces.
for face in results.multi_face_landmarks:
# Convert the Face landmarks x and y coordinates into their original scale,
# And store them into a numpy array.
# For simplicity, we are only storing face landmarks of a single face,
# you can extend it to work with multiple faces if you want.
face_landmarks = np.array([(landmark.x*width, landmark.y*height)
for landmark in face.landmark], dtype=np.int32)
# Extract the right eye landmarks.
right_eye_landmarks = [face_landmarks[RIGHT_EYE[0]], face_landmarks[RIGHT_EYE[1]]]
# Extract the left eye landmarks.
left_eye_landmarks = [face_landmarks[LEFT_EYE[0]], face_landmarks[LEFT_EYE[1]]]
# Extract the nose tip landmark.
nose_landmarks = face_landmarks[NOSE]
# Initialize a list to store the extracted landmarks
extracted_landmarks = [nose_landmarks, left_eye_landmarks, right_eye_landmarks]
# Check if extracted landmarks are specified to be drawn.
if draw:
# Draw the left eye extracted landmarks.
cv2.circle(output_image, tuple(left_eye_landmarks[0]), 3, (0, 0, 255), -1)
cv2.circle(output_image, tuple(left_eye_landmarks[1]), 3, (255, 0, 0), -1)
# Draw the right eye extracted landmarks.
cv2.circle(output_image, tuple(right_eye_landmarks[0]), 3, (0, 0, 255), -1)
cv2.circle(output_image, tuple(right_eye_landmarks[1]), 3, (255, 0, 0), -1)
# Draw the nose landmark.
cv2.circle(output_image, tuple(nose_landmarks), 3, (255, 0, 0), -1)
# Check if the output image is specified to be displayed.
if display:
# Display the output image.
plt.figure(figsize=[15,15])
plt.imshow(output_image[:,:,::-1]);plt.title("Output Image");plt.axis('off');
# Otherwise.
else:
# Return the extracted landamarks.
return extracted_landmarks
Now we will utilize the extract_landmarks() function created above to detect and extract the eyes and nose landmarks and visualize the results.
# Extract the left and right eyes corner landmarks and the nose tip landmark.
extract_landmarks(source_image, face_mesh, draw=True, display=True)
Cool! it is accurately extracting the required landmarks.
Create a function to calculate eyes center
Create a function calculate_eyes_center() that will find the left and right eyes center landmarks by utilizing the eyes corner landmarks that we had extracted in the extract_landmarks() function created above.
def calculate_eyes_center(image, extracted_landmarks, draw=False, display=False):
'''
This function calculates the center landmarks of the left and right eye.
Args:
image: The input image of the person's face whose eyes center landmarks needs to be calculated.
extracted_landmarks: A list containing the left and right eyes corner landmarks and the nose tip landmark.
draw: A boolean value that is if set to true the function draws the eyes center and nose tip
landmarks on the output image.
display: A boolean value that is if set to true the function displays the output image with the
landmarks drawn and returns nothing.
Returns:
landmarks: A list containing the left and right eyes center landmarks and the nose tip landmark.
'''
# Create a copy of the input image to draw landmarks.
output_image = image.copy()
# Get the nose tip landmark.
nose_landmark = extracted_landmarks[0]
# Calculate the center landmarks of the left and right eye.
left_eye_center = np.mean(extracted_landmarks[1], axis=0, dtype=np.int32)
right_eye_center = np.mean(extracted_landmarks[2], axis=0, dtype=np.int32)
# Initialize a list to store the left and right eyes center landmarks and the nose tip landmark.
landmarks = [nose_landmark, left_eye_center, right_eye_center]
# Check if the landmarks are specified to be drawn.
if draw:
# Draw the center landmarks of the left and right eye.
cv2.circle(output_image, tuple(left_eye_center), 3, (0, 0, 255), -1)
cv2.circle(output_image, tuple(right_eye_center), 3, (0, 0, 255), -1)
# Draw the nose tip landmark.
cv2.circle(output_image, tuple(nose_landmark), 3, (0, 0, 255), -1)
# Check if the output image is specified to be displayed.
if display:
# Display the output image.
plt.figure(figsize=[15,15])
plt.imshow(output_image[:,:,::-1]);plt.title("Output Image");plt.axis('off');
# Otherwise.
else:
# Return the left and right eyes center landmarks and the nose tip landmark.
return landmarks
Use the extracted_landmarks() and the calculate_eyes_center() function to calculate the central landmarks of the left and right eyes on the source image.
# Get the left and right eyes center landmarks and the nose tip landmark.
extracted_landmarks = extract_landmarks(source_image, face_mesh, draw=False, display=False)
calculate_eyes_center(source_image, extracted_landmarks, draw=True, display=True)
Working perfectly fine!
Create a function to rotate images
Create a function rotate_image() that will simply rotate an image in a counter-clockwise direction with a specific angle without losing any portion of the image.
def rotate_image(image, angle, display=True):
'''
This function rotates an image in counter-clockwise direction with a specific angle.
Args:
image: The input image that needs to be rotated.
angle: It is the angle (in degrees) with which the image needs to be rotated. -ve values can rotate clockwise.
display: A boolean value that is if set to true the function displays the original input image,
and the output rotated image and returns nothing.
Returns:
rotated_image: The image rotated in counter-clockwise direction with the specified angle.
'''
# Get the height and width of the input image.
image_height, image_width, _ = image.shape
# Get the center coordinate x and y values of the image.
(center_x, center_y) = (image_width / 2, image_height / 2)
# Get the rotation matrix to rotate the image with the specified angle at the same scale.
rotation_matrix = cv2.getRotationMatrix2D(center=(center_x, center_y), angle=angle, scale=1.0)
# Compute the new height and width of the image.
new_height = int((image_height * np.abs(rotation_matrix[0, 0])) +
(image_width * np.abs(rotation_matrix[0, 1])))
new_width = int((image_height * np.abs(rotation_matrix[0, 1])) +
(image_width * np.abs(rotation_matrix[0, 0])))
# Adjust the rotation matrix accordingly to the new height and width.
rotation_matrix[0, 2] += (new_width / 2) - center_x
rotation_matrix[1, 2] += (new_height / 2) - center_y
# Perform the actual rotation on the image.
rotated_image = cv2.warpAffine(image.copy(), rotation_matrix, (new_width, new_height))
# Check if the original input image and the output image are specified to be displayed.
if display:
# Display the original input image and the output image.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
plt.subplot(122);plt.imshow(rotated_image[:,:,::-1]);plt.title(f"Rotated Image angle:{angle}");plt.axis('off');
# Otherwise.
else:
# Return the rotated image.
return rotated_image
Utilize the rotate_image() function to rotate the source image at an angle of 45 degrees.
# Rotate the source image with an angle of 45 degrees.
rotated_img = rotate_image(source_image, 45, display= True)
Rotation looks good, but rotating the image with a random angle will not bring us any good.
Create a function to find the face orientation
Create a function calculate_face_angle() that will find the face orientation, and then we will rotate the image accordingly utilizing the function rotate_image() created above, to appropriately align the face in the source image.
To find the face angle, first get the eyes and nose landmarks using the extract_landmarks() function then we will pass these landmarks to the calculate_eyes_center() function to get the eyes center landmarks, then utilizing the eyes center landmarks we will calculate the midpoint of the eyes i.e., the center of the forehead. And we will use the detect_face() function created in the previous step, to get the face bounding box coordinates and then utilize those coordinates to find the center_pred point i.e., the mid-point of the bounding box top-right and top_left coordinate.
And then finally, find the distance between the nose, center_of_forehead and center_pred landmarks as shown in the gif above to calculate the face angle utilizing the famous cosine-law.
def calculate_face_angle(image, face_mesh, face_detection):
'''
This function calculates the face orientation in an image.
Args:
image: The input image of the person whose face angle needs to be calculated.
face_mesh: The Mediapipe's face landmarks detection function required to perform the landmarks detection.
face_detection: The Mediapipe's face detection function required to perform the face detection.
Returns:
angle: The calculated face angle in degrees.
'''
# Create a helper function to find distance between two points.
def calculate_distance(point1, point2):
'''
This function calculates euclidean distance between two points.
Args:
point1: A tuple containing the x and y coordinates of the first point.
point2: A tuple containing the x and y coordinates of the second point.
Returns:
distance: The distance calculated between the two points.
'''
# Calculate euclidean distance between the two points.
distance = np.sqrt((point1[0] - point2[0]) ** 2 + (point1[1] - point2[1]) ** 2)
# Return the calculated distance.
return distance
# Extract the left and right eyes corner landmarks and the nose tip landmark.
nose_and_eyes_landmarks = extract_landmarks(image, face_mesh, draw=False, display=False)
# Get the center of each eye, from Eyes Landmarks.
nose, left_eye_center, right_eye_center = calculate_eyes_center(image, nose_and_eyes_landmarks, draw=False, display=False)
# Calculate the midpoint of the eye center landmarks i.e., the center of the forehead.
center_of_forehead = ((left_eye_center[0] + right_eye_center[0]) // 2,
(left_eye_center[1] + right_eye_center[1]) // 2,)
# Get the face bounding box coordinates.
xmin, ymin, box_width, box_height = detect_face(image, face_detection, display=False)
# Get the mid-point of the bounding box top-right and top_left coordinate.
center_pred = int(xmin + (box_width//2)), ymin
# Find the distance between forehead and nose.
length_line1 = calculate_distance(center_of_forehead, nose)
# Find the distance between center_pred and nose.
length_line2 = calculate_distance(center_pred, nose)
# Find the distance between center_pred and center_of_forehead.
length_line3 = calculate_distance(center_pred, center_of_forehead)
# Use the cosine law to find the cos A.
cos_a = -(length_line3 ** 2 - length_line2 ** 2 - length_line1 ** 2) / (2 * length_line2 * length_line1)
# Get the inverse of the cosine function.
angle = np.arccos(cos_a)
# Set the nose tip landmark as the origin.
origin_x, origin_y = nose
# Get the center of forehead x and y coordinates.
point_x, point_y = center_of_forehead
# Rotate the x and y coordinates w.r.t the origin with the found angle.
rotated_x = int(origin_x + np.cos(angle) * (point_x - origin_x) - np.sin(angle) * (point_y - origin_y))
rotated_y = int(origin_y + np.sin(angle) * (point_x - origin_x) + np.cos(angle) * (point_y - origin_y))
# Initialize a tuple to store the rotated points.
rotated_point = rotated_x, rotated_y
# Do some mathematics to find a few numbers that will help us determine whether the angle has to be positive or negative.
c1 = ((center_of_forehead[0] - nose[0]) * (rotated_point[1] - nose[1]) - (center_of_forehead[1] - nose[1]) *
(rotated_point[0] - nose[0]))
c2 = ((center_pred[0] - center_of_forehead[0]) * (rotated_point[1] - center_of_forehead[1]) -
(center_pred[1] - center_of_forehead[1]) * (rotated_point[0] - center_of_forehead[0]))
c3 = ((nose[0] - center_pred[0]) * (rotated_point[1] - center_pred[1]) -
(nose[1] - center_pred[1]) * (rotated_point[0] - center_pred[0]))
# Check if the angle needs to be negative.
if (c1 < 0 and c2 < 0 and c3 < 0) or (c1 > 0 and c2 > 0 and c3 > 0):
# Make the angle -ve, and convert it into degrees.
angle = np.degrees(-angle)
# Otherwise.
else:
# Convert the angle into degrees.
angle = np.degrees(angle)
# Return the angle.
return angle
Utilize the calculate_face_angle() function created above the find the face angle of the source image and display it.
# Calculate the face angle.
face_angle = calculate_face_angle(source_image, face_mesh, face_detection)
print(f'Face Angle: {face_angle}')
Face Angle: -8.50144759667417
Now that we have the face angle, we can move on to aligning the face in the source image.
Create a Function to Align the Face and Crop the Face Region
Create a function align_crop_face() that will first utilize the function calculate_face_angle() to get the face angle, then rotate the image accordingly utilizing the rotate_image() function and finally crop the face from the image utilizing the face bounding box coordinates (after scaling) returned by the detect_face() function. In the end, it will also resize the face image to the size 256x256 that is required by the First-Order Motion Model.
def align_crop_face(image, face_mesh, face_detection, face_scale_factor=1, display=True):
'''
This function aligns and crop the face and then resizes it into 256x256 dimensions.
Args:
image: The input image of the person whose face needs to be aligned and cropped.
face_mesh: The Mediapipe's face landmarks detection function required to perform the landmarks detection.
face_detection: The Mediapipe's face detection function required to perform the face detection.
face_scale_factor: The factor to scale up or down the face bouding box coordinates.
display: A boolean value that is if set to true the function displays the original input
image, rotated image and the face roi image.
Returns:
face_roi: A copy of the aligned face roi of the input image.
face_angle: The calculated face angle in degrees.
face_bbox: A tuple (xmin, ymin, xmax, ymax) containing the face bounding box coordinates.
'''
# Get the height and width of the input image.
image_height, image_width, _ = image.shape
# Get the angle of the face in the input image.
face_angle = calculate_face_angle(image, face_mesh, face_detection)
# Rotate the input image with the face angle.
rotated_image = rotate_image(source_image, face_angle, display=False)
# Perform face detection on the image.
face_bbox = detect_face(rotated_image, face_detection, display=False)
# Check if the face was detected in the image.
if len(face_bbox) > 0:
# Get the face bounding box coordinates.
xmin, ymin, box_width, box_height = face_bbox
# Calculate the bottom right coordinate values of the face bounding box.
xmax = xmin + box_width
ymax = ymin + box_height
# Get the face scale value according to the bounding box height.
face_scale = int((box_height * face_scale_factor))
# Add padding to the face bounding box.
xmin = xmin - face_scale//2 if xmin - face_scale//2 > 0 else 0
ymin = ymin - int(face_scale*1.8) if ymin - int(face_scale*1.8) > 0 else 0
xmax = xmax + face_scale//2 if xmax + face_scale//2 < image_width else image_width
ymax = ymax + int(face_scale/1.8) if ymax + int(face_scale/1.8) < image_height else image_height
# Update the face bounding box tuple.
face_bbox = (xmin, ymin, xmax, ymax)
# Crop the face from the image.
face_roi = rotated_image[ymin: ymax, xmin : xmax]
# Resize the face region to 256x256 dimensions.
face_roi = cv2.resize(face_roi, (256, 256), interpolation=cv2.INTER_AREA)
# Save the image on the disk.
cv2.imwrite('source_image.jpg', face_roi)
# Check if the original input image, rotated image and the face roi image are specified to be displayed.
if display:
# Display the original input image, rotated image and the face roi image.
plt.figure(figsize=[15,15])
plt.subplot(131);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
plt.subplot(132);plt.imshow(rotated_image[:,:,::-1]);plt.title(f"Rotated Image angle: {round(face_angle, 2)}");plt.axis('off');
plt.subplot(133);plt.imshow(face_roi[:,:,::-1]);plt.title(f"Face ROI");plt.axis('off');
# Return the face roi, the face angle and the face bounding box.
return face_roi, face_angle, face_bbox
Use the function align_crop_face() on the source image and visualize the results.
Make sure that the whole face is present in the cropped face ROI results. Increase/decrease the face_scale_factor value if you are testing this colab on a different source image. Increase the value if the face is being cropped in the source image and decrease the value if the face ROI image contains too much background.
# Perform face alignment and crop the face.
face_roi, face_angle, face_bbox = align_crop_face(source_image, face_mesh, face_detection,
face_scale_factor=0.3, display=True)
I must say its looking good! all the preprocessing steps went as we intended. But now comes a post-processing step, after generating the output from the First-Order Motion Model.
Remember that later on, we will have to embed the manipulated face back into the source image, so a function to restore the source image’s original state after embedding the output is also required.
Create a function to restore the original source image
So now we will create a function restore_source_image() that will undo the rotation we had applied on the image and will remove the black borders which appeared after the rotation.
def restore_source_image(rotated_image, rotation_angle, image_size, display=True):
'''
This function undoes the rotation and removes the black borders of an image.
Args:
rotated_image: The rotated image which needs to be restored.
rotation_angle: The angle with which the image was rotated.
image_size: A tuple containing the original height and width of the image.
display: A boolean value that is if set to true the function displays the original
input image, and the output image and returns nothing.
Returns:
output_image: The rotated image after being restored to its original state.
'''
# Get the height and width of the image.
height, width = image_size
# Undo the rotation of the image by rotating again with a -ve angle.
output_image = rotate_image(rotated_image, -rotation_angle, display=False)
# Find the center of the image.
center_x = output_image.shape[1] // 2
center_y = output_image.shape[0] // 2
# Crop the undo_rotation image, and remove the black borders.
output_image = output_image[center_y - height//2 : center_y + height//2,
center_x - width//2 : center_x + width//2]
# Check if the original input image and the output image are specified to be displayed.
if display:
# Display the original input image and the output image.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(rotated_image[:,:,::-1]);plt.title("Rotated Image");plt.axis('off');
plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title(f"Restored Image");plt.axis('off');
# Otherwise.
else:
# Return the output image.
return output_image
Utilize the calculate_face_angle() and rotate_image() function to create a rotated image and then check if the restore_source_image() can restore the images original state by undoing the rotation and removing the black borders from image.
# Calculate the face angle and rotate the image with the face angle.
face_angle = calculate_face_angle(source_image, face_mesh, face_detection)
rotated_image = rotate_image(source_image, face_angle, display=False)
# Restore the rotated image.
restore_source_image(rotated_image, face_angle, image_size=source_image.shape[:2], display=True)
Step 4: Create the DeepFake
Now that the source image and driving video is ready, so now in this step, we will create a DeepFake video.
Step 4.1: Download the First-Order Motion Model
Now we will download the required pre-trained network from the Yandex Disk Models. We have multiple options there, but since we are only interested in face manipulation, we will only download the vox-adv-cpk.pth.tar file.
# Specify the name of the file.
filename ='vox-adv-cpk.pth.tar'
# Download the pre-trained network.
download = requests.get(requests.get('https://cloud-api.yandex.net/v1/disk/public/resources/download?public_key=https://yadi.sk/d/lEw8uRm140L_eQ&path=/' + filename).json().get('href'))
# Open the file and write the downloaded content.
with open(filename, 'wb') as checkpoint:
checkpoint.write(download.content)
Create a function to display the results
Create a function display_results() that will concatenate the source image, driving video, and the generated video together and will show the results.
def display_results(source_image, driving_video, generated_video=None):
'''
This function stacks and displays the source image, driving video, and generated video together.
Args:
source_image: The source image ((contains facial appearance info)) that is used to create the deepfake video.
driving_video: The driving video (contains facial motion info) that is used to create the deepfake video.
generated_video: The deepfake video generated by combining the source image and the driving video.
Returns:
resultant_video: A stacked video containing the source image, driving video, and the generated video.
'''
# Create a figure.
fig = plt.figure(figsize=(8 + 4 * (generated_video is not None), 6))
# Create a list to store the frames of the resultant_video.
frames = []
# Iterate the number of times equal to the number of frames in the driving video.
for i in range(len(driving_video)):
# Create a list to store the stack elements.
stack = [source_image]
# Append the driving video into the stack.
stack.append(driving_video[i])
# Check if a valid generated video is passed.
if generated_video is not None:
# Append the generated video into the stack.
stack.append(generated_video[i])
# Concatenate all the elements in the stack.
stacked_image = plt.imshow(np.concatenate(stack, axis=1), animated=True)
# Turn off the axis.
plt.axis('off')
# Append the image into the list.
frames.append([stacked_image])
# Create the stacked video.
resultant_video = animation.ArtistAnimation(fig, frames, interval=50, repeat_delay=1000)
# Close the figure window.
plt.close()
# Return the results.
return resultant_video
Step 4.2: Load source image and driving video (Face cropped)
Load the pre-processed source image and the driving video and then display them utilizing the display_results() function created above.
# Ignore the warnings.
warnings.filterwarnings("ignore")
# Load the Source Image and the driving video.
source_image = imageio.imread('source_image.jpg')
driving_video = imageio.mimread('crop.mp4')
# Resize the Source Image and the driving video to 256x256.
source_image = resize(source_image, (256, 256))[..., :3]
driving_video = [resize(frame, (256, 256))[..., :3] for frame in driving_video]
# Display the Source Image and the driving video.
HTML(display_results(source_image, driving_video).to_html5_video())
Step 4.3: Generate the video
Now that everything is ready, utilize the demo.py script that was imported earlier to finally generate the DeepFake video. First load the model file that was downloaded earlier along with the configuration file that was available in the First-Order-Model repository that was cloned. And then generate the video utilizing the demo.make_animation() function and display the results utilizing the display_results() function.
# Load the pre-trained (check points) network and config file.
generator, kp_detector = demo.load_checkpoints(config_path='config/vox-256.yaml',
checkpoint_path='vox-adv-cpk.pth.tar')
# Create the deepfake video.
predictions = demo.make_animation(source_image, driving_video, generator, kp_detector, relative=True)
# Read the driving video, to get details, like FPS, duration etc.
reader = imageio.get_reader('crop.mp4')
# Get the Frame Per Second (fps) information.
fps = reader.get_meta_data()['fps']
# Save the generated video to the disk.
imageio.mimsave('results.mp4', [img_as_ubyte(frame) for frame in predictions], fps=fps)
# Display the source image, driving video and the generated video.
HTML(display_results(source_image, driving_video, predictions).to_html5_video())
Step 4.4: Embed the manipulated face into the source image
Create a function embed_face() that will simply insert the manipulated face in the generated video back to the source image.
def embed_face(source_image, source_image_data, generated_video_path, debugging=False):
'''
This function inserts the manipulated face in the generated video back to the source image.
Args:
source_image: The original source image from which the face was cropped.
source_image_data: A list containing the information required to embed the face back to the source image.
generated_video_path: The path where the video generated by the model is stored.
debugging: A boolean value that is if set to True, the intermediate steps are displayed.
Returns:
output_video_path: The path where the output video is stored.
'''
# Resize the image to make its width 720, while keeping its aspect ratio constant.
source_image = cv2.resize(source_image, dsize=(720, int((720/source_image.shape[1])*source_image.shape[0])))
# Get the height and width of the image.
height, width, _ = source_image.shape
# Get the face coordinates in the original image and calculate the face angle.
(xmin, ymin, xmax, ymax), face_angle = source_image_data
# Rotate the source image with the face angle.
rotated_image = rotate_image(source_image, face_angle, display=False)
# Get the height and width of the rotated image.
rotated_height, rotated_width, _ = rotated_image.shape
# Create a black image with size equal to the rotated image.
mask = np.zeros(shape=(rotated_height, rotated_width), dtype=np.uint8)
# Get the width and height of the face bounding box.
bbox_width, bbox_height = xmax-xmin, ymax-ymin
# Calculate the center coordinate of the face bounding box.
center_x, center_y = xmin+(bbox_width//2), ymin+(bbox_height//2)
# Initialize a variable to store the weight.
weight = 1
# Get the approximate width and height of the face in the bounding box.
roi_width = int(bbox_width/1.3)
roi_height = int(bbox_height/1.2)
# Draw a white filled rectangle at the center of the face bounding box on the mask image.
mask = cv2.rectangle(mask, (center_x-(roi_width//2), center_y-(roi_height//2)),
(center_x+(roi_width//2), center_y+(roi_height//2)),
(255*weight), thickness=-1)
# Iterate until the roi size is less than the face bounding box.
while roi_width<bbox_width and roi_height<bbox_height:
# Draw a gray rectangle around the face rectangle on the mask image.
# This will help in blending the face roi in the source image.
mask = cv2.rectangle(mask, (center_x-(roi_width//2), center_y-(roi_height//2)),
(center_x+(roi_width//2), center_y+(roi_height//2)),
(255*weight), thickness=int(roi_height/40))
# Check if the roi width is less than the face bounding box width.
if roi_width<bbox_width:
# Increment the roi width.
roi_width+=bbox_width//40
# Check if the roi height is less than the face bounding box height.
if roi_height<bbox_height:
# Increment the roi height.
roi_height+=bbox_height//40
# Decrement the weightage.
weight-=0.1
# Draw a rectangle at the edge of the face bounding box.
mask = cv2.rectangle(mask, (center_x-(roi_width//2), center_y-(roi_height//2)),
(center_x+(roi_width//2), center_y+(roi_height//2)),
(255*weight), thickness=int(roi_height/40))
# Load the generated video file.
video_reader = cv2.VideoCapture(generated_video_path)
# Define the Codec for Video Writer.
fourcc = cv2.VideoWriter_fourcc(*"XVID")
# Specify the path to store the final video.
output_video_path = "final_video.mp4"
# Initialize the video writer.
video_writer = cv2.VideoWriter(output_video_path, fourcc, 24, (1280, int((1280/width)*height)))
# Merge the mask three times to make it a three channel image.
mask = cv2.merge((mask, mask, mask)).astype(float)/255
# Iterate until the video is accessed successfully.
while video_reader.isOpened():
# Read a frame.
ok, frame = video_reader.read()
# Check if the frame is not read properly then break the loop.
if not ok:
break
# Resize the frame to match the size of the cropped (face) region.
frame = cv2.resize(frame, dsize=(xmax-xmin, ymax-ymin))
# Create a copy of the rotated image.
rotated_frame = rotated_image.copy()
# Embed the face from the generated video into the rotated source image.
rotated_frame[ymin: ymax, xmin : xmax] = frame
# Blend the edges of the image.
output_image = (((1-mask)) * rotated_image.astype(float)) + (rotated_frame.astype(float) * (mask))
# Undo the rotation and remove the black borders.
output_image = restore_source_image(output_image.astype(np.uint8), face_angle, image_size=source_image.shape[:2],
display=False)
# Resize the image to make its width 1280, while keeping its aspect ratio constant.
output_image = cv2.resize(output_image, dsize=(1280, int((1280/width)*height)))
# Write the frame.
video_writer.write(output_image)
# Check if debugging is enabled.
if debugging:
# Display the intermediate steps.
plt.figure(figsize=[15,15])
plt.subplot(121);plt.imshow(mask, cmap='gray');plt.title("Mask Image");plt.axis('off');
plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title(f"Output Image");plt.axis('off');
break
# Release the video writer, video reader and close all the windows.
video_writer.release()
video_reader.release()
cv2.destroyAllWindows()
# Return the output video path.
return output_video_path
Now let’s utilize the function embed_face() to insert the manipulated face into the source image.
# Discard the output of this cell.
%%capture
# Embed the face into the source image.
video_path = embed_face(cv2.imread(os.path.join(os.getcwd(), IMAGES_DIR , image_name)),
source_image_data=[face_bbox, face_angle], generated_video_path="results.mp4",
debugging=False)
# Check if the video with the FPS already exists.
if os.path.exists('final_video_with_fps.mp4'):
# Remove the video.
os.remove('final_video_with_fps.mp4')
# Add FPS information to the video.
!ffmpeg -i {video_path} -filter:v fps=fps=23 final_video_with_fps.mp4
The video is now stored on the disk, so now we can display it to see what the final result looks like.
# Download the video.
files.download('result_with_audio.mp4')
# Load the video.
video = open("result_with_audio.mp4", "rb").read()
# Decode the video.
data_url = "data:video/mp4;base64," + b64encode(video).decode()
# Display the video.
HTML(f"""<video width=400 controls><source src="{data_url}" type="video/mp4"></video>""")
Step 5: Add Audio (of the Driving Video) to the DeepFake Output Video
In the last step, first copy the audio from the driving video into the generated video and then download the video on the disk.
# Discard the output of this cell.
%%capture
# Check if the video with the audio already exists.
if os.path.exists('result_with_audio.mp4'):
# Remove the video.
os.remove('result_with_audio.mp4')
# Copy audio from the driving video into the generated video.
!ffmpeg -i crop.mp4 -i final_video_with_fps.mp4 -c copy -map 1:v:0 -map 0:a:0 -shortest result_with_audio.mp4
# Download the video.
files.download('result_with_audio.mp4')
The video should have started downloading in your system.
Bonus: Generate more examples
Now let’s try to generate more videos with different source images.
# Discard the output of this cell.
%%capture
# Specify the path of the source image.
image_path = 'elon.jpeg' # face_scale_factor=0.45
# image_path = 'drstrange.jpeg' # face_scale_factor= 0.55
# image_path = 'johnny.jpeg' # face_scale_factor=0.7
# image_path = 'mark.jpeg' # face_scale_factor=0.55
# Read another source image.
source_image = cv2.imread(os.path.join(os.getcwd(), IMAGES_DIR , image_path))
# Resize the image to make its width 720, while keeping its aspect ratio constant.
source_image = cv2.resize(source_image, dsize=(720, int((720/source_image.shape[1])*source_image.shape[0])))
# Perform face alignment and crop the face.
face_roi, angle, bbox = align_crop_face(source_image, face_mesh, face_detection,
face_scale_factor=0.45, display=False)
# Resize the Source Image to 256x256.
face_roi = resize(face_roi, (256, 256))[..., :3]
# Create the deepfake video.
predictions = demo.make_animation(face_roi[:,:,::-1], driving_video, generator, kp_detector, relative=True)
# Read the driving video, to get details, like FPS, duration etc.And get the Frame Per Second (fps) information.
reader = imageio.get_reader('crop.mp4')
fps = reader.get_meta_data()['fps']
# Save the generated video to the disk.
imageio.mimsave('generated_results.mp4', [img_as_ubyte(frame) for frame in predictions], fps=fps)
# Embed the face into the source image.
video_path = embed_face(source_image, source_image_data=[bbox, angle], generated_video_path="generated_results.mp4")
# Check if the video with the FPS already exists.
if os.path.exists('final_video_with_fps.mp4'):
# Remove the video.
os.remove('final_video_with_fps.mp4')
# Add FPS information to the video.
!ffmpeg -i {video_path} -filter:v fps=fps=23 final_video_with_fps.mp4
# Check if the video with the FPS already exists.
if os.path.exists('result_with_audio.mp4'):
# Remove the video.
os.remove('result_with_audio.mp4')
# Copy audio from the driving video into the generated video.
!ffmpeg -i crop.mp4 -i final_video_with_fps.mp4 -c copy -map 1:v:0 -map 0:a:0 -shortest result_with_audio.mp4
# Download the video.
files.download('result_with_audio.mp4')
# Load the video.
video = open("result_with_audio.mp4", "rb").read()
# Decode the video.
data_url = "data:video/mp4;base64," + b64encode(video).decode()
# Display the video.
HTML(f"""<video width=400 controls><source src="{data_url}" type="video/mp4"></video>""")
And here are a few more results on different sample images:
After Johnny Depp, comes Mark Zuckerberg sponsoring Bleed AI.
And last but not least, of course, comes someone from the Marvel Universe, yes it’s Dr. Strange himself asking you to visit Bleed AI.
You can now share these videos that you have generated on social media. Make sure that you mention that it is a DeepFake video in the post’s caption.
Conclusion
One of the current limitations of the approach we are using is when the person is moving too much in the driving video. The final results will be terrible because we are only getting the face ROI video from the First-Order Motion Model and then embedding the face video into the source image using image processing techniques. We can’t move the body of the person in the source image if the face is moving in the generated face ROI video. So for the driving videos in which the person is moving too much, you can skip the face embedding part or just train a First-Order Motion Model to manipulate the whole body instead of just the face, I might cover that in a future post.
A Message on Deepfakes by Taha
These days, It’s not a difficult job to create a DeepFake video, as you can see, anyone with access to the colab repo (provided when you download the code) can generate deepfakes in minutes.
Now these fakes are although realistic but you should be easily be able to tell between fake manipulation and real ones, this is because the model is particularly designed for faster interference, there are other approaches where it can take hours or days to render deepfakes but those are very hard to tell from real ones.
The model I used today, is not new but it’s already been out there for a few years (Fun fact: we were actually working on this blogpost since mid of last year so yeah this got delayed for more than a year) Anyways, the point is, the deepfake technology is fast evolving and leads to two things,
1) Easier accessibility: More and more high-level tools and coming which makes the barrier to entry easier and more non-technical people can use these tools to generate deepfakes, I’m sure you know some mobile apps that let common generate these.
2) Algorithms: algorithms are getting better and better such that, you’re going to find a lot of difficulty in identifying a deepfake vs a real video. Today, professional deepfake creators actually export the output of a deepfake model to a video editor and get rid of bad frames or correct them so people are not able to easily figure out if it’s a fake and it makes sense if the model generates a 10 sec (30fps) frames then not all 300 outputs are going to be perfect.
Obviously, deepfake tech has many harmful effects, it has been used to generate fake news, spread propaganda, and create pornography but it also has its creative use cases in the entertainment industry (check wombo) and in the content industry, just check out the amazing work syntheisia.io is doing and how it had helped people and companies.
One thing you might wonder is that in these times, how should you equip yourself to spot deepfakes?
Well, there are certainly some things you can do to better prepare yourself, for one, you can learn a thing or two about digital forensics and how you can spot the fakes from anomalies, pixel manipulations, metadata, etc.
Even as a non-tech consumer you can do a lot in identifying a fake from a real video by fact-checking and finding the original source of the video. For e.g. if you find your country’s president talking about starting a nuclear war with North Korea on some random person’s Twitter, then it’s probably fake no matter how real the scene looks. An excellent resource to learn about fact-checking is this youtube series called Navigating Digital Information by Crashcourse. Do check it out.
[optin-monster slug=”n7m5f6assjorcd80egr9″]
Hire Us
Let our team of expert engineers and managers build your next big project using Bleeding Edge AI Tools & Technologies
Join My Course Computer Vision For Building Cutting Edge Applications Course
The only course out there that goes beyond basic AI Applications and teaches you how to create next-level apps that utilize physics, deep learning, classical image processing, hand and body gestures. Don’t miss your chance to level up and take your career to new heights
You’ll Learn about:
Creating GUI interfaces for python AI scripts.
Creating .exe DL applications
Using a Physics library in Python & integrating it with AI
Advance Image Processing Skills
Advance Gesture Recognition with Mediapipe
Task Automation with AI & CV
Training an SVM machine Learning Model.
Creating & Cleaning an ML dataset from scratch.
Training DL models & how to use CNN’s & LSTMS.
Creating 10 Advance AI/CV Applications
& More
Whether you’re a seasoned AI professional or someone just looking to start out in AI, this is the course that will teach you, how to Architect & Build complex, real world and thrilling AI applications
