How does facial recognition work?

What is Facial Recognition?

How does facial recognition work? Facial recognition is a way of recognizing a human face through technology. A facial recognition system uses biometrics to map the facial features of a photograph or video. Compare the information with a database of known faces to find a match.

Most people are comfortable using facial recognition using it in Instagram filters and Face ID. But this relatively new technology can get a little attention when you stop to think about it. Your face is like a fingerprint, and the technology behind facial recognition is complex.

As with any new technology, facial recognition has its downsides. These drawbacks are becoming more apparent as the military, police, advertisers, and counterfeit creators find new ways to take advantage of facial recognition software.

Now, more than ever, it is essential that people understand how facial recognition works. It is also important to know the limitations of facial recognition and how it will develop in the future.

Face recognition is surprisingly simple

Before entering the different means of facial recognition, it is important to understand how the facial recognition process works. Here are three applications for facial recognition software, and a simple explanation of how faces are recognized or identified:

• Basic facial recognition: For Animoji and Instagram filters, your phone’s camera “searches” for the defining features of a face, specifically a pair of eyes, a nose, and a mouth. Then, it uses algorithms to block a face and determine which direction it is facing, if its mouth is open, etc. It is worth mentioning that this is not a facial ID, it is just a software that searches for faces.
• Face ID and similar programs: By configuring Face ID (or similar programs) on your phone, the application takes a photo of your face and measures the distance between your facial features. So every time you go to unlock your phone, it “looks” through the camera to measure and confirm your identity.
• Identify a stranger: When an organization wants to identify a face for security, advertising, or surveillance purposes, it uses algorithms to compare it to a large database of faces. This process is almost identical to Apple’s Face ID but on a larger scale. Theoretically, any database (ID cards, Facebook profiles) can be used, but a database of clear, pre-identified photos is ideal.

All right, let’s get to the bottom of the matter. Because the “basic facial recognition” used for Instagram filters is such a simple and harmless process, we are going to focus entirely on facial identification and the many different technologies that can be used to identify a face.

Most facial recognition depends on 2D images

Unsurprisingly, most facial recognition programs rely exclusively on 2D images. But this is not done because the 2D facial image is super accurate, it is done for convenience. The vast majority of cameras take photos without depth, and public photos that can be used for facial recognition databases, for example, Facebook profile photos, are all in 2D.

Why are 2D facial images not super accurate? Well, because a flat image of your face lacks identifying features, like depth. With a flat image, a computer can measure the distance of the pupil and the width of the mouth, among other variables. But it cannot tell the length of your nose or the prominence of your forehead.

Additionally, 2D facial imaging is based on the visible light spectrum. This means that 2D facial images do not work in the dark, and can be unreliable in dark or funky lighting conditions.

Clearly, the way to overcome some of these shortcomings is to use 3D facial images. But how is it possible? Do you need special equipment to see a face in 3D?

IR cameras add depth to your identity

Although some facial recognition applications are based solely on 2D images, it is not uncommon for facial recognition to be based on 3D images as well. In fact, his experience with facial recognition probably involves a bit of 3D.

This is accomplished through a technique called lidar, which is similar to sonar. Essentially, facial scanning devices, like your iPhone, blow up a harmless infrared matrix on your face. This matrix, a wall of lasers, is reflected on your face and is captured by an IR camera (or ToF camera) on your phone.

Where does 3D magic occur? Your phone’s infrared camera measures the time it takes for each bit of infrared light to bounce off your face and return to the phone. Naturally, the light reflected from your nose will take a shorter trip than the light reflected from your ears, and the IR camera uses this information to create a unique depth map of your face. When used in conjunction with basic 2D imaging, 3D imaging can significantly increase the accuracy of facial recognition software.

The Lidar image is a strange concept that can be difficult to understand. If it helps you, try to imagine that the infrared mesh of your phone (or any facial recognition device) is a toy or frame where you support your face and the surface adapts to it. Like such aboard your face leaves an imprint on the infrared mesh, where your nose is noticeably deeper than, say, your eyes.

Thermal imaging allows facial recognition to work at night

One of the flaws of 2D facial recognition is that it relies on the visible spectrum of light. In simple terms, basic facial recognition doesn’t work in the dark. But this can be solved using a thermal imaging camera (yes, like in action movies).

“Wait a minute,” you might say, “isn’t the thermal image based on infrared light?” Yes, it is. But thermal imaging cameras do not emit bursts of infrared light; they simply detect the infrared light emitted by objects. Hot objects emit a ton of IR light, while cold objects emit a negligible amount of IR light. Expensive thermal imaging cameras can even detect subtle temperature differences on a surface, making them the ideal technology for facial recognition.

There are a handful of different ways to identify a face with thermal imaging. All of these techniques are incredibly complicated, but they share some fundamental similarities, so let’s try to keep things simple with a list:

• Multiple photos are required: A thermal imaging camera takes multiple photos of a subject’s face. Each photo focuses on a different spectrum of IR light (long, short, and medium waves). Typically, the longwave spectrum provides the greatest facial detail.
• Blood vessel maps are helpful: These infrared images can also be used to remove blood vessel formation from a person’s face. It’s creepy, but blood vessel maps can be used as unique facial fingerprints. They can also be used to find the distance between facial organs (if typical thermal imaging produces poor-quality imaging) or to identify bruising and scarring.
• The subject can be identified: A composite image (or data set) is created using multiple IR images. This composite image can be compared to a facial database to identify the subject.

Of course, thermal facial recognition is often used by the military, it’s not something you’ll find in a department store, and it’s not something that will come with your next mobile phone. Also, thermal imaging doesn’t work well during the day, in generally well-lit environments, so it doesn’t have many potential applications outside of the military.

Limitations of facial recognition

We have spent a lot of time talking about deficiencies in facial recognition. As we have seen in infrared and thermal imaging, it is possible to overcome some of these limitations. But there are still some problems that have not been solved yet:

• Obstruction: Unsurprisingly, sunglasses, and other accessories can trip up facial recognition software.
• Poses: Facial recognition works best with a neutral, forward-facing image. Tilt or turn of the head can make facial recognition difficult, even for IR-based recognition software. Also, a smile, puffy cheeks, or any other pose can change the way a computer measures your face.
• Light: All forms of facial recognition depend on light, be it the visible spectrum or IR light. As a result, unusual lighting conditions can decrease the accuracy of facial identification. This may change as scientists are currently developing sonar-based facial recognition technology.
• The Database: Without a good database, facial recognition can’t work. Along these same lines, it is impossible to identify a face that has not been correctly identified in the past.
• Data Processing: Depending on the size and format of a database, computers may take a while to correctly identify faces. In some situations, such as the police, limitations on data processing restrict the use of facial identification for everyday applications (which is probably a good thing).

As of now, the best way to get around these limitations is to use other forms of identification along with facial recognition. Your phone will ask for a password or a fingerprint if it doesn’t identify your face, and the Chinese government uses ID cards and tracking technology to close the margin of error that exists in its facial recognition network.

In the future, scientists will surely find a way to get around these problems. They can use sonar technology in conjunction with lidar to create 3D face maps in any environment, and they can find ways to process face data (and identify strangers) in an incredibly short time. Either way, this technology has great abuse potential, so it’s worth keeping up to date.

How does facial identification technology work?