Benchmarking and Evaluation: What Fuels Biometric Algorithm Accuracy?

Summary: The article discusses the importance of third-party algorithm testing, draws on the evaluations conducted by the National Institute of Standards and Technology (NIST) and provides an overview of independent benchmarks.

Biometrics are driven by algorithms that help recognize individuals based on their unique biological markers, whether it's the iris, face, palmprint, fingerprint or voice. Yet, how to distinguish a trustworthy and precise algorithm? A biometric technology vendor can evaluate the quality of their algorithm using specific industry-leading evaluations. So how can an algorithm be measured? The development of biometric recognition algorithms and their reliability improvement is driven by benchmarks. These evaluative measures are used to assess and compare performance, which helps proprietary technologies move forward.

These benchmarks are particularly valuable as they allow for cross-vendor comparisons, providing an objective way to measure the performance of different algorithms. While developers can test accuracy themselves, independent tests by organizations like the National Institute of Standards and Technology (NIST) are important for providing a neutral, third-party validation of an algorithm's performance.

In this article, we will explain the process in greater detail and provide insights into algorithm development.

What is the role of independent testing and why does it matter?

Independent testing is important because it provides a clear, impartial benchmark for algorithm performance. These evaluations are conducted using large, standardized datasets, and offer a way to objectively compare different algorithms, while encouraging continuous technology development.

The tests provide an overall performance evaluation of a biometric algorithm. Although they don't show you exactly which images from the dataset are doing badly, they do give you an idea of how to improve your algorithm so that it performs better in general. Technology vendors then use the results to identify and address potential weaknesses. The thorough, third-party inspection of these technologies raises confidence in their potential use in fields like voter management, law enforcement, finance and security, as it provides an objective measure of their performance.

NIST conducts open evaluations, where different biometric providers can submit their algorithms, consequently creating a competitive environment and instilling the willingness for technological improvement.

"NIST's position of open testing and its ongoing principle create a kind of competition among algorithm developers, and in doing so, it acts as one of the biggest motivators for the researchers working on the technology. This process encourages us, the vendors, to elevate our solutions and allows us to truly assess which solutions are the most reliable." Evaldas Borcovas
Biometric Team Lead at Neurotechnology

What are some of the key NIST evaluations for biometrics?

NIST runs a series of specialized evaluations that serve as industry benchmarks for different biometric technologies.

Fingerprint

Minutiae Interoperability Exchange (MINEX) III: This program tests the performance and interoperability of fingerprint template generators and matchers, ensuring that systems from different vendors can work together.
Proprietary Fingerprint Template (PFT) III: This evaluation measures the performance of proprietary, one-to-one (verification) fingerprint matching.
Slap Fingerprint Segmentation Evaluation (SlapSeg) III: SlapSeg III tests how well an algorithm performs slap fingerprint (simultaneous impression of four fingers or two thumbs) segmentation.
FRIF TE E1N: This evaluation measures the performance of proprietary, one-to-many (identification) fingerprint matching.
Evaluation of Latent Friction Ridge Technology (ELFT): ELFT specifically evaluates the accuracy and computational performance of algorithms designed to identify latent fingerprints, which are used in forensic and law enforcement applications.

Face

FRTE Projects focus on evaluating the performance of face recognition systems, primarily identifying who is in an image:
- FRTE 1:1 Verification: evaluates how well an algorithm performs face recognition in one-to-one verification scenarios.
- FRTE 1:N Identification: measures algorithm performance in one-to-many identification scenarios.
- FRTE Demographic Effects: this evaluation tests how face recognition algorithms perform across different demographic groups, measuring variations in false matches and false non-matches by age, sex and ethnicity.
- FRTE Face Mask Effects: assesses algorithm accuracy when people are wearing face masks.
- FRTE Paperless Travel: focuses on algorithm performance in one-to-many identification scenarios for airport boarding and security without the use of paper documents.
- FRTE Twins Demonstration: tests how effectively face recognition algorithms can distinguish between identical and same-sex fraternal twins.
- FRTE FIVE: evaluates the performance of face recognition algorithms in detecting and identifying people in video sequences.
FATE Projects: concentrate on analyzing facial images for attributes other than identity recognition:
- FATE MORPH: tests how well algorithms can detect and analyze face morphing, including distinguishing morphed photos from genuine ones and recovering original identities.
- FATE Quality: measures how well algorithms can assess the quality of facial images and predict when poor-quality photos may lead to recognition errors.
- FATE PAD: focuses on the detection of presentation attacks against face recognition systems, such as photos, videos or masks used to spoof authentication.
- FATE Age Estimation & Verification: determines how accurately algorithms can estimate a person's age from facial images and verify whether they meet age-related requirements.

Iris

Iris Exchange (IREX) 10: NIST IREX 10 is an ongoing evaluation program that assesses the accuracy and performance of iris recognition algorithms.

Voice

SRE24: measures how well algorithms can detect and recognize speakers across multiple sources and languages, including telephone calls and video recordings.

These evaluations serve as an unbiased platform for comparing technologies and establishing effective benchmarks.

What is the role of large datasets?

Developing a biometric algorithm requires an immense amount of data to ensure it can perform reliably across a wide range of inputs and demographics. Developing a biometric algorithm requires a great amount of data to ensure it can perform reliably across a wide range of inputs and demographics. For this reason, many developers combine real-world data with synthetic data to have a more extensive collection. However, it is practically impossible for developers to generate the immense volume of real-world data needed for an impartial evaluation. Luckily, NIST, which operates under specific government regulations, possess the necessary datasets to conduct a truly accurate and conclusive test of these algorithms.

Why is it important to use the right method?

Not all evaluative methods are suitable for all applications. For example, some are ideal for biometric verification (a one-to-one match), while others are better suited for identification (a one-to-many search). By selecting and applying the correct methods, developers can ensure their algorithm is not only high-performing but also perfectly aligned with the specific requirements of its application (e.g., for law enforcement scenarios).

For example, The FRTE 1:N Identification evaluation is an ongoing assessment by NIST, which tests facial recognition algorithms specifically for both civilian(identification) and law enforcement (investigation) applications. The FRTE includes a fully automated mode and a semi-automated mode, which requires the review of a human expert.

The scope of NIST's biometric testing extends beyond just faces; it also includes evaluations like ELFT and IREX 10, all of which emphasize metrics critical for investigative work. In all of these evaluations, NIST closely measures rank accuracy, which is an important indicator for the investigation scenario because it determines how effectively the system returns the correct match within the top candidates for a human expert to review.

Final remarks

As we have learned, benchmarks are important for the evaluation and validation of algorithm accuracy. The process of evaluation, conducted by third-party institutions such as NIST, measures the accuracy and performance of algorithms, which is essential for technological progress and ensures their reliability before they are deployed.