Eighty percent is a respectable test grade for a subject that a student is unfamiliar with, but may not be for a police facial recognition system being used to investigate crimes.
That is the similarity score that Delhi Police are using to indicate a positive match, the force has confirmed in a response to a query from the Internet Freedom Foundation, an Indian non-governmental organization advocating for civil liberties online.
India's The Economic Times misinterpreted the response, reporting that Delhi Police's face biometrics system peaks at 80 percent accuracy.
Law enforcement officials approved the use of the algorithms for identifying participants in a pair of recent public protests, one of which turned into a riot.
The software's accuracy registered as low as 2 percent with Delhi Police in 2019, and it can be as high as 99.97 percent, according to the Times, but a subject would almost have to be posing in direct, even light and facing the camera.
Data scientists have long said that common methods of calculating and expressing the accuracy of biometric systems are wrong. System accuracy is made up of both false positive and false negative, one of which can be improved at the expense of the other. Given the confidence threshold confusion, all numbers from the report should be treated with caution.
Regardless, 80 percent is often considered inadequate for high-stakes biometric application like law enforcement investigation. Amazon was criticized in the U.S. for even offering it as an option, while recommending 95 percent.
The U.S.' National Institute of Standards and Technology hosts an ongoing performance assessment of facial recognition systems, called the Face Recognition Vendor Test. It is not known if the India police's algorithm has been submitted for testing.
A test of an Idemia 1:N matching algorithm by the U.S. DHS’s S&T in an airport scenario last year showed a true identification rate of 99.7 percent, when tuned to a false identification rate of 1 in 100,000, which the agency considers acceptable for the use case’s security needs.
This post was updated at 7:22pm Eastern on August 19, 2022 to correct that the Delhi Police statement refers to confidence threshold, not accuracy.
Eighty percent is a respectable test grade for a subject that a student is unfamiliar with, but may not be for a police facial recognition system being used to investigate crimes.
That is the similarity score that Delhi Police are using to indicate a positive match, the force has confirmed in a response to a query from the Internet Freedom Foundation, an Indian non-governmental organization advocating for civil liberties online.
India's The Economic Times misinterpreted the response, reporting that Delhi Police's face biometrics system peaks at 80 percent accuracy.
Law enforcement officials approved the use of the algorithms for identifying participants in a pair of recent public protests, one of which turned into a riot.
The software's accuracy registered as low as 2 percent with Delhi Police in 2019, and it can be as high as 99.97 percent, according to the Times, but a subject would almost have to be posing in direct, even light and facing the camera.
Data scientists have long said that common methods of calculating and expressing the accuracy of biometric systems are wrong. System accuracy is made up of both false positive and false negative, one of which can be improved at the expense of the other. Given the confidence threshold confusion, all numbers from the report should be treated with caution.
Regardless, 80 percent is often considered inadequate for high-stakes biometric application like law enforcement investigation. Amazon was criticized in the U.S. for even offering it as an option, while recommending 95 percent.
The U.S.' National Institute of Standards and Technology hosts an ongoing performance assessment of facial recognition systems, called the Face Recognition Vendor Test. It is not known if the India police's algorithm has been submitted for testing.
A test of an Idemia 1:N matching algorithm by the U.S. DHS’s S&T in an airport scenario last year showed a true identification rate of 99.7 percent, when tuned to a false identification rate of 1 in 100,000, which the agency considers acceptable for the use case’s security needs.
This post was updated at 7:22pm Eastern on August 19, 2022 to correct that the Delhi Police statement refers to confidence threshold, not accuracy.