Dear Ms. Hanaoka, Mr. Grother, and Ms. Ngan,
Human Rights Watch and 5Rights Foundation write in response to the United States National Institute of Standards and Technology (NIST)’s invitation to submit comments regarding its newly proposed track: Face Recognition Vendor Test – Age Estimation.
As noted by NIST, policymakers are increasingly mandating online services to verify their users’ ages so to identify and apply protections for children.[1] Given this, we focus our recommendations on NIST’s proposed evaluation of age estimation as it relates to children.
- NIST should calculate all accuracy metrics by age in discrete years, including but not limited to the age range 0-18 years. Where age groups are used as an additional basis for calculation, 18 should be included as a boundary.
Under international law, a child is defined as every human being below the age of 18.[2] Similarly, in the US, most states have set the age of majority at 18, with the exception of three states that have set a higher age of majority.[3]
Privacy is a human right, and children of all ages are entitled to special protections that guard their privacy and the space for them to grow, play, and learn.[4] This is recognized in the California Age-Appropriate Design Code Act, the first child online privacy law passed in the US, which declares that all children under 18 should be afforded privacy protections online.[5]
In NIST’s recent evaluations in which the agency examined the performance differentials of face recognition algorithms as a function of age, it computed accuracy metrics on the basis of two systems of age groupings: [12-20, 20-35, 35-50, 50-65, 65-99][6], and [0-4, 4-10, 10-16, 16-20, 20-24, 24-28, 28-32, 32-36, 36-40, 40-48, 48-56, 56-64, 64-72, 72-120].[7]
These groupings do not allow for the assessment of algorithmic performance for the full range of childhood (0-18 years of age). These groupings may also obscure biases and errors for children of different ages, a significant concern given known high error rates of facial recognition algorithms when applied to children, particularly young children.[8] Finally, these groupings do not reflect how children’s capacities and need for protections in online environments – the stated purpose of age verification laws – may evolve throughout their childhood.
We recommend that NIST calculate its accuracy metrics by age in discrete years, so to enable granular evaluations of how variations and errors in algorithmic performance might affect children at each age. An example of this can be found in NIST’s Interagency Report 7995, in which NIST computed cumulative scores and mean absolute errors for children aged 0-14 in discrete years.[9] However, we recommend that NIST perform its calculations by age in discrete years for the full range of childhood – 0-18 years – and, ideally, for the range 0-25 years, so to assess the prevalence of errors in which young adults are mistakenly classified as older children.
Where age groupings are used as an additional basis for calculation, NIST should set 18 as a boundary age, for example, [16-17, 18-19], or [16-18, 19-21]. Age groupings that combine children and adults (for example [16-18]) should only be used as an additional basis for calculation, never as the sole basis.
- As part of its set of accuracy metrics, NIST should add a new statistical indicator that captures the direction of error.
For its new facial age estimation track, NIST has stated that it will compute and report “statistics of the difference between the actual and estimated ages, e.g., Mean Absolute Error,” as well as false positive and negative rates.[10]
In addition to these performance metrics and those calculated by NIST in its 2014 report on age estimation[11], NIST should consider adding a new statistical indicator, such as mean error, that would capture the direction of errors when they occur. For example, if a given algorithm is found to have a mean absolute error of 2.3 years when estimating the age of 13-year-old children, it would be valuable to know the mean error (+/- 2.3 years) so to understand whether that algorithm estimates, on average, 13-year-old children to be 2.3 years older or younger than their actual age.
- NIST should disclose further information about its testing datasets to organizations and researchers focused on child and human rights.
Offering greater transparency and information on NIST’s testing datasets would help enable child and human rights impact assessments of age verification algorithms, with a focus on interpreting NIST’s results to understand, and develop safeguards for, the likely impacts of age estimation algorithms on children’s rights in real-world conditions.
Thank you for your consideration. Please do not hesitate to contact us if we can provide further information.
Sincerely,
5Rights Foundation
Human Rights Watch
[1] US National Institute of Standards and Technology (NIST), “FRVT Age Estimation,” https://pages.nist.gov/frvt/html/frvt_age_estimation.html (accessed July 21, 2023)
[2] See e.g. Convention on the Rights of the Child, art. 1, Nov. 20, 1989, 1577 U.N.T.S. 3 (entered into force September 2, 1990); and Human Rights Committee, General Comment No. 17, 1989, HRI/GEN/1/Rev.8, para. 4. The United States signed the Convention on the Rights of the Child in 1995 but has not ratified it.
[3] Alabama and Nebraska have set their age of majority at 19; Mississippi has set its age of majority at 21. Ala. Code § 26-1-1; Neb. Rev. Stat. § 43-2101; Miss. Code tit. 1, § 1-3-27.
[4] International Covenant on Civil and Political Rights (ICCPR), arts. 17, 24, December 16, 1966, 999 U.N.T.S. 171 (entered into force March 23, 1976, and ratified by the United States June 8, 1992). In his 2021 report to the UN Human Rights Council, the Special Rapporteur on the right to privacy stated that children’s right to privacy “enables their access to their other rights critical to developing personality and personhood, such as the rights to freedom of expression and of association and the right to health, among others.” See UN Human Rights Council, Rep. of the Special Rapporteur on the Right to Privacy, paras. 67-76, U.N. Doc. A/HRC/46/37 (Jan. 25, 2021). See alsoConvention on the Rights of the Child, art. 16; Convention on the Rights of Persons with Disabilities (CRPD), art. 22, December 13, 2006, 2515 U.N.T.S. 3 (entered into force May 3, 2008). The United States signed the CRPD in 2009 but has not ratified it.
[5] California Age-Appropriate Design Code Act, State of California Assembly Bill No. 2273, Section 1(a)(1)- (7) and art. 1798.99.30 (b)(1).
[6] US NIST, “Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects,” NISTIR 8280, December 2019, https://pages.nist.gov/frvt/reports/demographics/nistir_8280.pdf (accessed July 21, 2023), p. 50, figure 14; and p. 53, figure 16.
[7] Ibid, p. 51, figure 15. See also: US NIST, “Ongoing Face Recognition Vendor Test (FRVT) Part 1: Verification,” NISTIR XXXX Draft, June 16, 2023, https://pages.nist.gov/frvt/reports/11/frvt_11_report.pdf (accessed July 21, 2023), pp.410-453, figures 335-376.
[8] See e.g.: US NIST, “Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects,” NISTIR 8280, December 2019, https://pages.nist.gov/frvt/reports/demographics/nistir_8280.pdf (accessed July 21, 2023), pp. 8, 52; Srinivas, N., et al., “Face Recognition Algorithm Bias: Performance Differences on Images of Children and Adults,” IEEE/CVS Conference on Computer Vision and Pattern Recognition Workshops, June 2019, https://ieeexplore.ieee.org/document/9025674 (accessed July 21, 2023); Michalski, D., Yiu, S.Y., Malec, C., “The Impact of Age and Threshold Variation on Facial Recognition Algorithm Performance using Images of Children, International Conference on Biometrics, February 2018, https://ieeexplore.ieee.org/document/8411225 (accessed July 21, 2023).
[9] US NIST, “Face Recognition Vendor Test (FRVT) Performance of Age Estimation Algorithms,” NISTIR 7995, March 20, 2014, https://nvlpubs.nist.gov/nistpubs/ir/2014/NIST.IR.7995.pdf (accessed July 21, 2023), p. 22, tables 10 and 11.
[10] US NIST, “Face Recognition AE (Age Estimation), Application Programming Interface Version 0.8,” July 3, 2023, https://pages.nist.gov/frvt/api/FRVT_ongoing_AE_api_draft.pdf (accessed July 21, 2023), p. 3.
[11] US NIST, “Face Recognition Vendor Test (FRVT) Performance of Age Estimation Algorithms,” NISTIR 7995, March 20, 2014, https://nvlpubs.nist.gov/nistpubs/ir/2014/NIST.IR.7995.pdf (accessed July 21, 2023), pp.9-10.
