A map of monitoring gaps
NIST AI 800-4 draws on practitioner workshops and a literature review. It organises challenges in monitoring deployed AI, rather than presenting a single monitoring tool or a certification scheme. [1]
More than uptime
Its categories cover functionality, operations, human factors, security, compliance and large-scale impacts. Reported barriers include detecting drift and joining up fragmented logs. The report also raises questions about monitoring frequency and combining automated checks with human validation. This March publication remains background research, not a new October announcement. [1]
ADM perspective
Use separate indicators for availability, decision quality and user outcomes. A dashboard can stay green while an automated process makes the wrong choice repeatedly. Define who reviews each signal and what action follows a threshold breach.
Read the original sources
The original publications provide the full detail, evidence and limitations.
- NIST / CAISI · Primary source - official publicationNew Report: Challenges to the Monitoring of Deployed AI Systems
An AI-assisted, source-led summary. “ADM perspective” is editorial interpretation, separate from the source’s findings. Our editorial approach.