When a model is wrong, who is allowed to stop it?

Human oversight is not a person standing near a model. It is the authority, evidence and safe state that let somebody refuse an output, interrupt a process...