I agree with Your Uncle Bob's points that controversial questions are not such an issue on this site (at least nowhere near the level they might be on, say, one of the religion sites) and that quality control is going to be the main reason for manually removing a question from HNQs.
Of course the formula - stated in approximate form here...
(MIN(AnswerCount, 10) * QScore) / 5 + AnswerScore
-------------------------------------------------
MAX(QAgeInHours + 1, 6) ^ 1.4
...is supposed to take account of quality, in that it looks at how many answers a question has, what the question score is, and so on. So perhaps the only questions that should have to be dealt with manually would be ones where those metrics are obviously misleading in some way - for example, if someone has written a question that's unclear, and the 'answers' are in fact more like extended comments pointing out the issues with the question.
If it seems that a question has been well-received by the core community (e.g. picked up genuine upvotes before it goes 'hot'), I wouldn't imagine that there are many circumstances where it should be taken off the HNQ list manually.