Personally, I have not been systematically flagging or downvoting these answers. I don't feel that answers written in good faith should be necessarily downvoted; however, "in good faith" here would mean not only that the answer is otherwise acceptable, but that the answerer reasonably believed that the question was okay.
For example, if the question is marked as a duplicate but the answer is substantially different than the linked post, my current logic would not fault the answerer. If the answer is different, that is evidence enough to me that this was written with the belief that this question is not a duplicate. I don't mean to say that intent should be evaluated in a merit-based vote system, but I recognize that there are gray areas with some duplicates.
For off-topic questions, I think it's important to note that even experienced users can disagree on the validity of questions every once in a while, and if I think they may have a legitimate argument for its topicality, I would be okay with that answer as well.
Most cases, though, are night and day: either on- or off-topic, unique or duplicate. When a question clearly should be closed, I will often hold answers to that question in some contempt. Generally, those are bad-faith answers, and while I don't have a hard rule, it doesn't take much for that kind of answer to get the down arrow from me. A flag is an available option, but honestly it's worth considering a comment. That might be the better option in a lot of cases, particularly with newer users.
Happily, most black-and-white SE questions don't attract any answers (Yay community!). So when a question IS being considered for closure and you believe it should be open,
- Don't vote to close. You may want to review as "Leave Open", but "Skip" is probably the better option when you're not sure.
- Leave a comment explaining why you think it should remain open. This will give future voters-to-close (and if things go south, voters-to-reopen!) your perspective on the question. Other users may disagree with your logic, but it's never a bad thing to get some perspectives out in the open. If things get long-winded,
- Inquire here on Meta M:PT for a community ruling and discussion.
Answering is possibly a bit presumptive if others are indicating that they strongly believe the question is a duplicate or off-topic, so use good judgement in that regard.
These are just my current opinions on the issue, and I would be happy to hear other ideas.