I believe that those answer should be deleted, here is why
Since closed question tends to stay around for a bit, other people see that a question is closed but they also see that the question got answers. I believe this may lead other people to post a question without worrying too much to know if it would be on-topic because they know they will have an answer anyway.
An even worse case scenario is when people refuse to make they question on-topic because they see that they have answers (this was the case here, currently the question has been edited again by someone other than OP to make it on-topic but I worry that OP will rollback the changes a 3rd? time).
I think that, if we delete answers from an off-topic question, people will stop answering questions that, they believe, might get closed and will, instead, try to prompt OP to make the question on-topic (which will lead to more quality questions \o/).
Another problem to answers to an off-topic question is that bad answers might to not be downvoted because: "why waste my downvote (who cost reputation) on something that won't be deleted?".
And because why regular users with downvote privilege would waste their time reading an off-topic question? The problem in this second case is that not-so-regular-user with upvote privilege might read the question and upvote the answer, even if the answer isn't backed-up and might contain terrible advice (if it's not backed-up, you might not realize that the answer is terrible).
Also, since the answer is closed, the terrible answer can't be challenged by another answer that would have been written after reading the terrible answer.