First of all, during the course of this discussion, let's try to keep up the respect we ought to have for other people, even those who have opinions which differ from our own and which we might find stupid. I know most people on this forum would do that anyway, and I don't really expect things to go "bad", this is just a friendly reminder. This is a topic on which discussion can get quite heated when our own beliefs come into it, so it doesn't hurt to remember this.
The answer you give depends on the person. If it's a stranger you wanted to perform magic for, the answer should be, as pointed out by a few people already: "Thank you and have a nice day." There's really no need to get into a theological discussion with that person.
If it's somebody you know well and you can have this kind of discussion with him, I think it's worth discussing. For the sake of argument, I'll just assume the person is a christian; I think it's similar for a jew, but it may be different with a muslim.
Usually the reason for this person's opinion is a wrong understanding of what the Bible is saying. When the Bible talks about magic it talks about meddling with supernatural forces, oftentimes even channelling demons. Naturally, the Bible tells us not to do this.
The problem is: In English, as well as German and, I assume, many other languages, the word used for this sort of magic and
our kind of "magic" (sleight of hand, trickery, maths etc.) is the same, so it can easily lead to confusion.
Of course most people have a good understanding that today's magic tricks aren't "magic" in that sense, but there might be confusion to the performers motives: Does the performer try to look like he's channelling supernatural powers? If so, this might be considered similar to somebody pretending to commit murder: Not the actual "bad" thing, but definitely not amusing, to say the least.
When discussing this, it's important to understand why people think that magic tricks are against their religion. As I said, I think it's worth having this discussion with people you know well and who think that way, because they really miss out on something.
But the most important thing is to remember that for some people - whether it's right or not - this belief is deeply ingrained in their culture and upbringing, and whether or not we agree we should always respect that.
If somebody wants to read a little more on this subject, there's a good essay on the Magic Café Forums about it:
https://www.themagiccafe.com/forums/viewtopic.php?topic=164580&forum=198
Thank you all for reading all of this, I'm sorry if I rambled a little along the way

Have a good day, everybody!