Oh, I see. In that situation, I would have just said yes, and then false cut the deck or something.
Sometimes, people just aren't paying close attention to what you're doing, and they miss things that actually make them seem more observant than they really are. I wouldn't second guess yourself...