I don't blame you. It gets on my nerves when people do it and if I was a veteran, I'd have to fight myself not to get angry with them. Are you wearing clothes that give you away like your army suit or a hat that says veteran on it? Try to dress casually and see if things change.
Good Luck!