Drinking while wearing fraternity/sorority letters?

The unwritten rule is that you shouldn't wear your greek letters in a bar while drinking. Why is that? Also, what if you are at a party? Can you wear your letters then, or does it not matter?