My wife and I started buying legos when we got married, and have been doing so for going on 19 years now. It's hard not to notice we love legos, when you step through our front door, Legos are one of the first things you see.
I really don't have a problem letting people know I enjoy playing with them, I usually have built lego models on my desk at work and many of my customers are interested and make comments about them. As for my co-workers, they already know I'm nuts.