I absolutely agree with you. Well I'm 14 and only my family knows it. My friends knew that I still played with it when I was 12, but after that I always clean up my room before friends are coming. It's not really that I am ashamed for it, I just know people are going to bully me cause the already did that when I was 12. I also don't like to call it playing with Lego, It's just collecting and building with it. But I think if you're 21, like you, you can be open in it. Not everyone has to know it, but your best friends understand it on that age in my opinion. Just try it, and if they don't understand you, maybe they're not the bestest friends in the world
By the way: I don't really play with it anymore, just building. Yesterday I bought battle of Alamut, and than you just play a little bit. Fighting with minifigs is always good!