I recently got into Lego again after a 20 year hiatus. And my family (as in my parents and mother-in-law) did raise som eyebrows when I bought my first Creator-set. I do have a problem with my husband now - he got the Lego-bug too, and the money's tight for us. There can be som heated discussions about what what sets to buy as we don't have the same taste in Lego.
But seriously, my grown up friends dont care. They are used to my strange behavior and tastes in hobbys (aikido someone?)OR has a similar taste themself. The children I work with (as a teacher) has a hard time to wrap there head around the fact that I actually buy Lego for myself. My standard answer is "You dont have to stop play because you grow up.". That's a lesson I hope they learn.