The first set I distinctly remember REALLY wanting, yet knowing I would never have, was 10185 - Green Grocer. I was in grad school at the time, and visited the newly opened Lego store in my mall, and just couldn't believe how awesome Lego had gotten since my childhood. It was at a point in my life where it was a choice between buying the set or buying groceries for the next month, so... sadly I passed and it retired before I ever had income to even consider getting it. I've been slowly Bricklinking it, though, so one day my dream will come to fruition, even if slightly modified.
As as a kid, I think my parents must have hidden the catalogs from us, because I don't remember ever wanting specific sets. But between the three of us (all KFOLs turned AFOLs now in our 30s), we always got most of a wave between birthdays and Christmas, and my mom was big into auctions (before people knew Lego was valuable) and regularly supplemented our stash with cheap buys of bulk loose pieces. So yes, I was Lego previliged as a child.