For me growing up the downside of it was that as a kid you don't want to stand out. You don't want to have a famous father let alone get a job because of your famous father you know? But I'm a product of nepotism. That's how I got my foot in the door through my dad.
There are a lot of downsides to being male. We age faster and die younger. But give us this: we're lifetime baby-making machines. Women's reproductive abilities start to wane when they're as young as 35. Men? We're good to go pretty much till we're dead.