I don't know if high society is different in other cities but in Hollywood important people can't stand to be invited someplace that isn't full of other important people. They don't mind a few unfamous people being present because they make good listeners.
You can become famous but you can't become unfamous. You can become infamous but not unfamous.