Ultimately, it comes down to 3 things:
There is no inherent benefit, to the player or society at large, to making Charisma explicitly based on appearance.
There are inherent negatives to society at large by linking Charisma and appearance, however minor.
There is nothing stopping any individual...