You realize you would also have to get all the caster fans to agree with this? That works for your own table, not so much for anything beyond that, at least as you state it.
Ultimately, balance is the reasonable compromise position. All classes balance, martial or magical. It's fair.
Don't think of it as getting caster fans - who, yes, enjoy playing far more powerful classes alongside other players who chose trap classes, and have no reason to approve any change for any reason - to agree to a game that treats everyone fairly instead of giving them a privilege (or or hard-earned reward, in the case CharOP system mastery, rather than just happening to want to play Gandalf or Harry Potter). Think of it as inviting them to empathize with other players and consider the benefit of playing fairly in a balanced game.
But that requires more than presenting the balanced alternative, a compromise between an extreme position and a reasonable one is still likely to be unreasonable.
So, consider the purely hypothetical alternative...
D&D, in a fantasy setting, there are dragons and other magical monsters (including once-mortals who have crossed a line, like a lich or a warlock who has surrendered soul & free will to become a magic-wielding tool of some devil/fey/GOO/whatever), there are gods and other supernatural powers above mortals, and there are magical items placed in the world by such beings to tempt, bedevil, aid, or just mess with mortals. All PCs are mere mortals. Martial characters, much as they are or were in some preferred edition. Casters, OTOH, like 1e magic-users, must devote themselves to the study of the arcane/divine, from early childhood into middle age to even have a chance of gaining magical powers, lifetimes of study, conteplation, and/or prayer, with no time or effort spared to become proficient with mundane armor, weapons, or tools (caligraphy or something like that, sure), nor physical or social skills of any kind, in return, the learn the magical secrets available to mortals - knowledge of magic & suprenatural beings, including rituals that offer some protection from them or invoke them for aid (rarely answered) or bargaining (tho mortals have little to offer but their souls or servitude, which means no longer a PC), and how to use appropriate magic items. When such aspirants become frustrated with their lack of progress, they may well go out adventuring in the hopes of finding a magic item that may grant them the kinds of powers they seek. The ability to use such items also being a benefit of the class.
Imagine that's what D&D has been for, like, 48 of the last 50 years. How would fixing the caster/martial gap in that hypothetical scenario sound?