I think there were three pushes behind 3.5, though they may not be equal and I'm not sure what was more important.
First, after the initial sales of the 3.0 core books, book sales tapered off. So some "genius" likely thought the best course of action would be to sell new core books all over again, thinking that would recapture the enthusiasm of the 3.0 release. Unfortunately, 3e as a whole ended up falling into the splat trap as a way of making money, and from what I've been reading of the whole business side of things, that's not where the real money is in D&D. D&D's real money comes from licensing fees and the like. But back in the 3e days, WotC and Hasbro didn't have full control over licenses, partially from TSR selling the rights all over the place as the fell into bankruptcy. 5e's able to avoid the splat trap because Hasbro has stronger control over the IP and its licensing.
Second, there was probably a reaction against the d20 glut from 3pp. There was a lot of trash released under the d20 name, but not all was bad. Some of it may have been cutting into WotC's sales, and some of WotC's 3.0 splats were considered subpar even before 3.5. I wouldn't be surprised if there were corporate lawyers or some such at Hasbro who hated the idea too. But I think the proverbial straw that broke the camel's back here was the infamous Book of Erotic Fantasy. That may have been technically 3rd party, but a number of people who worked on it had been WotC employees, and WotC and Hasbro were probably paranoid about controversy over it. So the whole community standards clause or whatever it was in the updated OGL/SRD/whatever may have necessitated a reissue of core books or some other nonsense.
There was the new miniatures line for D&D. The Chainmail game didn't work out so they went with a new minis game that was more compatible with D&D. And someone perhaps felt the combat rules needed some tweaks to pull it off.
Employee turnover at WotC probably didn't help either. Many of the people who worked on 3.0 moved on, and a number of the people who worked on 3.5 were different. So this might have contributed to project creep.
Looking at some of the material I'm getting a feel for how I want to approach things overall. I'm going to go with 3.5's skills over 3.0. That is more or less an improvement, and there's not a lot of serious change there that bothers me. Some of the class and race changes I'll go with, but other stuff will be dropped. No gnome bards or pokemounts. Ranger will largely be updated, but keep 3.0's d10 HD. Things like that.
I'm divided on weapon sizes and damage reduction. The 3.5 changes here do make some sense, but add complications and breaks with AD&D tradition.
I'm going to ignore things like Sudden and Immediate Actions which WotC seemed to think were terribly important to add to the game but which just look like an unnecessary complication. A lot of the new races and base classes I'll pass on as well. Charopers seem to like those, but I don't see them as filling an important niche in the game. And the base classes make class imbalances worse rather than fixing them, IMO.
Most of the rest of the material tends to be feats, prestige classes, spells, magic items and monsters. I want to keep prestige classes somewhat rare, and they're supposed to be under the control of the DM in the first place. I see the published ones as helpful shortcuts for the DM. Charopers seem to think unlimited access to prestige classes should be a thing, but they're not reading the rules (a traditional sign of munchkinism!). Also, while the earlier prestige classes tended to be a little more archetypical, the later ones tend to have too much flavor baked into them. I'd like to tighten up the prestige classes by going with 5 level rather than 10 level ones, and tweak them to avoid dipping problems. Taking a prestige class should represent some level of commitment on the PC's part, so I want to avoid 1 or 2 level dips, and also restrict things to 1 prestige class per PC. On a more positive note for the player, I'd like to get rid of the really stupid feat and skill taxes as well.
Feats, spells, magic items, monsters and the like can be handled case by case. I can always test things with opponents before letting PCs have unrestricted access anyway.