I used to worry about the shift - but now I don't notice it, or care if I do. It occurred to me that in all the time I spend surfing the web, I have never been distracted by that happening in anybody else's site that I've visited.
It just seemed annoying at the time to see it happening when testing pages I'd spent time on designing and tweaking. My clients have never mentioned it! Some exponents of 'best practice' html might even say that it is more of an annoyance to force an unnecessary behaviour on to an end-user's browser.