Yes, calculations on the Earth's surface are problematic. That explains the lateral offset, but not the varying results at different zoom levels.
When zooming in using the same center point, the sectors should at least remain at the same location (even if it's not accurate due to the sphere).
But I completely agree with you that the important thing is that the caches in a sector remain the same even when zooming, even if the position of the selected sector changes.
And that (I think) is currently the case.