That's exactly right. The condition of the scope at the moment Oswald fired the shots can't be known for the reasons you note. I seem to recall that the scope was also removed to check the rifle for prints before it was test fired.
We know enough to make a few logical assumptions:
1) According to the DPD and the photos of their discovery of the rifle, it was carefully laid between boxes and not dropped from a distance that would have damaged and grossly misaligned the scope.
2) A sharpshooter would always "sight-in" a scope on the rifle he intended to assassinate the POTUS with.
3) The FBI said the scope was grossly misaligned and required 3 shims before it could be used.
4) There were no shims on the rifle when Oswald supposedly used it.
When a scope requires 3 shims to zero, it is either bent or has never been sighted-in. The DPD were careful not to damage the scope when they laid it down between the boxes because they wanted it to match the backyard photos exactly. That's why the misaligned scope was on the rifle in the first place. Same reason they left the strap on the rifle.
Since 4 contradicts 2, the MC was never sighted-in and LHO did not use it to assassinate JFK. QED