You missed the point. How would this hypothetical Oswald on the sixth floor know how they got to the elevator, even if he saw them outside, heard the elevator, and recognized their voices below him? Pretty silly to make up a sighting at the domino room without knowing they went by the domino room.
Not a problem since Oswald hadn't committed to saying which way the two men entered. If the highly-unlikely scenarios that you're grasping at came to light, Oswald could easily amend his "ghost alibi" to say he saw the men while on his way to the second floor, after being in the Domino Room.
Now if Oswald really did see Jarman and Norman go by the lunchroom, why didn't he give specifics?
The first-floor lunchroom was about the only option Oswald had for an alibi. He couldn't say he spent ten minutes on the second-floor buying a Coke or in the main area trying to get change. He's not normally allowed on the third and fourth floors, and would have to know someone not from the warehouse to be at a south-facing window. He couldn't say he was on the fifth floor because he knew people were there. Of course Oswald's not going to say the sixth floor. Maybe the seventh, but then he has even further to go to make it to the second-floor encounter.
Oswald couldn't claim to be out standing on the front entrance or the sidewalk, knowing he wouldn't have been seen.