Dougherty takes the west elevator to the sixth after Jarman and Norman take it to the fifth. They closed the gates allowing it to be called by Dougherty. He did not close the gates as he presumably was collecting books on the sixth then the fifth. Truly was unable to call the west elevator because Dougherty left the gates open.
A possible scenario:
Jarman and Norman take the west elevator to the 5th. They don't see Dougherty because he's still down in the Domino room having his lunch.
Williams hears Norman and Jarman down on the 5th and goes down to join them. He doesn't see Dougherty who is still on the first floor
Dougherty finishes his lunch and calls the west elevator down and takes it to the 6th while Norman, Jarman and Williams are at the south-east corner.
Dougherty finishes what he's doing on the 6th and goes down to the fifth to carry on working.
Shortly after arriving on the fifth he hears a shot coming from the floor above. He is stood near the north-west corner of the 5th floor, Williams and co. are in the south-west corner.
After Truly has seen the elevators on the 5th and before Williams and co. run across the floor Dougherty gets in the west elevator and takes it down to the first.
Problems with this scenario:
Oswald states he is in the Domino room when Norman and Jarman enter the building. He doesn't see Dougherty there and Dougherty doesn't see him.
While Williams is on the 6th, then Dougherty, the assassin has to construct the SN (and assemble the rifle?)
Rowland sees the man with the rifle on the 6th way before Norman and Jarman go up to the 5th meaning he is there at the same time as Williams (and while Oswald is still in the Domino room).
Williams is insistent no-one was up there for the 20+ minutes he was up there.