The man shooting a rifle out the window was described as wearing a light-colored shirt. Oswald could have been wearing only the white t-shirt during the shooting, with the brown arrest shirt laying on a nearby box. After the shooting, while making his way across the sixth floor towards the stairs, Oswald uses the brown arrest shirt to wipe down the rifle in an attempt to clear it of prints.
This theory doesn't hold up. The rifle had prints on the trigger guard (the first place Oswald would have wiped) which were not smeared.
Your post assumes two things.
One, Oswald would be sure to wipe the trigger guard.
Two, Oswald, if he tried to wipe the trigger guard, did it successfully.
Personally, I think it's as simple as the wooden stock of the rifle was too rough to hold prints.