Once away from the TSBD and where people knew him it wasn't so much that Oswald was invisible but unremarkable, more likely people saw but took not notice and therefore didn't remember him.
However he should have been noticed by the TSBD staff when leaving the building.
Numerous witnesses came forward after LHO's picture was plastered all over t.v. and in the newspaper, but no one from the 7 blocks area could be found. Is this just another coincidence? I doubt it.
It was best not to find anyone since it was just a made up story anyway.