No, he was watching them switch the train cars.
Did Frazier say he didn't look at Oswald again until he arrived at the door?
No, he watched Oswald enough to realize that he was getting further and further away.
And watching the trains doesn't rule out a Frazier head-angle that could be at a shallow angle off-plumb, with Oswald well-within his peripheral vision. The corner of his eye, and all that.
Someone intent on ditching that package is hardly going to do it in front of a person walking only 50' (at best) behind him.