If you look carefully at the floor of the dovetail on both sides of your existing sight you should be able to determine from which side it was installed.There will be slide or worn marks on one side and nothing on the other side. That will always tell you from which side your sight was originally installed. You will usually need a good magnifying glass to spot it.
I do not think the COLT brand pistol slides have a tapered dovetail but, they usually come out easier if you knock them out from the opposite side that they originally went in.
But, if THAT (above info) does not help you..............WITH THE REAR OF THE SLIDE FACING YOU..........The COLT rear sights were factory installed from the LEFT to the RIGHT. SO....With the rear of the slide facing you...you would drift it out from the right side to the left.
My REM Slide was Right to Left. (it features standard Colt factory dovetail)
My PARA 14/45 was Right to Left too, but the Para 13/45 was Left to Right.
My Colt Commander (which still has the factory installed sights) was Left To Right as was my antique Colt.
BUT, I just now checked a Series 70 with my magnifying glass (also still with original factory sights) and it is Right To Left.