Born in 1945 in Brooklyn, Paterson's father Basil was a labor lawyer, and later a state senator, secretary of state, and deputy mayor of New York City. Paterson when was three, he suffered an ear infection that spread to his optic nerve and left him with little to no vision in either of his eyes. His family moved to Long Island so that he could attend a mainstream school, and he was the first disabled student to graduate from Hempstead High School in 1971.