Before the site where Irvington is now was was settled by Europeans, it was inhabited by the Wickquasgeck Indians, one of the Lenape (Delaware) tribes which dominated lower New York state and New Jersey. The land was originally part of the Bisightick tract of the Van der Donck grant purchased by Frederick Philipse in 1682, but in 1785 the state of New York confiscated the land from his grandson, Frederick Philipse III, after he sided with the British in the Revolution, and sold it to local farmers. This is presumably how part of it came to be the farm of William Dutcher. Dutcher sold half of his farm to Justus Dearman in 1817, who then sold it to Gustavo F. Sacchi in 1848 for $26,000. Saachi sold the parcel to John Jay that same year, and Jay laid it out as a village which he called "Dearman", and sold lots at auction in New York City starting on April 25, 1850. At that time, a ferry point ran across the Hudson from Dearman to Piermont on the west bank, and the hamlet of "Abbotsford" (which would later become Ardsley-on-Hudson) was forming along Clinton Avenue.

In 1854 the village changed its name, by popular vote, to "Irvington", after the American author Washington Irving, who was still alive at that time and living in nearby "Sunnyside" – which is today preserved as a museum. Influential residents of the village prevailed upon the Hudson River Railroad to change the name of the train station to "Irvington", and also convinced the Postmaster to change the name of the local post office as well. It was thus under the name of "Irvington" that the village incorporated on April 16, 1872.

The side streets off of Irvington's Main Street, which originally were designated "A", "B", "C" and so forth, are today named after many of the area's early settlers, such as Barent and William Dutcher, Captain John Buckhout (who lived to 103) and Wolfert Ecker (or "Acker"). It was Ecker's house, then owned by Jacob van Tassel, which was burned by the British in the Revolutionary War because it had become a notorious hang-out for American patriots. Washington Irving later wrote about it under the name of "Wolfert's Roost", and purchased and re-modeled the house to become "Sunnyside". Another early settler was Capt. Jan Harnse, and the Harnse-Conklin-Odell Tavern on Broadway was built in 1693. (see below Brtish troops camped near Odell's Tavern during the Revolution.