If These Walls Could Talk

Meet the Families of the Rogers Mansion

To commemorate the 375th Anniversary of the Founding of Southampton in 2015, the Southampton History Museum curated an exhibit exploring the history of Southampton through the families who built and lived in the Rogers Mansion. In 1648, the Rogers family was the first to settle on the property, establishing a farm, and the first of three influential Southampton families who would follow. A whaling captain, a country doctor, and a retired lawyer-turned-philanthropist each made it their proud residence and when its days as a private home were over, the Mansion was sold to Southampton Village and used as headquarters for the YMCA, the Red Cross, a community center and presently the Southampton History Museum. This exhibit highlighted the families who lived here and the role the Mansion has played throughout Southampton’s long history. If These Walls Could Talk was originally on display at the Rogers Mansion from March 7, 2015 to December 31, 2015.


Images of the Exhibit when it was on display in the Rogers Mansion


Early Southampton

A Land and Sea of Plenty

Imagine a fertile land with rolling hills surrounded by the sea, a wilderness without our villages, houses and cars that crowd this narrow stretch of land today. Native Americans roamed and thrived on the abundant fish and shellfish harvested from Southampton’s waters and the wildlife that filled the woods. Southampton had this quiet existence for thousands of years, until 1640, when everything changed.


Left: John White - Praying Around the Fire with Rattles, 1590

Right: Theodor de Bry - The Manner of Making Their Boats, 1588


In June of 1640, the sloop carrying the original 10 English-born settlers from Lynn, Massachusetts, entered Peconic Bay and anchored in the harbor today known as North Sea under the high sand dune we know as Homes Hill. Tradition tells us that when the one woman stepped ashore, she exclaimed, “For conscience sake, we’re on dry land!” This place has ever since been called Conscience Point. The settlers most likely followed the Indian trail that became North Sea Road and settled near the shores of today’s Old Town Pond. Their first homes were where Southampton Hospital stands today.


Conscience Point Memorial by Jeff Heatley


These first homes were far simpler than the glamorous mansions that surround us today. They were in fact pits in the ground, encased with timber and lined with bark. With helpful advice from the Shinnecock, they quickly planted hay, oats and corn in hopes that they could survive the first winter. Though wary of each other, the native Shinnecocks and the settlers created a relationship with benefits for both. The settlers, with their abundance of firearms, provided protection for the Shinnecock while the Shinnecock permitted the settlers to purchase their land and taught them the secrets of Southampton’s land and sea. The first winter was extremely difficult for the settlers but the population continued to increase and before the New Year the original ten settlers had grown to 150.

1650 Rogers Mansion A New Land, A New Life, A New Story Begins…

The Town of Southampton was first settled in 1640 when settlers from Lynn, Massachusetts came down to Long Island and obtained land from the local Shinnecock Indian Nation. The first group of settlers arrived at Conscience Point and they included eight men, one woman and one boy. They established the first settlement on the road that is today known as Old Town Road in Southampton. Soon after the original European settlers arrived, more came to join them. This included a man named William Rogers who arrived in 1642. It is at this pivotal point in the history of the New World that the story of the Rogers Mansion begins.


By 1648, the settlers felt they would be better off on land further east and moved the village to its permanent location a mere half mile away. Southampton’s Main Street was divided into forty lots and William Rogers received the lot on which we stand today. As the Southampton colony continued to grow at its new site the community remained virtually self-governed but maintained a close tie with New England, just a few hours’ sail away. A census report, taken in Southampton in 1686, lists the total number of inhabitants, men, women, children, servants, transient persons and slaves at 787. While life was easier by 1686 than it was in the earliest days of the settlement, survival still depended on everyone assuming a share of the work. The main industry was agriculture, supplemented by occasional whaling and fishing.

Close up view of a section from Willaim S. Pelletreau's 1878 map of Main Street, Southampton showing the Rogers Property

In 1655, William Rogers’ oldest son, Obadiah, was given the Rogers homestead in Southampton after his marriage to Mary Russell. During their long marriage they had seven children who were born in this house. At this point, Southampton remained the domain of the original undertakers, while newcomers were carefully vetted before they were allowed to purchase land and settle.

Unfortunately, no images of the earliest structures erected on the Rogers property exist today. But we can assume that during the Colonial chapter of Southampton’s history, when self-sufficiency was imperative to everyday life and architecture was the servant of Puritan practicality, there would have been few aesthetic refinements. Colonial houses in the 1650’s were rustic structures with small rooms. Even the most affluent colonists were still trying to create homes from what was available in the wilderness. There was little time for frivolity in their lives and little room in their homes for frivolous things. At this time form followed function, not beauty.

​Images from the exhibit showing an imagined 1650s Rogers Homestead


1850 Rogers Homestead Southampton Grows and Prospers…

Two hundred years after the move from Old Town, profits from a booming whaling economy brought prosperous times to Southampton in the pre-Civil War era. It was a brief but heady chapter in the life of the village when the worldly class of captains and their backers spent lavishly on handsome homes that were architectural reflections of their status.

Sag Harbor became one of the busiest ports in the U.S. in the mid 1800's due to the American Whaling Industry which many Southampton residents, like the Rogers, were involved in

Captain Albert Rogers was born on January 10th, 1807 at the Rogers’ Southampton home. At the age of 20 he became the 6th generation of the Rogers Family to own and live on this very land. Mary Halsey became Albert’s first wife around 1828 but just about six years later, at the young age of 25, Mary died from unknown causes and without children. Two years after Mary’s death Captain Rogers married her older sister Cordelia Halsey. Born only five days apart in Southampton, they likely had known each other all their lives. Cordelia and Albert had four children together, presumably all born in the Rogers Mansion.

Portrait of Captain Albert Rogers

In 1843, Captain Albert Rogers was wealthy enough to build a house worthy of the successful whaling captain he was. He chose the popular Greek-Revival style for the Rogers Mansion, testimony to his refined tastes and local prestige. In response to the gold rush of 1849 many whaling captains from Southampton joined together to form the “Southampton California Mining and Trading Company.” Albert Rogers became a member and joined George White, Pyrrhus Concer and many other gold-hungry 49ers who ventured to California at this time. During his travels hunting for both gold and whales he brou