A Black history of the Highlands
History is a prism that can be viewed from many angles. The white wealthy men who forged the nation took one view, and Blacks, Native Americans, women and the impoverished took others. Sometimes the colors produced by the prism cross and blend. But any honest view of history must turn the prism now and again.
The 1619 Project, which won a Pulitzer Prize for The New York Times, turned the prism, and the reaction was swift, with accusations that looking at our shared history in ways other than as it was written by white men in the 19th century was teaching white children to bear the guilt of past crimes.
Yet even before The 1619 Project, many Hudson Valley institutions had begun to reexamine the contributions of Black people, most of whom were enslaved during the 200 years before the institution was abolished in New York in 1827. In February, the John Jay Homestead in Katonah hosted a program highlighting the fact that the future chief justice enslaved people for more than 40 years after he first called for New York to ban it. In 2020, Boscobel began a project to examine the people enslaved by States Dyckman, who built the early 19th-century home that was relocated in 1956 from Montrose to Garrison.
In 2019, Historic Hudson Valley, based in Westchester County, produced a documentary, People Not Property, that acknowledged the earliest Black residents of the Hudson Valley and named their enslavers, including the owners of Van Cortlandt Manor in Croton and Philipsburg Manor in Sleepy Hollow.
“Slavery was the brutal foundation upon which the entire United States, north and south, was built,” it stated.
Philipsburg Manor, which Historic Hudson Valley has preserved, was constructed by Africans enslaved by a Dutch merchant, Frederick Philipse (1626-1702), whose property covered about a quarter of modern-day Westchester. He and his son would become major slave traders; the family is honored today by the name Philipstown. According to Columbia University, before a Philipse descendant donated three boxes of family papers to the university in 1930, she removed nearly every document related to its human trafficking.
A.J. Williams-Myers, a longtime professor at SUNY New Paltz who died in July, noted that Black people are almost nonexistent in standard local histories such as the General History of Dutchess County, published in 1877, or The History of Putnam County (1886), “other than to appear as a statistic when counting material possessions.”
There are many examples of this oversight. The late Robert Murphy, who for 38 years edited the Beacon Historical Society newsletter, said that the best single source he found for Black history from the 19th century was the Fishkill Standard, although “a careful reading of the paper leaves only the briefest of sketches of what life was like for the Black men and women of that time.”
The exclusion is reflected in recent artwork by Jean-Marc Superville Sovak, who has a studio in Beacon. In a series of prints, a-Historical Landscape, he took idyllic 19th-century landscape engravings typical of the Hudson River School and inserted images from anti-slavery almanacs and abolitionist tracts of the same period. “What makes these works so American, I think, is not what is depicted but also what’s missing,” he says.
The relative lack of archival resources is a loss, says Michael Groth, author of Slavery and Freedom in the Mid-Hudson Valley, because “there’s nothing more fascinating in our history than the central paradox of slavery and freedom — how can a nation so committed to the ideals of democracy and equality also become the largest slaveholding society in the Americas? And the freedom struggle is as dramatic and powerful a story as any that can be told.”
This series is designed to be an introduction to the Black history of the Highlands. Its title is borrowed from social historian Erica Armstrong Dunbar, who was referring to Black people during the time of the Revolutionary War, when Americans fought for white freedom: “They were always present, but never seen.”
The first Black people brought to the Hudson Valley came against their will, enslaved by the Dutch, who had arrived in 1624 to establish what is now New York City. Investors organized as the Dutch West India Company faced an immediate labor shortage because few whites were desperate enough to emigrate to work in isolation on rented land. If anything, they aspired to be merchants.
The company’s initial shipment of humans to New Netherland, in 1625 or 1626, included about a dozen men and their wives; the men built public works projects, cut timber and burned lime, while the women were assigned to various company officials as domestic servants. Slavery would flourish in New York for another two centuries. It was never as integral to the economy as in the American South or West Indies, but many historians believe that, without African labor, the white aristocracy here could not have existed. The enslaved enabled the wealth of families whose names today are memorialized on maps and street signs in Philipstown and Beacon.
The vast majority of African slaves kidnapped and sold to New York owners were Kongo, Coromantin, Paw Paw and Malagasy, typically brought in small shipments and often offered as payment for debts. They usually were first taken to the West Indies for “seasoning” on sugar plantations; owners in New York complained that Black people shipped directly from Africa too often rebelled against their loss of freedom and the brutal conditions.
When white settlers were convicted of crimes, they were sometimes sentenced to “work with the Negroes,” a suggestion of the hard labor that Africans were compelled to do, observes historian Andrea Mosterman. Enslaved people were given the dirtiest work, such as clearing the streets of animal carcasses and flogging and executing criminals. One of the earliest slaves in Hudson Valley records — from 1646 — was the hangman at the Dutch settlement that is now Albany.
Even after 1640, when more white immigrants were available for hire, the Dutch preferred slaves. One historian estimated that the break-even for buying an enslaved person versus hiring a freeman was about a year. At the same time, many farmers found it more economical to rent, rather than own, enslaved people because they didn’t have to feed or house them during the winter.
The Philipse family
Members of the Philipse family were among the most active slave traders in the Hudson Valley. Their DNA is so much a part of the region that property deeds executed in Philipstown still grant mineral rights to their descendants.
The patriarch, Frederick Philipse, born in 1626, came to the colonies as a young man to work as a carpenter for the Dutch West India Company. Within about 20 years, he was one of the richest men in New Netherland — and that was before he became a slave trader. (In 1977, a hagiography in The New York Times portrayed his life as the classic “rags-to-riches” tale of “the immigrant son of a roofer.”)
In 1664, the British took over New Netherland from the Dutch. The Philipse family stayed in the good graces of the new regime, becoming friendly with the governor, who granted Frederick wide swaths of land that would become Westchester County. In 1666, Frederick married a widow who had been left a fortune in merchant ships and Manhattan property. He began trading with suppliers in the West Indies, England, Holland and Portugal and experimented with the slave trade by providing “servants” to other wealthy New Yorkers.
Crossing the Atlantic
Michael Lord is the director of content for Historic Hudson Valley. These comments were taken from the documentary People Not Property.
In 1685, a Phillipse-owned ship named the Charles sailed from New York City. It sailed along the coast of West Africa, picking up grain and fresh water and eventually settling at the port of M’Pinda Soyo, which is on the mouth of the Congo River. The Congo kingdom was in the midst of a centuries-long civil war, and the losers of battles ended up being the ones sold as African captives onto these ships.
The captain of the Charles was able to negotiate for the purchase of 146 Congo men, women and children. The Charles was a type of ship known as a pink. It was about 100 feet long and would have had two cargo holds, one for food and water, and one for human cargo. The cargo hold on a ship the size of the Charles was not much more than 20 feet by 20 feet, a little larger than a living room. These men would be stripped naked, they would have been shackled together at the arms and at the ankles. They would have been laid down on their backs on the floor of this cargo hold.
If that floor cannot fit 100 people, they would have built a second half-deck two-and-a-half feet off the floor. They would have laid another group of men. The journey across the Atlantic Ocean in 1685 could take as long as 12 weeks. Those men in that cargo hold would spend weeks below deck. They would need to remain down there until they were out of sight of land because the threat of these individuals who see land jumping ship and trying to swim back was too great.
In bad weather, individuals remain below deck. There’s no ventilation. You’re in equatorial heat. There are no facilities. People died every night and were tossed overboard; there was a line of sharks behind these boats. The women remained above deck. They were at the discretion of this all-male crew.
Of the 146 men, women and children who started on this voyage, 105 survived. They pulled into Barbados, where they were quickly purchased to work on sugar cane plantations. The remainder — too ill, too weak to be of any value to the sugar cane planter — those 23 men, women and children set sail from Barbados to New York City. It was a relatively easy journey. Fourteen more died.
By the time they disembarked in Rye, there were nine that survived. One was a small boy with one eye. He was led to Manhattan. The other eight were taken by Frederick Phillipse’s son, Adolph, and marched across Westchester, where they were set to work on building what is now called Phillipsburg Manor. These eight Congo survivors, some of the first Africans in Westchester County, built a manor house, a mill, the barn and the old Dutch church in what is now called Sleepy Hollow.
It’s unimaginable to try to comprehend the size and the scope of this transatlantic trade without breaking it down to these individual voyages and understanding the nightmare that took place on each and every one of them.
The food that made the manorial system in the Hudson Valley work was grain. Europe and the West Indies needed flour, and traders built sawmills and gristmills and claimed monopolies over access to the rivers and streams. The Philipse family built two mills: one near Tarrytown (Upper Mills) and the other where City Hall now stands in Yonkers (Lower Mills).
In 1685, Frederick sent one of the family ships, the Charles, to Soyo, a city on the west coast of Africa in what was then the kingdom of Kongo (present-day Angola and Republic of the Congo). There, the crew loaded 146 captured men, women and children for a journey to Barbados for seasoning. When the Charles arrived, however, only 105 had survived.
The 23 sickest were deemed unfit for sale and shipped to New York City as “refuse cargo.” By the time they arrived at Wall Street, only nine were alive. Frederick’s son, Adolph, then 19, sent eight to the Upper Mills and the ninth, a boy with one eye, to the Philipse mansion in Manhattan.
These enslaved workers came from an agricultural society in Kongo, where residents grew manioc, yams, taro, groundnuts, maize and palm and fruit trees using hoes, writes Dennis Maika, a historian with the New Netherland Institute, based in Pleasantville. Men cleared the fields, tended the trees, fished and hunted, and the women did the daily field work. The Kongolese had experience with other cultures because of their proximity to the Kongo River, which drew traders and navigators. They spoke the Bantu language, as did many ethnic groups.
When it came to Black labor, Frederick Philipse was on the hunt for bargains. The slave market on the African mainland was controlled by the Royal African Co., which had a license from the British government, so he shifted his sights to Madagascar, using agents and pirates to work around the royal monopoly.
In 1691, Frederick heard from an Englishman who lived on the island and offered him slaves for 30 shillings each. Frederick dispatched a ship loaded with shoes, stockings, shirts, hats, tools, rum, Madeira wine, cannon powder and books to trade. But instead of 200 slaves, representing £4,000 in profit (about $1.2 million today), as he expected, the ship returned with 34 slaves, 15 head of cattle and 57 bars of iron.
Frederick was unhappy. “Negroes in these times will fetch £30 and upward the head, unless they be children or superannuated,” he complained. “It is by Negroes that I find my chiefest profit.” He noted that of the 34 slaves, 15 were children and three still suckling. Frederick also had lost one of his enslaved workers who went on the voyage as his representative but decided not to return.
Frederick continued dispatching the family ships but they returned with only a few slaves. For one trip, made by the Margaret in 1698, he tried incentives: The captain, Samuel Burgess, was promised a commission for each enslaved worker he brought back; the first mate and physician were each allowed to transport one slave free; and the doctor earned 12 pence for each captive who arrived alive.
After reaching Madagascar, the Margaret’s crew filled the hold with 114 captured African men and departed for New York. The ship and slaves never arrived, however, as the ship was seized by British authorities on allegations the crew had worked with pirates. Burgess was forced to sell the enslaved men at the Cape of Good Hope; he was sent to England and imprisoned.
The following August, Adolph Philipse sailed to London to recover the ship and its cargo — minus the humans. He managed to free Burgess. But while he was overseas, his father died in New York.
Most of Frederick Philipse’s land, livestock and enslaved workers went to Adolph and to Frederick II, the son of Frederick’s late brother. In the years that followed, Adolph assigned an overseer to the Upper Mills. By the 1720s, he had slave quarters built about 90 feet from the main house; at that point, the mills were likely being operated entirely by enslaved workers, Maika writes.
By 1741, Frederick II was a respected merchant in New York City, where 20 percent of the population was enslaved. That year, about 150 enslaved and free Blacks and whites were accused of insurrection; Frederick II was one of the two judges assigned to hear the case.
The plot had been discovered, according to testimony, when conspirators were overheard discussing it at the Philipse mansion in Manhattan. Thirteen Blacks were burned at the stake, 16 Blacks and four whites were hanged and 70 Blacks and seven whites were banished.
Among those burned alive was Cuffee, a slave owned by Adolph. Cuffee lived in one of the family’s Manhattan homes; was bilingual in English and Spanish; and had liberty to travel around the city. He allegedly confessed to setting fire to a warehouse owned by Frederick II, who — in the role of both victim and judge — found him guilty and ranted about the “monstrous ingratitude of this Black tribe.”
(The only member of the Philipse family who freed enslaved workers was the second wife and widow of the patriarch Frederick, and it was an empty gesture because she made no provision in her will for a £200 security that was required for each freed slave.)
When Adolph Philipse died in 1750, an inventory of his estate listed 23 enslaved workers at the Upper Mills, including eight children. Their names suggest families that included several generations, Maika says.
Frederick II inherited his uncle’s property. Within months, he began selling the enslaved workers, through newspaper ads, in part. By the time of his own death the following year at age 53, he had dispatched 21 men, women and children, almost certainly breaking up families in the process.
His son, Frederick III, continued the sale, except for a boy named Charles, whom he gifted to his widowed mother. Frederick III likely finally destroyed a Kongolese culture that had arrived at the mills in the 1680s and been preserved there, says Maika.
The British arrive
The British forced the Dutch out of New Netherland in 1664 and took over and expanded its slave system.
The Dutch had an “institutional ambiguity” toward slavery, says Maika. Enslaved people could hire themselves out, raise their own food, own moveable property, sue and be sued and marry and raise children. But historian Jeroen Dewulf cautions against the impression that Dutch slavery was benign. He cites the response of a slave who was told he was probably happier than a poor person because his master fed him well: “That may be true, sir, but put bird in cage, give him plenty to eat, still he fly away.”
The British transformed New York from a “society with slaves” to a “slave society,” notes historian Ira Berlin. They passed progressively restrictive laws, such as prohibiting any gathering of three or more slaves under punishment of 40 lashes. (Each town was instructed to appoint a whipper.) The slave code was more severe than in other northern colonies but not as severe as laws in the South. Owners could not harm or kill an enslaved person (although, legally, a slave could not be raped by a white person), were required to provide “adequate” food and clothing and were fiscally responsible for the infirm and elderly.
At the same time, historian David Korbin writes, “since the Negro was considered to be property, if the slave was executed the owner would usually be compensated by a levy charged to all the slaveholders of the county involved.” When large numbers of slaves were executed, such as after a 1712 insurrection, a special assessment was made.
In 1717, the New York Legislature allowed owners to free their slaves, but only if they paid a £200 deposit plus £20 annually for maintenance. The goal, wrote Williams-Myers, was to dangle the idea of freedom to maintain a “hearty, obedient, docile but dependable labor force.”
To purchase humans, Hudson Valley landowners typically hired agents to bid at Wall Street auctions. In 1721, for example, Cadwallader Colden wrote from his estate in Newburgh to place an order for two males of about 18 years old and a girl of about 13 years that his wife requested “to keep the children and to sow.”
It is not clear when the first Black people arrived in what is now Beacon, but they were almost certainly enslaved, and they may have been among the four slaves listed in the 1714 census of Fishkill Landing. (Beacon was created in 1913 from Fishkill Landing and Matteawan.) Three were owned by Roger and Catherine Brett, a daughter of Francis Rombout, who 30 years earlier had “purchased” 85,000 acres from the Wappinger tribe.
Did Every White Person Own Slaves?
When Thomas Davenport, who in 1730 was the first settler of what is today Cold Spring, made his will in 1746, he left “my Negro Jack” to his widow. He instructed that if she remarried, or at her death, Jack was to be sold with the other goods and chattel and the proceeds divided among his children.
His son, Thomas Davenport, who lived in a log cabin near Indian Brook, owned four slaves in 1790, according to the federal census, more than any other Philipstown resident. In his 1797 will, Thomas freed an enslaved worker named Dob but made no mention of others. In the 1800 census, his son William had two slaves.
That even the pioneers of what would become a tiny village enslaved people might make you wonder who didn’t. When three activists, for a grassroots campaign they call Slavers of NY, in 2020 began placing stickers on street and subway signs and in neighborhoods in New York City named for prominent men who were slaveholders, someone asked that question in an addendum:
In response, the activists acknowledged that, while many New Yorkers did not enslave people, “they also did not necessarily work toward abolition.” While prominent men such as Alexander Hamilton, John Jay, Thomas Jefferson and James Monroe expressed abolitionist views or personal moral conflicts with slavery, they did not feel compelled to free anyone they had enslaved.
Michael Groth, a history professor at Wells College and author of Slavery and Freedom in the Mid-Hudson Valley, says that many of his students are surprised to learn that, at the time of the Civil War, three-quarters of white Southern households did not include enslaved people. Of those that did, most held fewer than 10 — far from the vast plantations that Gone with the Wind placed in the popular imagination. This was also true in the North, where in 1790 the average owner enslaved three people. The exceptions were a few family manors.
Whatever the numbers, more important is that New York state, like the South, became what historian Andrea Mosterman, author of Spaces of Enslavement, describes as a “full-fledged slave society.” Over two centuries, the institution infiltrated all aspects of daily life, and every white resident benefited from an economy boosted by enslaved workers.
In 1709, the couple had built a mill. After her husband drowned in 1718, Catherine continued to oversee it. Before her own death in 1764, Madam Brett — whose home is on Van Nydeck Avenue — left her “Negro wench” Molly to her eldest son, ordered that Old Negro Sam be cared for and allowed a slave named Coban to choose her own master.
In 1790 — nearly a decade after the American Revolution — Henry Schenk, who married a Brett granddaughter, owned eight people. Other area slaveholders included Isaac Adriance, who was a partner in a storehouse at the mouth of Fishkill Creek; the Brinckerhoffs, who owned 30 slaves; and Philip Verplanck, whose 12 slaves tended to his mill, store, storehouse and farm.
The early Dutch settlers had planned to entice white immigrants to clear and farm Dutchess County — and pay rent. But whites who had fled aristocratic Europe preferred to own their own property. The first two settlers and their families had been alone for 25 years after their arrival in the 1680s, although a sloop came up the river occasionally. By 1714, when Dutchess was deemed to have enough people to govern itself (rather than being subservient to Ulster), it had a population of 447, including 29 enslaved workers. Nearly everyone lived in one of three settlements along the river — Fishkill, Poughkeepsie or Rhinebeck — each separated by a day’s ride.
After 1714, the population of Dutchess doubled each decade until 1750, after which it had doubled again by 1771 and again by 1790. The 418 white residents grew to 42,980 and the enslaved to 1,856, including 600 in Fishkill.
Dutchess farmers enslaved relatively few workers — the average in 1790 was 2.8 per household — so their slaves had to be, by necessity, skilled in many trades. The phrase “understands all kinds of farm work” was common in advertisements. The skills might include plowing, planting, harvesting, felling trees, clearing fields, tending livestock, building barns, repairing fences, fixing tools, carting produce to market, tending orchards and processing grain, notes Groth, a history professor at Wells College in the Finger Lakes.
The most enduring testament to the work of enslaved people in the Hudson Valley may be “the agricultural contours of the counties” marked by cleared fields and stone walls, says Peter Bunten, chair of the Mid-Hudson Antislavery History Project.
A 19th-century historian, Mary Humphreys, described how, “for every department of the household, there was a slave allotted. They hoed, drilled, shod horses, made cider, raised hemp and tobacco, looked after the horses and the garden, made and mended the shoes, spun, wove, made nets, canoes, attended to fishing, carpentering, each household sufficient unto itself.” Enslaved women worked mostly as domestics but might have learned skills such as dairying, milking and butter and cheese making. John Dumont of Ulster County, who enslaved Isabella (later known as Sojourner Truth) boasted that she could “do a good family’s washing in the night, and be ready in the morning to go into the field.”
In wealthier homes, slaves might be charged to start fires in fireplaces at stated hours. The men were waiters, butlers, craftsmen and coachmen, such as those who handled the reins when Madam Brett made the rounds of Fishkill Landing. When Benjamin Franklin was leaving the Schuyler mansion in Albany in May 1776, headed for New York City, Mrs. Schuyler insisted that her slave Lewis drive him. Once he arrived, Franklin wrote Philip Schuyler to thank him because “part of the Road [Old Albany Post Road] being very stony and much gullied, where I should probably have overset and broken my own Bones; all the Skill and Dexterity of Lewis being no more than sufficient.”
Slaves and indentured servants took care of collective white obligations to maintain that road, also known as the King’s Highway. And during the second half of the 18th century, an enslaved man named Quam captained a rowboat and a pirogue (a two-masted vessel without a jib) that served as a ferry connection between Fishkill Landing and Newburgh.
At the manors owned by the Philipse family, enslaved workers likely created gardens and perhaps found the sweet potatoes similar to the manioc and yams of their homeland, says Maika. They used their knowledge of herbalism for medicines. Enslaved people built the mills, the dams and the mansions.
Before historians began in the past few decades to take a second look at the lives of enslaved Black people in the Hudson Valley, their predecessors generally adopted one of two approaches: They ignored Black people except for noting the names of slaves mentioned in wills, or they reported that, despite the trauma of being held captive, slaves were happy-go-lucky and better off than they would have been in Africa.
As an example of this revisionism, Helen Wilkinson Reynolds, a longtime and respected member of the Dutchess County Historical Society, in 1941 attempted a history of Black people enslaved in the county.
First, she noted that because most farmers could afford only one or two slaves, their property typically lived in the cellar or attic. That was certainly uncomfortable, but “there was probably no intention of cruelty,” Reynolds mused, “for the knowledge of hygiene and sanitation was limited.”
As for runaways, Reynolds wrote, “while in some cases it may have been prompted by harsh treatment of the slave by the white owners, in others it was presumably due to wanderlust, a desire to get abroad and see something new, to have amusement and to do as he or she pleased.”
She then turned to “happier parts of the story,” such as the Hudson Valley family who loved their enslaved humans so much they recorded their births in their Bible. Others set aside land to bury their enslaved.
Not surprisingly, slave owners emphasized the “happy.” In the 1760s, Hector St. John de Crevecoeur, a Frenchman who lived near Newburgh, noted that “the few Negroes we have are at best but our friends and companions.” They were expensive to purchase and to cloth, and they might run away, he wrote, but “the simple Negro fiddle” was much respected.
He wrote that his enslaved humans sometimes appeared “as happy and merry” as if they were free. “The sight of their happiness always increases mine, provided it does not degenerate into licentiousness, and this is sometimes the case, although we have laws to prevent it.”
“In the main, the relation between the races was amicable and often it was one of devotion and attachment,” Reynolds wrote, although she conceded that “occasionally there were instances unpleasant to tell of, as they reveal each race at its worst — the Negro in underdevelopment; the white in brutality.”
The problem with these sanitized accounts, says Groth, is that while mentions of scars and injuries in newspaper ads hint at the brutality, “explicit details of such traumatic encounters will likely remain hidden forever.”
Some whites and the people they enslaved may have expressed affection; it’s unlikely that all slaveholders were physically abusive, if only to avoid damaging valuable goods. Groth cites a study of black rural life in Virginia where historians found evidence that whites perceived slaves in general in harsher terms than those they had personally enslaved.
There is historical evidence that “white and Black residents of Dutchess worked, lived, ate, drank, played, sang and danced together,” notes Groth; some slaves used that familiarity to win concessions.
Yet even if slaves were not physically abused, they lived with the constant possibility of being sold and separated from loved ones. In 1717, Cadwallader Colden, near Newburgh, expressed no remorse shipping one of his slaves to Barbados. In a letter to a friend, he wrote: “I send by this vessel … a Negro woman and child. She is a good house Negro. Were it not for her allusive tongue, her sullenness, I would not have parted with her. … I have several of her children I value and I know if she would stay in this country she would spoil them.”
Sojourner Truth, who saw her nine siblings sold off in Ulster County, would later describe what she saw in older slaves as “the misery.”
For the latest episode of The Current podcast, Chip Rowe spoke with Michael Groth, author of Slavery and Freedom in the Mid-Hudson Valley, about the overlooked history of slavery in New York and the African American struggle for freedom.
This series was funded by a grant from the William G. Pomeroy Foundation and contributions to our Special Projects Fund.
Thank you for detailing some of the history of slavery here in the Highlands. White readers like myself would be wise to consider in what ways we behave like Madam Brett, who in her will gave one person she enslaved the right to choose her next master.
It almost sounds like compassion. But she did not free this person. Instead she continued to benefit from and uphold the region’s “slave society.” In what ways do we similarly claim solidarity when it’s convenient and requires nothing of us, while openly benefiting from — and refusing to overturn — unjust systems that oppress our neighbors?
I read both of your articles in this series with great interest, as I have a direct ancestor connection to the Nelsons, Mandevilles and Davenports who settled this area.
In some of my historical research, I had learned that the Davenports were heavily involved in the extremely brutal sugar plantations in Barbados for many years and was aware that they owned “Negros” because they listed them as property in their wills. It’s important to tell about history as it actually happened, not as we wish it had happened.