Always Present, Never Seen

The third part of our Black history of the Highlands

Enslaved workers were brought by the Dutch to New York beginning in 1625 or 1626, but it wasn’t until 1799 that state legislators ended the practice — sort of. Ever practical, lawmakers decided it was best to take it slow.

According to the 1799 law, children born to enslaved mothers on or after July 4 of that year would not be slaves but instead be “bound in service” to their mother’s owner until age 25 (for girls) or 28 (for boys) — provided the owner taught them to read the Holy Scriptures.

In 1817, the Legislature fiddled with Black freedom again. It decreed that every slave born in New York would be free as of July 4, 1827. But it also said that children born to enslaved mothers from March 31, 1817, to July 3, 1827, would be bound in service to their mother’s owner until age 21.

These children were New York’s last generation of enslaved workers, and while owners might be willing to voluntarily free older slaves, they weren’t so eager to give up the kids. This was reflected, noted historian Vivienne Kruger, in the number of Black children who lived in white households in the 1820 census. Because of high childhood mortality rates, many did not live long enough to be free, she wrote.

The 1799 law does appear to have prompted many white patriarchs to free their slaves, with various degrees of compassion. Zacharias Van Voorhis of Rombout Precinct [Fishkill] instructed his executors “to order my Negros into that apartment in my house where I died, and there in the most solemn manner proclaim to them their freedom.” Adolph Myers of Fishkill freed two of his slaves, Harry and Jane, but gave three others to his children and directed the sale of a fourth to create an endowment for his grandchildren.

In most wills, freedom was promised — later. John Ackerman of Fishkill bequeathed Adam and Isaac to his sons with instructions to free the men on their 36th birthdays. John Lancaster of Beekman said that because it was “wrong and wicked to hold any people in a state of slavery,” he would release a 3-year-old girl he had purchased — when she turned 18. Egbert Bogardus of Fishkill told his executors that they could free Pegg as long as his estate would have no continuing financial liability. (If not, they were to sell her.)

Testimony: Sophia Burthen Pooley

A few slaves who were allowed to hire themselves out saved enough to buy their freedom, notes Michael Groth, author of Slavery and Freedom in the Mid-Hudson Valley. But the costs could be prohibitive. Caty Stevenson of Poughkeepsie made an agreement with her owner to receive her freedom after six years for $30. But after hiring herself out for four years, she had saved only $5. The enslaved also had to trust their owner’s word, since they could not enter into contracts. Anthony Murphy of Poughkeepsie agreed to buy his wife’s freedom for $10 but, when the time came, the price had increased to $20.

It took 30 years, and two centuries, but the legal ownership of humans in New York eventually came to an end. By 1820, more than 80 percent of Blacks in Suffolk, Queens and Westchester counties were free, along with more than 95 percent in New York City, notes historian David Gellman.

Ten years later, after the 1827 emancipation, there were fewer than about 50 enslaved people in the state, according to an analysis of census records by historian Michael Douma of Georgetown University, including four in Putnam County but none in Dutchess. Most were likely children of enslaved mothers who were legally indentured servants, he says. Three of the four people recorded as enslaved in Putnam lived in the household of Daniel Travis in Philipstown.

The People vs. Alvin, a Black Man

In 1816, Alvin, a Black man enslaved in Putnam County, was charged with grand larceny. He was alleged to have stolen a $20 bank note from the home of Stephen Frost. The court found him guilty and sentenced him to three years of hard labor. That didn’t sit well with Joseph Crane of Southeast, who owned Alvin and his hard labor. Although it was illegal under New York law to sell an enslaved person out of state, the court granted Crane a 14-day exemption so he could recoup his loss.

In 1840, census takers tallied four enslaved people in New York state, including an older woman in Putnam; in 1850 there was one: an older man in Putnam. But Douma suspects these were errors and those marked as slaves were actually free Black people who continued to live with their former owners.

Pursuing the vote

In 1821, New York lawmakers changed the state constitution to allow all white men age 21 and older to vote. They also allowed free Black men to vote if they had lived in New York for three years and owned more than $250 in property. Fourteen years later, in 1835, there were seven qualified Black voters in Albany and four in Buffalo. (The earliest record that the Beacon Historical Society has of a Black person owning property is a deed for a home purchased by Isaac Atkins in 1819.)

In the 1830s, free Black people began to organize conventions to strategize about how to expand their role in civic life. The voices split between those such as Frederick Douglass, who preached restraint and working within the system, and those, such as the Rev. Henry Highland Garnet, who were more militant and less willing to work with white abolitionists. Eventually, Douglass’ approach won out.

“If a dissolution of the Union must take place, let it be so! If civil war, which gentlemen so much threaten, must come, I can only say, let it come!”

~ Rep. John Tallmadge, who represented Dutchess County in Congress, in 1819 after he introduced a bill that would allow Missouri to join the union only if it banned slavery

“Radical” abolitionists who called for immediate emancipation faced a tough audience in Dutchess County. By 1837, there were 19 antislavery societies in upstate counties, but none in the Hudson Valley. After an open call in Poughkeepsie in 1838 to create the Dutchess County Antislavery Society, a mob disrupted the proceedings. The group was formed, but when it asked pastors in Poughkeepsie to announce its prayer meetings, four of the five refused. The minutes of the society end abruptly in May 1840.

Abolitionist candidates did terribly in Dutchess County; of the 10,000 votes cast for governor in 1840, Gerrit Smith received 13. The Free Party candidate for the state Assembly tallied 29 votes in 1839, four in 1841 and 22 in 1842. An anti-slavery newspaper founded in Dutchess County in 1839, Bow of Promise, topped out at 131 subscribers, with 17 paid.

Most residents of the Hudson Valley were not sympathetic to abolitionists in part because of the region’s economic ties to New York City, and New York City’s economic ties to the South. “Most places on the Hudson River,” said abolitionist Sam Ringgold Ward, who escaped slavery to become a newspaper editor and labor leader, were “thoroughly and hopelessly pro-slavery.”

At a state constitutional convention in 1846, delegates argued over whether to open the vote to all Black men, not just property owners. Two of the three representatives from Dutchess voted “no,” but it was decided to hold a referendum. The question was defeated by a nearly 3-to-1 margin; in Dutchess County, it lost by an 8-to-1 margin. In Putnam County, it fell 42-to-1.

By 1850, according to the federal census, there were 60 Black people and 12 “mulattoes” in what is now Beacon, and five Black families who owned their homes: Robert DeWitt, Cornelius Schofield, James F. Brown, Samuel Lampons and Susan Washington. There were two boatmen, a gardener (Brown) and several laborers. All the homes appear to have been on the same street, which may have been what is today Rombout Avenue.

Most Blacks in Dutchess County — about 75 percent — lived in river towns such as Fishkill Landing (Beacon). A few Black men found modest economic success, but the vast majority in the 1850 census were identified as tenants, sharecroppers or hired hands. Only 15 of the more than 500 Black men in Dutchess whose occupations were recorded owned their own farms.

Barbering was one of the few trades dominated by Blacks, writes Groth. They catered to a largely white clientele but had to put up with racism with self-effacing banter. Frederick Douglass called on parents to make their sons “mechanics and farmers — not waiters, porters and barbers.” In 1853, his Frederick Douglass’ Paper ran a series called “Learn Trades or Starve.”

Black churches

In the absence of social support, Black people formed churches and mutual aid societies, “pooling their meager resources to provide for their own sick, widows and orphans, and decent burials,” wrote Lawrence Mamiya, a historian who taught religion and Africana studies at Vassar College.

In 1804 in New York City, the African Methodist Episcopal Zion (AMEZ) church was created under the guidance of the Methodists. In 1827, the Rev. George Matthews established a congregation in Newburgh, as well as a “station” in Fishkill Landing that by 1834 had 23 members.

James Brown, an escaped slave from Maryland who became a gardener at the Verplanck estate in Fishkill Landing, noted in a May 1844 entry in his journal the presentation of a parcel of land to build an AMEZ church; when it was dedicated in September, he underlined the entry. The St. James AMEZ church on Academy Street is the oldest church building in Beacon; its original site was across South Avenue. The church moved to its current location sometime after the Dutchess Tool Co. bought the property in 1890.

A Northerner Defends Slavery

James Kirke Paulding (1779-1860), a member of the first board of directors of the West Point Foundry, which was formed in Cold Spring in 1818, had a wide range of interests. He is credited with the tongue-twister “Peter Piper picked a peck of pickled peppers” and also served as secretary of the Navy. Two of his sons are buried in Philipstown, and Paulding Avenue is named for the family.

One of his interests was slavery. In his 1817 book, Letters from the South, he noted that the institution was at odds with the preamble of the Constitution. But he argued that its evils were exaggerated — that slave traders, not owners, were responsible for any brutality, and that enslaved Black workers seemed to be the happiest people on earth. He condemned any owner who “adds one feather to the weight they are destined to bear.”

In 1835 Paulding reissued the book, and a year later published Slavery in the United States, in which he noted that Jesus is never cited in the Bible saying slavery is wrong. He warned that freeing the enslaved would destroy the country because Blacks would demand political power despite their “natural and incurable inferiority.”

The next year, in 1836, Paulding wrote The Book of St. Nicholas, which helped popularize the Dutch Santa Claus tradition in America.

With the end of slavery, many Black communities began to emphasize self-improvement. The AMEZ church directed its pastors to stress the importance of education; its Sunday schools taught reading, writing and moral truths. New York State allowed districts to create separate public schools for Black children but there were often not enough to do so in rural Dutchess communities. As a result, children of all races attended together. And as might be expected, Black students suffered daily harassment and “petty annoyances,” notes historian Dennis Maika. In 1846, the state superintendent estimated that only 25 percent of Black children attended school.

Carleton Mabee, a historian who, until his death in 2014, wrote extensively about Black education in New York, concluded that the first public school for Black children and adolescents in Fishkill Landing probably opened in 1859. By 1863, it had 10 to 15 students. Few Blacks attended high school or college. As late as 1900, Vassar refused to admit Blacks; Beacon High School did not have its first Black graduate, William Howes, until 1925.

Letters Home from the Civil War

There were multiple attempts in Dutchess County to create Black colleges, including in 1870, when a group that included Samuel Jones, a laborer from Fishkill Landing, began an unsuccessful campaign to raise $300,000 for a 15-acre campus in Poughkeepsie named for Toussaint L’Overture, the Haitian revolutionary. But many Black people did not want Black colleges. Delegates to the New York Colored Citizens Convention in Troy in 1872 argued that they should instead advocate opening more white institutions to African Americans.

In 1883, the Black school in Fishkill Landing was added as a “department” to the white school; in 1890 the Black students were meeting in the “colored annex” when the school board decided segregating the five or six Black students wasn’t worth the expense and sent them to the white elementary school. While Black public schools continued to exist elsewhere in New York into the 1940s, the Fishkill Landing school was the last in Dutchess County, Mabee wrote.

Getting the vote

On Aug. 11, 1870, Frederick Douglass visited Newburgh, where he spoke at the Opera House (now the site of the Newburgh Free Library) to 4,000 people gathered to celebrate the ratification six months earlier of the 15th Amendment to the Constitution, which gave all African-American men the right to vote.

On the stage with him were 29 young Black women, representing each of the 29 states that ratified the amendment. The Newburgh City Band played an overture and an all-Black choir sang, “Now the Glorious Day Has Come.”

Douglass, then 52, told the crowd that he had been a fugitive slave (posing as a sailor, he escaped from Maryland by train in 1838), and then an advocate for slaves. But now he appeared as an American citizen, “one of the greatest privileges of which a human being can boast.”

“The colored people now had equality at the three boxes — the jury box, the ballot box and the cartridge box,” he said, according to an account in the Newburgh Daily Journal. “The lesson of the hour was to get the knowledge box.” The 15th Amendment, he said, meant that “hereafter the colored man was to be thrown upon his own resources. It meant fair play, and if he could live by that, he could live.”

But Douglass also noted that, without vocations, Black people would face fierce competition from white immigrants. One advantage the European newcomers had was “intergenerational mobility” — fathers passed their jobs to their sons. The least-skilled Black workers tended to wander in search of work. In 1860, most Black women were domestic servants, but they lost their jobs to Irish women and were pushed into doing laundry.

Return Visit

“After the Emancipation Proclamation by Lincoln, he concluded it was safe enough to visit his old master’s home. So in 1867 he started for Maryland. An absence of over 40 years had very much changed the appearance of the place, and his old master had departed this life, but others had not forgotten him. He was made happy by meeting those who had once known him when working on the plantation. His brother and sister, who are still living at an advanced age, gave him a hearty welcome.”

~ From the 1879 obituary of Joseph Thomas, 75, who had escaped to Poughkeepsie at age 20. He married a freed slave and they moved to Glenham, where he bought property. “He made many friends, and was much respected,” the obituary said. “He was always fond of joking and had a pleasant word for those who addressed him.”

James F. Brown

For 37 years, from 1829 to 1866, a Black gardener who lived in Fishkill Landing kept a diary that eventually filled 10 volumes. It was “a remarkable act for most Americans of his day and a virtually unheard of undertaking for an African American man,” says his biographer, Myra Young Armstead, a history professor at Bard College.

James F. Brown may have been the first Black man to own property in what is now Beacon, and to vote.

He was born a slave in 1793. At age 29, he was owned by a widow in Maryland, Susan Williams, who, over the next five years, rented him out, first to her brother-in-law and then to a businessman named Jeremiah Hoffman. In 1827, Hoffman allowed Brown to visit Delaware.

Brown did not return. Instead, he sent Hoffman an apologetic letter.

“I know that you will be astonished and surprised when you become acquainted with the unexpected course that I am now about to take, a step that I never had the most distant idea of taking,” he wrote. “But what can a man do who has his hands bound and his feet fettered? He will certainly try to get them loosened by fair and honorable means and, if not, so he will certainly get them loosened in any way that he may think the most advisable.”

Brown made his way to New York City and found work as a waiter at the home of lawyer and banker Daniel Verplanck. According to family lore, a Southern visitor to the Verplanck home recognized Brown and alerted Williams. She contacted Verplanck, who, Armstead believes, negotiated a $300 installment plan for Brown to buy his freedom and that of his new wife, Julia, who remained in Baltimore, for another $100. Armstead says there is a cryptic reference in the diary to Brown writing Susan Williams about three years later, which she speculates was his last annual payment.

By 1828, Brown had relocated to the Verplanck country home, Mount Gulian, in what is now Beacon. He worked chiefly as a coachman before taking over as gardener in 1836. The Verplancks shared an interest in horticulture with Henry Winthrop Sargent, who had a sprawling estate nearby.

By then, Brown had saved enough to buy a house — the exact location is not known — and pay property taxes. That qualified him to vote, which he did on Nov. 8, 1837.

“The election at Fishkill took place this day at which place James F. Brown voted for the first time,” he wrote in his journal. At the time, the Democratic party was working to keep Blacks away from the polls, so many gravitated to the Whigs, a party formed in 1834.

Brown was an activist for Black home-ownership, Armstead writes. Several of his friends began negotiating with whites to buy homes. James diligently paid his annual tax bill and made sure his neighbors’ were paid as well, even if they were employed elsewhere in the Hudson Valley or Manhattan or New Jersey.

Brown also worked to establish a cemetery for Black residents; in late 1851 he and four other trustees bought a parcel from John DeWindt for $125 and created the Colored Peoples Union Burying Ground. The first burial was John Henry Roose on Oct. 31. There had been a Black burial ground in Fishkill since at least 1832, but it may have been too small — only nine gravestones have been found there.

The burying ground, which measured 50-by-130 feet, is in the southern section of the Methodist Cemetery on North Walnut Street. In 1939 the Beacon City Council considered extending Oak Street to connect North Walnut and North Brett, but the city surveyor discovered the property was still owned by the descendants of the five original trustees.

“Today the cemetery’s most recognizable tombstones belong to a handful of black Civil War veterans,” noted Robert Murphy of the Beacon Historical Society in 2019. “A thorough history of Beacon’s black community has not been written, but when undertaken, it must begin in a cemetery.”

James died in 1868 at age 75. (He is buried in St. Luke’s Cemetery.) When Julia died in 1890, his journal passed to the Verplanck family, who donated it in 1942 to the New-York Historical Society.

James made his last entry in 1866: “The lowest tide that has been for many years in the Hudson River was this day — the flats was bare from the Long Dock down to Denning’s Point — so that persons could walk down to get eels and fish with one hand.”

For the latest episode of The Current podcast, Editor Chip Rowe spoke with Myra Young Armstead, the author of Freedom’s Gardener: James F. Brown, Horticulture and the Hudson Valley in Antebellum America, about this remarkable resident of Fishkill Landing — now Beacon — and his journal.

This series is funded by a grant from the William G. Pomeroy Foundation and contributions to our Special Projects Fund (highlandscurrent.org/donate).

Part 1 | Part 2 | Part 3 | Part 4 | Part 5