Always Present, Never Seen

Most schoolchildren have heard the story of John Andre, Benedict Arnold and George Washington, three men whose actions during the Revolutionary War changed the course of history. Far fewer know of Jack Peterson, a 34-year-old Black patriot who changed the course of Andre.

A British major who conspired with Arnold to seize West Point from colonialist control, Andre received a map of the fort in September 1780 from the American general, then its commander. He planned to deliver it and other documents to his superiors in New York City. Had he succeeded, and the British taken West Point, they would have controlled the Hudson River, and the war may well have been lost.

Andre met Arnold in Haverstraw, on the western shore. He had been brought to the spot by a skiff from the 14-gun British sloop HMS Vulture — the same ship that would later rescue the traitorous Arnold from his headquarters in Garrison — and planned to return the same way. Instead, Andre found himself walking in disguise through Westchester, with the incriminating documents in his boot. He was captured by three militiamen near Tarrytown on Sept. 23.

After dropping Andre at Haverstraw, where he and Arnold stayed overnight, the Vulture retreated south to the relative safety of Croton’s Point. As it waited, about 20 British soldiers rowed in a skiff toward the east bank, accompanied by a small boat with a cannon at its bow.

By one account of what happened next, according to Erik Weiselberg, the principal historian of Revolutionary Westchester 250, Jack Peterson deserves credit for a small act of defiance that led to American democracy — for others. He was one of the many Blacks, free and enslaved, who aided the patriot cause in the Hudson Valley even though the freedom they fought and sometimes died for would not be their own.

The journey to shore of the soldiers on the skiff was interrupted by small arms fire from Peterson, a sharpshooter whose nickname was “Rifle Jack.” This is according to an 1848 history that said Peterson and a 19-year-old white soldier named Moses Sherwood had been making cider when they spotted the Vulture. (The captain of the Vulture later alleged the Americans lured them to shore under a false flag of truce. “Surely the British were up to no good, but the claim does make the simple heroic narrative of brave shoreline defenders minding their own business a bit more complex,” Weiselberg notes.)

Peterson, who grew up in the family of Job Sherwood in modern-day Sleepy Hollow, was described in his obituary as “mulatto,” but it is unclear whether he had been enslaved. When Sherwood’s son, Isaac, joined the Continental Army at Peekskill, Peterson joined with him and helped defend the city against a British raid in March 1777, the first of several battles he fought in over the next three years.

He left the army in 1780 but remained a member of the militia. On the morning of Sept. 20, he and Moses (a cousin to Isaac) spotted the Vulture. According to the 1848 account, the men hid behind rocks to fire. One shot from Peterson caused a Redcoat to drop his oar; the skiff turned back while the gunboat lingered to provide cover.

Peterson and Sherwood traveled 5 miles north to Fort Lafayette at what is now Cortlandt to report the encounter. A colonel sent several cannons to Croton Point, where, at first light on Sept. 22, they bombarded the Vulture for two hours, forcing it to retreat downriver.

Without the Vulture, Maj. Andre was forced to walk. Following his capture, he was hanged on Oct. 2 as a spy.

Jack Peterson was later taken prisoner by the British and held in a ship in New York harbor, according to his pension papers, but escaped by crawling down an anchor chain. He lived in Cortlandt and Peekskill, and in 1818, at age 63, received a pension but never owned the minimum amount of property required of Black men to participate in civic life. Despite his pinpoint shot that may have preserved a nation, Rifle Jack could not vote.

The white revolution

At the time of the Revolution, Fishkill was the largest village in Dutchess County, with two churches, a schoolhouse, a hotel and a printing press.

When the war began in the spring of 1775, anxious landowners organized a committee to monitor the “affairs of the Negroes.” In Newburgh, the council imposed a sundown curfew for the enslaved on punishment of 35 lashes. Provincial lawmakers authorized the shooting of any slave found more than a mile from home.

Communities on both sides of the Hudson River became armed encampments. Fishkill was a major supply depot, hospital, prison and burial ground.

About a third of Dutchess residents refused to swear allegiance to the American cause, and the Continental Army was plagued by enlistees who came and went. Although nearly 400,000 men were recruited to fight, there were never more than 35,000 in the field. The British forces never exceeded 42,000, so the Americans could easily have overwhelmed them at even quarter strength, historians note.

“African Americans participated in the American Revolution from its first day,” points out historian Thomas Fleming, writing in the Journal of the American Revolution. “That is why it is shameful that for a long time, they received no credit for their courage and enthusiasm for liberty. The minutemen who fought at Lexington and Concord on April 19, 1775, included at least nine Black soldiers.”

Testimony: Black Patriots

Benjamin Latimore

Benjamin Latimore enlisted in the Continental Army in September 1776 and by April was at Fort Montgomery on the Hudson River under the command of Gen. George Clinton.

“In the early part of [October 1777], some vessels belonging to the enemy came in sight and when within five miles of the fort, the wind having slackened, they disembarked at the place called Dunkerbarach [Dunderberg] or Thunder Hill and marched from thence to the fort,” Latimore recalled in his pension application.

The British commander met with Clinton outside the fort. After the usual salutations, the British officer, Col. Mungo Campbell, was asked by Clinton the nature of his business, and he replied that he “came to demand a surrender of the fort, which if done within an hour and our troops grounded their arms they would be permitted to go as it was not wished to make them prisoners because they (the enemy) had more men with them than could be accommodated in the fort.”

Latimore said that Clinton told Campbell that Fort Montgomery would not be surrendered “as long as he had a man able to fire a gun.” Campbell replied that Clinton “would sleep in the fort that night, or in hell.”

The fort held until 6 or 7 p.m., Latimore said. Campbell was killed during the attack, and Clinton escaped. But Latimore and others were held in the fort until October, when it was destroyed and the British soldiers and their prisoners sailed to New York City.

Latimore was forced to become a servant to a British officer until he was liberated by American forces and taken before Major Gen. Israel Putnam, who ordered him to rejoin the 5th Regiment at New Windsor.

Thomas Rosekrans

Thomas Rosekrans enlisted in the fall of 1778 at Continental Village in Philipstown. He was a substitute for Thomas Johnson and was assigned to serve as a waiter to his owner’s son, Maj. James Rosekrans.

“At the time I enlisted I recollect that an officer took hold of my hand and made me write my name that I would serve during the war, that I would be true to my country and that when I had enlisted the said Thomas Johnson, whose substitute I was, was then discharged,” Rosekrans recalled in his pension application. “I was then called Joseph Johnson on account (as I suppose) of my having Thomas Johnson’s place.”

Mary Dewitt, the sister of James Rosekrans, testified in support of the former slave’s pension application. She remembered he “left Fishkill on Sunday morning to join the army and that I furnished him with some food to put in his knapsack to eat by the way.”

Like Rosekrans, other enslaved men who fought expressed uncertainty about what name had been assigned to them during their service. Marlin Brown, when applying for a pension in 1834, noted he was known as Marlin Roorback after his owner in Wallkill, who gave him permission to enlist, but that he was now known as Brown after a later owner, or more generally just as Marlin.

Charles Stourman

Charles Stourman enlisted in the spring of 1776 to serve under Col. Hugh Hughes, the commissary of the military stores at Fishkill.

“I entered immediately in the hospital service situated about three-fourths of a mile from Fishkill, acting as a waiter or assistant to Dr. McKnight, who was a surgeon at the hospital,” he recalled. “During the summer and fall I attended on the sick. The hospital and barracks were full of sick and dying.

“I continued there during the winter, cutting and preparing wood for the hospital and sometime in the next summer [1777] I went to Fort Montgomery and was there at the taking of the fort. I took a team and baggage wagon and drove it up the river and saved it from the British. I went back to Fishkill and stayed there that winter and attended to the same duties I had formerly, to wit tending the hospital. The next spring [1778] I was sent by Col. Hughes with a team to get provisions and forage and followed that business during that summer and fall.”

After serving for three years and receiving a discharge, Stourman fought for another eight months in Virginia.

That was not Washington’s choice. Only days after he took command in June 1775, he decreed that no Black man could join the ranks. As a Virginian, he was reacting to a Southern fear that arming slaves would not be prudent. Many Black men contributed in other roles: as drivers, orderlies, waiters, cooks, bakers (especially at Tarrytown, Continental Village and Fishkill, which had bread ovens), craftsmen and laborers. They were also engaged at New Windsor at the iron works where workers forged the chain that was stretched across the river as a defense against British ships.

After Washington told his Black troops their service would expire on Dec. 31, a group appeared at his headquarters outside Boston to protest. As a compromise — and because in November the British began offering freedom to slaves who joined his Majesty’s troops — Black soldiers were allowed to re-enlist. By 1777, there was such a manpower shortage that Washington required men to enlist for at least three years and each town was given a quota. New York had earlier authorized owners to send an enslaved worker to serve for them in the militia; in 1781, the Legislature promised to free slaves who enlisted for the minimum tour or until they were honorably discharged. Their owners received land grants as compensation.

Along with Peterson, there were many other Black heroes of the Revolution, such as Pompey Lamb. The capture of Stony Point in Rockland County in 1779 by Gen. “Mad” Anthony Wayne was the result of Pompey’s ability to move between his owner’s home and the fort to sell fruits and vegetables to the British. He eventually acquired the password from the guards: “The fort is ours”; Pompey was among those who overpowered the sentries and allowed Wayne’s 1,350 continentals (including many free Blacks) to take the fort on July 16.

Testimony: Nanna

In 1874, Henry Bailey recalled in a memoir the former enslaved woman he knew as a child, Nanna, who had been purchased in 1805 by his grandfather for £40. His grandfather was a newlywed and needed someone to keep house.

Nanna shared recollections with young Henry of the Revolutionary War, such as when the British fleet came up the Hudson River and everyone in the Van Voorhis family of Fishkill Landing (Beacon), which had enslaved her since birth, fled to Great Nine Partners (near Millbrook), except for the father. He said he would never leave his house, and Nanna said she could not leave him.

When the British ships arrived in Newburgh Bay, they began firing cannonballs at the Van Voorhis house. She and her owner went to the cellar kitchen until the fleet passed.

Nanna said she remembered Gen. Israel Putnam arriving at Fishkill Landing on horseback. She also saw Gens. Washington and Lafayette, and the American army camped on the flats north of the Highlands.

On one occasion, she helped prepare the home of Robert Brett (the Van Wyck Homestead on Route 9) for Washington, who was quartered there. She helped light the candles, she said.

Nanna, who could speak Dutch and English, was freed in 1827 when slavery was abolished in New York state. “Her son and daughter came to our house to take her home with them,” Bailey recalled. “My parents tried to persuade her to remain with them [with] what little time was allotted to her on earth, but the boon of freedom was too great for her. When she left the house, we all wept.” She did not know her age but was probably about 70, he said.

“Nanna found freedom far different than she expected,” Bailey wrote. “The colored people were poor, and there was no fuel in the country but wood,” which was expensive, “and she found that the fire of freedom did not suffice. She did not find wood and provisions as plenty there as she had in the days of slavery, and she wished herself back again in the old kitchen, a slave under her old master.”

He said the family sent for her but were told she had recently died in a little house near where the Dutchess Hat Works was later built [now Madam Brett Park].

Among the Black men with George Washington at Valley Forge in the winter of 1777-78 were Philip Field of Dutchess County, the son of a slave, and Henry Crandle, who was compelled to fight by his owner, John Crandle of Fishkill.

The Black people of the Hudson Valley during the Revolution had a choice to make. The historian Benjamin Quarles notes that the enslaved worker was most likely “to join the side that made him the quickest and best offer in terms of those ‘inalienable rights’ of which Mr. [Thomas] Jefferson had spoken.”

Male slaves who responded to the 1775 promise from the British that they would be freed for their service were assigned to what became known as the Ethiopian Regiment; they wore sashes with the words “Liberty to Slaves.” In 1779, British Gen. Henry Clinton created the Black Pioneers, while also noting any Black man caught fighting for the Americans would be sold.

To gather support for their cause, America’s first leaders cited the prospect of hordes of free, angry and armed Black people should the British prevail. During the war, Congress assigned Benjamin Franklin to write a children’s book about the evils of British rule; in it, he claimed the British would pay slaves to kill their masters and kidnap white women and children. While imagining the engravings for the book, Franklin envisioned a scene of armed Black plunderers that he thought could be engraved on the first U.S. coins.

“The matter of choosing sides ultimately proved to be an agonizing experience” for enslaved people in New York, says historian Michael Groth, author of Slavery and Freedom in the Mid-Hudson Valley. “Practical considerations weighed heavily.” Most who escaped to New York City were young men, he notes; the majority of enslaved workers in the Hudson Valley seemed to adopt a wait-and-see attitude.

Gen. Washington and Billy Lee

Many years after the fact, John Trumbull painted this portrait of George Washington from memory. It depicts the general in 1780 on a bluff overlooking the Hudson River with his slave, Billy Lee, on horseback.

Lee, purchased by Washington in 1768, was the general’s valet during the Revolutionary War and the only slave he freed outright in his will. After the war, Lee broke both of his kneecaps in accidents, which ended his work as a valet (he took up shoemaking instead). He is buried at Mount Vernon.

Lee is probably the “domestic” involved in the court-martial of Joshua Hett Smith, who was accused of complicity in the betrayal by Gen. Benedict Arnold. Maj. Gen. Marquis de Lafayette attempted to interrogate Smith on Sept. 26, 1780, at the Robinson House, which stood on Route 9D in Garrison, but Smith was an experienced lawyer and apparently embarrassed the Frenchman, who was teased by his fellow officers.

Smith, who was being kept in solitary confinement, learned of Lafayette’s discomfort from one of Washington’s enslaved workers, “who daily brought me provisions.” That person — certainly Billy Lee — said that Lafayette, when Smith’s name was mentioned, would bristle.

Smith’s account suggests Billy Lee was a part of Washington’s inner circle, notes historian William Ferraro. “He was trusted enough by Washington to spend time alone with an important prisoner.” In addition, the episode demonstrates “that Lee felt comfortable discussing a touchy subject with a highly educated professional.”

The portrait is now owned by the Metropolitan Museum of Art in New York City.

Some white people, such as Abigail Adams, the future First Lady, recognized the hypocrisy of fighting for freedom from British rule, but not for Black people. “It always appeared a most iniquitous scheme to me to fight ourselves for what we are daily robbing and plundering from those who have as good a right to freedom as we have,” she wrote to her husband John in September 1774.

In fact, many American leaders during the Revolution voiced support for emancipation, but only in principle. As early as 1771 the delegates at the New York Constitutional Convention had voted 31-5 to abolish slavery, but no action was taken.

After the war, in 1785, the state Assembly considered a bill that would gradually emancipate the enslaved. However, the proposal awarded Blacks only second-class citizenship, denying them the right to vote, hold public office, marry white people or give testimony in court against white people. That didn’t fly in the more liberal Senate, and the bill died.

The Assembly, historian Edgar McManus observed, “feared Negro suffrage more than it desired emancipation.” In the end, the Legislature managed only to pass a law banning the importation of slaves.

The New York Manumission Society was formed in 1785, and by the 1790s, freeing the enslaved had become a common cause, with the editors of The Poughkeepsie Journal and other papers questioning the morals of slave owners. The economics of forced labor had changed because of an influx of white immigrants. Enslaved workers required capital investment and maintenance costs; not so the hired hand. With jobs harder to find, the working poor grew to resent slaves and former slaves. Even free Blacks were brutalized and excluded from skilled positions.

Fighting for freedom

After Lord Cornwallis surrendered at Yorktown in Oct. 19, 1781, the defeated British evacuated New York City. On Oct. 25, Gen. Washington issued an order that any “Negroes or Molattoes” be rounded up and put under guard so they could be returned to their owners. One condition of the 1783 Treaty of Paris that officially ended the war was that the British would not “carry away any Negroes or other Property of the American inhabitants.”

Escape!

“I prayed [for freedom] for 20 years but received no answer until I prayed with my legs.” ~Frederick Douglass

The enslaved fleeing from their enslavers was a problem for white New Yorkers from the beginning. As early as 1705, the colonial legislature forbade slaves from traveling alone more than 40 miles north of Albany (i.e., toward Canada); in 1755, it added the death penalty. As late as 1775, in Newburgh, slaves found without a pass were to be whipped 35 times.

The descriptions in newspaper ads offering rewards for the return of escapees provide testament to the brutality inflicted on Black people, noting limps and scars. Most escaped slaves were described as highly intelligent or “cunning,” since doing otherwise would not reflect well on the owner who let them get away.

In a collection of about 600 advertisements published between 1735 and 1831 for escaped Hudson Valley slaves and collected by Susan Stessin-Cohn and Ashley Hurlburt-Biagini for their book, In Defiance, about two dozen were placed by residents of Fishkill. The earliest was from 1747 (Robert Shere seeking 40-year-old Harry), and the latest from the early 1820s, including one by Medad Raymond of Peekskill, who was looking for Sam, 17, who had been allowed to visit his enslaved parents in Fishkill but never returned.

There were no fixed routes but runaways tended to flee along the Hudson River or through the communities of Quakers (who opposed slavery) in the center of Dutchess County or near the Connecticut border. Enslaved people fleeing the South likely found refuge among the large Black population of Fishkill Landing (Beacon), which also had a Black church, St. James AME Zion, founded in 1844. Joe Collin or Collis, a Black fish peddler, is thought to have transported fugitives across the river to Newburgh, supposedly communicating with horn blasts and lamps.

At its second annual meeting in 1840, members of the Dutchess County Anti-Slavery Society adopted a resolution vowing they “collectively and severally will do all in our power to assist those brethren, coming through this county, who may have thus far escaped the iron grasp of tyranny, by giving them meat, money and clothes, to enable them to prosecute their journey to a land of liberty.”

This was a reference to the Underground Railroad, although as historian Albert Bushell Hart noted, it was “not a route, but a network; not an organization but a conspiracy.” It was so underground, noted historian A.J. Williams-Myer, that many fugitives may not have been aware it existed.

The Quakers had stops on the railroad from the 1830s in Millbrook and Pawling. Many fugitives were also sent up the Hudson on barges and, after 1851, the railroad. In 1855, Harriet Tubman helped three of her brothers escape north from Maryland by buying them tickets at Grand Central Station.

The Mid-Hudson Antislavery History Project notes that “it is important to keep the Underground Railroad in context. Its ‘conductors’ freed only a tiny fraction of enslaved people. The majority of white Northerners did not support the road, and it seldom ran ‘underground.’ Most tunnels or secret hideaways found in historical properties probably had other uses” than hiding runaways.

Nevertheless, the Hudson River was a natural path to Canada, where escaped slaves would be outside the jurisdiction of an 1850 Fugitive Slave Act that required enslaved workers be returned to their owners, even from the free north.

This led to an unusual case in 1851 in Dutchess County, where John Bolding had fled from bondage in South Carolina a few years earlier and assumed a new identity. A Southern woman who had been staying in Poughkeepsie recognized him and notified his former owner, who obtained a warrant.

Bolding was arrested by federal agents and taken to New York City to be shipped south, but supporters managed to get him in front a federal judge. Two lawyers hired on his behalf argued that, in fact, the light-skinned Bolding was “as white as a great many white men” and so could not have been a slave. The judge ruled against them, but Dutchess abolitionists raised $2,000 to buy Bolding’s freedom and he returned to Poughkeepsie, where he lived for the next 25 years until his death in 1876.

Black people in Dutchess County were in a precarious position during the 1850s as the country split apart, notes Michael Groth, author of Slavery and Freedom in the Mid-Hudson Valley. Merritt Green of Fishkill was so fearful of being kidnapped by slavecatchers that he attempted suicide, vowing that “he had rather die than be enslaved.” Black people, even in the North, had few allies. White immigrants from Europe were arriving in larger numbers, competing with Black residents for even meager housing and jobs. The Poughkeepsie Telegraph in 1851 told its readers that preserving the Union was more important than the “rescue of a few slaves from their bondage.”

The peace agreement allowed that any enslaved person who had reached British lines before Nov. 30, 1782, would be freed, but only with restitution to his or her owner.

To determine which Black people were eligible, a group of American and British officials met each Wednesday for six months at a Black-owned tavern on Broad Street in New York City to compile what became known as the “book of Negroes.” It contained the name, age and date of escape of every runaway slave who had reached the city during the war, including at least four who had escaped from enslavement in Dutchess County:

■ Ceaser Nicolls, 22, described as a “stout Blk Man,” had reached the British lines 3½ years earlier [in 1780] after fleeing from the Van Wyck family in Fishkill. He planned to travel aboard the Ann with his certificate of freedom issued by the British for a settlement in Nova Scotia called Port Roseway.

■ Bristol Storm, 40, a “stout fellow,” had escaped three years earlier [in 1780] from Garret Storm of Fishkill. He was destined for Annapolis/St. John, Nova Scotia, aboard the Clinton. Unlike Nicolls, he did not have a certificate of freedom.

■ Massey Asten, 31, a “stout wench,” had escaped six years earlier [1777] from her Dutchess County owner, Joseph Jenkins. She was headed to Annapolis, Nova Scotia, aboard the sloop Cato.

■John Simonsbury, 43, “healthy,” from Fredericksburg [now Patterson in Putnam County], had fled from his owner, Charles Collins, four years earlier [1779]. He was headed to Port Roseway on the London Frigate.

In 1783, about 3,000 Black men, women and children with certificates were allowed to leave for Canada; another 400 sailed for London; and 1,200 journeyed to Africa to create a community in Sierra Leone. In addition, about 1,500 enslaved people are believed to have been taken to Canada by their Loyalist masters.

By one estimate, 5,000 to 9,000 Black men fought with the Continental Army. But another 20,000 fled to the British in a bid for freedom that wasn’t being offered by the men who produced the Declaration of Independence.

“Our revered founders — intent on rallying mass support for a revolt intended to replace one set of colonial elites with another — indulged in egalitarian rhetoric that most of them did not believe,” observed historian Gregory Urwin in the Journal of the American Revolution. “What redeemed the Revolution is the fact that so many common Americans took that rhetoric literally.”

Reparations

Beverley Robinson, the son-in-law of Frederick Philipse, a large landowner and early slave trader whose family is the namesake of Philipstown, was an active Loyalist during the Revolutionary War. (His home in Garrison was seized by the Continental Army for use by Gen. Benedict Arnold.) After fleeing to London at war’s end, Robinson was one of 50 men from Dutchess County who made claims to be reimbursed by the crown.

He estimated his losses at £68,784, which included 60,000 acres, two mills and eight enslaved workers he left behind: Harry, 19 (valued at £80); Rose, 20 (£60); Coobaugh, 20 (£60); Belinda, 18 (£60); Sarah, 18 (£60); Phillis, 15 (£50); Candis, 15 (£50); and Clarinda, 13 (£50).

By comparison, Robinson valued a pair of oxen at £18 and a bushel of wheat at £7. He was awarded £23,287, or about a third of his claim.

This series is funded by a grant from the William G. Pomeroy Foundation and contributions to our Special Projects Fund (highlandscurrent.org/donate).

Part 1 | Part 2 | Part 3 | Part 4 | Part 5