Before dawn on November 16, the British and Hessian troops moved out. Knyphausen and his troops were ferried across the Harlem River on flatboats and landed on Manhattan. The flatboatmen then turned down the river to ferry Mathew's troops across the river. However, due to the tide, they were unable to get close enough to the shore to bring the British troops across. Thus, Knyphausen's troops were forced to halt their advance and wait until Mathew could cross. Around 7:00 a.m., Hessian guns opened fire on the American battery on Laurel Hill, and the British frigate Pearl began to fire at the American entrenchments. Also, south of the Fort, Percy had his artillery open fire on the fort itself. Percy's artillery aimed at Magaw's guns which had damaged the British ships several week before.
By noon, Knyphausen and his Hessians restarted their advance. As soon as the tide was high enough, Matthew, accompanied by Howe and his troops, were ferried across the Harlem River and landed, under heavy fire from the American artillery, on the Manhattan shore. The British troops charged up the hillside and dispersed the Americans until they reached a redoubt defended by some Pennsylvania Volunteer companies. After brief fighting, the Americans turned and ran towards the fort.
To the north of the fort, the Hessian right, commanded by Johann Rall, moved up the steep hillside south of Spuyten Duyvil Creek against almost no resistance from the Americans. The Hessians began to bring up their artillery. At this point, the main body of Hessians, 4,000 men, under Knyphausen began to advance down the Post Road, which ran between Laurel Hill and the hill Rall was on. The Hessians crossed swampy land and when they approached the wooded hillside near the fort, they were fired upon by 250 riflemen under the command of Colonel Moses Rawlings. Rawlings' men hid behind rocks and trees and darted from place to place to shoot at the Hessians as they tried to advance through the fallen trees and rocks. The first and second charged by the Hessians were repulsed by Rawlings' riflemen.
Around the same time, to the south, Percy advanced with some 3,000 men. Percy advanced in two columns with his brigade of Hessians on the left and Percy himself leading the right. About 200 yards from the American lines Percy halted the advance, waiting for the feint by Sterling to take place. Facing Percy was Alexander Graydon and his company. Graydon's superior was Lambert Cadwalader, Magaw's second in command, who was in charge of holding the three defensive lines south of Fort Washington. After hearing that there was a landing on the shore in his rear, Cadwalader sent 50 men to oppose it. The 50 men ran into the feint by Sterling and the 42nd Regiment of 800 men. Where Sterling landed happened to be the least defended area of the American defenses, and when Cadwalader heard how many men were there, he sent another 100 men to reinforce the 50 he had sent earlier. The British landing parties spread out, looking for a path through the rough terrain on the landing spot. The Americans took up a position on a hilltop and began firing at the British troops who were still crossing the river, killing or wounding 80 men. The British troops charged the American position, dispersing them.
Upon hearing the shooting, Percy ordered his troops to continue their advance.British artillery fire forced Graydon in the first defensive line to fall back to the second line, where Washington, Greene, Putnam and Hugh Mercer were located. The four were encouraged to leave Manhattan, which they did immediately, sailing across the river to Fort Lee. Magaw realized that Cadwalader was in danger of being surrounded and sent orders for him to withdraw toward the fort. Cadwalader's force was pursued by Percy's troops at the same time the troops opposing Sterling's landing were also being chased back to the fort. Sterling's troops, landed in the rear of Cadwalader, paused, believing that there were troops in the entrenchments. Some of the retreating Americans engaged Sterling, giving most of the rest of the American troops enough time to escape.