On the evening of May 3 and all day May 4, Hooker remained in his defenses while Lee and Early battled Sedgwick. Sedgwick, after breaking Early's defenses, foolishly neglected to secure Fredericksburg. Early simply marched back and reoccupied the heights west of the city, cutting Sedgwick off. Meanwhile, Lee directed the division of Maj. Gen. Richard H. Anderson from the Chancellorsville front and reinforced McLaws before Sedgwick realized just how few men were opposing him. Sedgwick, as it turned out, was as resolute on the defensive as he was irresolute on the attack, and he stood his ground that day before withdrawing back across the Rappahannock at Banks's Ford during the pre-dawn hours of May 5. This was another miscommunication between him and Hooker; the commanding general had wanted Sedgwick to hold Banks's Ford, so that Hooker could withdraw from the Chancellorsville area and re-cross the river at Banks's to fight again. When he learned that Sedgwick had retreated back over the river, Hooker felt he was out of options to save the campaign, and on the night of May 5–6, he also withdrew back across the river.
Wounded soldiers being tended in the field after the Battle of Chancellorsville, May 2, 1863.
Stoneman, after a week of ineffectual raiding in central and southern Virginia in which he failed to attack any of the objectives Hooker set out for him, withdrew into Union lines east of Richmond on May 7, ending the campaign.
The battle was fought under terrible conditions. Soldiers tended to get lost in the impenetrable maze of undergrowth, and many fires started during the course of the battle. Reports of wounded men being burned alive were common.
Lee, despite being outnumbered by a ratio of about five to two, won arguably his greatest victory of the war. But he paid a terrible price for it. With only 52,000 infantry engaged, he suffered more than 13,000 casualties, losing some 25% of his force—men that the Confederacy, with its limited manpower, could not replace. Just as seriously, he lost several top generals, most notably Jackson, his most aggressive field commander.
Confederate dead behind the stone wall of Marye's Heights, Fredericksburg, Virginia, killed during the Battle of Chancellorsville, May 1863. Photograph by A.J. Russell.
Hooker, who began the campaign believing he had "80 chances in 100 to be successful", lost the battle through miscommunications, the incompetence of some of his leading generals (most notably Howard and Stoneman, but also Sedgwick), and through some serious errors of his own. Hooker's errors include abandoning his offensive push on May 1 and ordering Sickles to give up Hazel Grove and pull back on May 2. He also erred in his disposition of forces; some 40,000 men of the Army of the Potomac scarcely fired a shot. When later asked why he had ordered a halt to his advance on May 1, Hooker is reputed to have responded, "For the first time, I lost faith in Hooker." However, Stephen W. Sears has categorized this as a myth:
Nothing has been more damaging to General Joseph Hooker's military reputation than this, from John Bigelowe's The Campaign of Chancellorsville (1910): "A couple of months later, when Hooker crossed the Rappahannock [actually, the Potomac] with the Army of the Potomac in the Campaign of Gettysburg he was asked by General Doubleday: 'Hooker, what was the matter with you at Chancellorsville? ... Hooker answered frankly ... 'Doubleday ... For once I lost confidence in Hooker'".
Sears's research has shown that Bigelowe was quoting from a letter written in 1903 by an E. P. Halstead, who was on the staff of Doubleday's I Corps division. There is no evidence that Hooker and Doubleday ever met during the Gettysburg Campaign, nor was there any chance of them meeting—they were dozens of miles apart. Finally, Doubleday made no mention of such a confession from Hooker in his history of the Chancellorsville Campaign, published in 1882. Sears concludes:
It can only be concluded that forty years after the event, elderly ex-staff officer Halstead was at best retailing some vaguely remembered campfire tale, and at worst manufacturing a role for himself in histories of the campaign ... Whatever Joe Hooker's failings at Chancellorsville, he did not publicly confess them.
Of the 90,000 Union men who bore the brunt of the fighting, just over 17,000 fell in battle, a casualty rate much lower than Lee's, and this without taking into account the 4,000 men of the XI Corps who were captured without a fight in the initial panic on May 2.
Modern photograph of the battlefield, as preserved as part of Fredericksburg and Spotsylvania National Military Park
Hooker's tactic of forcing Lee to attack him was clearly sound in concept, but it was terribly flawed in the way he and his subordinates implemented it. The actual fighting showed the Union army had become as formidable in battle as Lee's heretofore "unbeatable" soldiers.
The Union was shocked by the defeat. President Abraham Lincoln was quoted as saying, "My God! My God! What will the country say?" A few generals were career casualties. Hooker relieved Stoneman for incompetence. Couch was so disgusted by Hooker's conduct of the battle (and his incessant political maneuvering) that he resigned and was placed in charge of the Pennsylvania militia. Hooker was relieved of command on June 28, just before the Battle of Gettysburg.
The Battle of Chancellorsville, along with the May 1864 Battle of the Wilderness fought nearby, formed the basis for Stephen Crane's 1895 novel The Red Badge of Courage.
Portions of the Chancellorsville battlefield are now preserved as part of Fredericksburg and Spotsylvania National Military Park.