George, Jr. did not have a happy childhood. His parents worked long hours in the tavern, leaving their son to take care of himself much of the time. Eventually, when Babe was seven years old, his father took him to St. Mary's Industrial School for Boys, a reformatory and orphanage. There he signed custody over to the Xaverian brothers, Catholic missionaries who ran the school.
During his 12 years at St. Mary's, young George rarely saw his family. They did not come to visit on holidays or on the one Sunday per month when family could visit the boys at the school. Little George was an unruly student, infamously classified as "incorrigible." Much of this was due the young man's inability to adapt to the regimented and structured environment or St. Mary's.
Estrangement from his parents led George, Jr. to find a father figure in Brother Matthias, the Prefect of Discipline at St. Mary's school. Brother Mathias would have a very positive influence on George's life, despite his reputation for unruliness. Brother Matthias, a very large, muscular man, became an inspiration to George in baseball as well as in other aspects of his life. The time spent with Brother Matthias not only helped hone George's swing, but the guidance and encouragement gave him much needed support that would translate into George's unfettering love of children in later years.