Soon after, his parents sent him to the Minnesota State Training School October 11, 1903. While there, he was repeatedly beaten, tortured, and raped by staff members in what attendees dubbed "The Painting House", because children would leave "painted" with bruises and blood. Panzram hated this place of torture so much that he decided to burn it down, and did so without detection. In late 1905, Panzram was released from the school. By his teens, he was an alcoholic and was repeatedly in trouble with the authorities, often for burglary and theft. He ran away from home at the age of 14 in January 1906 to become a hobo on the rails . He often traveled via train cars; he later claimed that on one train he was gang raped by a group of hobos.
In his autobiography, Panzram wrote that he was "rage personified" and that he would often rape men whom he had robbed. He was noted for his large stature and great physical strength-due to years of hard labor at Leavenworth and other prisons - which aided him in overpowering most men he encountered. He also engaged in vandalism and arson. By his own admission, one of the few times he did not engage in criminal activities was when he was employed as a strikebreaker against union employees. On one occasion, he tried to sign aboard as a ship's steward on an Army transport vessel, but was discharged when he reported to work intoxicated.