There are risks involved with this job, though it's nowhere near the risks that are involved with a job like welding. The problem is that you are often tied to the celebrities that you manage. If one of them does something bad, like in the case of Louis CK, it's not only the celebrity that goes down, but often the agent too. Not only do they lose revenue from their celebrity not being a hot commodity anymore, they also become a pariah themselves. Whether or not that's right doesn't matter. It's just the truth of the situation.
In Hollywood you're more likely to make money off of actors than from being an actor. This is a prime example of that. If you feel like coming to Hollywood, try teaching acting classes. You'll make money much quicker than you will from being an actor.