Deepwank's comments about him on reddit.
This is a tough question to answer in a short comment, but I'll try to give a brief summary. Perelman himself felt a deep sense of isolation from the mathematical community. He was polite, but firm in his insistence that he did not want to be a figurehead or a representative of the community. He had an ideal picture in his mind: 1) people do mathematics because they love it, and this love should transcend jealousy or rivalry; 2) credit will always be given when it's due, and mathematicians should acknowledge it and share it when it's appropriate; 3) it is far better to publish a good result rarely than mediocre results often.
Perelman realized that none of these principles actually held true in the mathematical community. He realized that the way one gets a post-doc or permanent position had to do a lot with string-pulling and secret phone calls in the background, rather than by the merit of one's work. If you don't ally yourself with an influential person or kiss the appropriate ass, you won't get a position, and this is what he realized. You may read in his wiki biography that he rejected a lot of positions in the US after proving the soul conjecture in 1994, but he actually applied for these same positions immediately after receiving his PhD and was rejected. One can argue that he wasn't yet accomplished enough, but that's a shady argument, since the people that were getting these same positions had arguably accomplished less than Perelman at the time. By the mid-nineties, he was already jaded, and word was that he saw a taste of his colleagues' jealousy in the Soviet Union which he couldn't really understand, though you and I probably would. His conception of doing mathematics was unreasonably pure: if you really love mathematics, you celebrate the results of others, even if they beat you to the punch and prove something you've been working on your whole life.
The credit aspect is shaky too. Mathematicians are always fighting over credit for various results, and due to the timeless nature of mathematical results, citation is extremely important. But as soon as Perelman posted his work on the arXiv, it was a race between the other leading geometers to basically fill in the details of Perelman's proof, make it more readable, and publish it as soon as possible. Make no mistake, it was clear that this was Perelman's work being repackaged and everyone knew it, but those geometers couldn't resist the credit they would receive for merely explaining what Perelman did to other geometers. And hell why not, after all it's yet another publication in an excellent journal.
Which brings me to the last point, and this one is far more poisonous to mathematics. In an effort to maximize publication lists, people publish crap. Lemmas become propositions, and propositions become theorems. It's sad thatthe number of publications has any bearing on your worth as a mathematician. It's more understandable in fields like engineering, computer science, biology, and even physics. But proving a worthwhile mathematical theorem takes a long fucking time, even for geniuses. So what do most career mathematicians do? They publish whatever they can get away with. Perelman returned to the Steklov Institute after proving the soul conjecture for a simple research position in order to quietly go about doing math. He was working on the Poincare Conjecture the whole time he was there, and in the meantime didn't publish at all. Rumor has it (and it's a very plausible rumor) that the department chairman at Steklov threatened to fire Perelman for not publishing anything in years. It isn't clear whether Perelman was waiting to finish the proof or if he was boycotting math journals altogether, but it took him from 1995 to 2002 to finish his work. He was irritated with having his position threatened and for his bosses to only care about the quantity of his research output rather than the quality.
However, he still cared enough about mathematics to give a series of lectures and talks after he proved Poincare, to try to explain to other mathematicians how his use of Ricci flow solved the problem. It took the leading geometers 3 years to really understand his proof, upon which the community wanted to shower him with awards. But, Perelman was already ticked off with the mathematical community and his own boss at Steklov, that he quit his position less than a year after posting his proof to the arXiv, after he had done his US circuit explaining his work.
Rejecting the prizes wasn't the action of a troubled genius or a mentally unstable person. He just didn't want to represent the same group that had betrayed the principles they themselves had set. That being said, it may be likely he was a troubled genius or mentally unstable. But in all accounts of him refusing the prizes, he was calm, polite, and firm. There was no emotion or wavering.
And while the layman dismisses Perelman's refusals of the prize as foolish, most mathematicians do not. It is because deep down we realize that our system is broken and corrupted, that there aren't enough jobs to go around, and many people with serious talent are either quitting or changing fields because of how jobs are handed out. Those who mock Perelman for living with his mother, tending her flowerbed, are free to do so. I assure you Perelman doesn't care. He has nothing left to prove.