Overpayment letter

Shocked yesterday to get a demand for 2.5K ESA to be repaid!
One year ago I had to attend a phone call where I was told the occupational pension I had taken out should have been declared. The pension was £61 per month. After the phone call they reduced my ESA by approx £25 A month. They said that was it, all sorted.
Now I have a letter saying I didn't declare the pension so have a civil fine of £50 as well as nearly £2.5 K to repay.
The tone of the letter is hostile as though I have been cheating.
Apparently I didn't make a reasonable excuse although being in the support group and having a severe stress disorder affecting mental functioning was explained.
I can't see any way they have got to 2.5 K so I'm going to contact them but wondered if anyone had any extra perspective for me on this. Thanks.