Janet
Based on your post you are only receiving Contribution Based ESA only, despite what you have been told by the DWP, however, you may have been eligible for Income Related ESA during the period in question.
ESA(IR) is dependant on your household income, your savings and assets, and any hours that your partner works. See
Asset rule for ESA(IR) 24 hour work rule for ESA(IR) If you think that you are then contact the DWP, they had a duty to assess you for ESA(IR) when they moved you to ESA, but in many cases failed to do so.
If you are/were eligible then it should be possible to claim the fine back.
Gordon