Hi Caz,
There are really 2 aspects to this.
The first is that you need to clear this with DWP and confirm that it
would be classed as Permitted Work. This must be done before starting such work.
The second aspect is whether this could count against you at a future reassessment. It is not possible to give a definitive answer here but I think you should consider it a definite possibility. The closer the work conforms to your stated limitations, the more trouble it is likely to cause.
Permitted Work