Work-focused interviews can if you prefer be conducted by telephone at the discretion of the ESA personal adviser, so you may wish to request WFIs using this method?
Insofar as work-related activity arises, this is arranged through an external provider on behalf of the Secretary of State and the rules for attendance have not as yet been established, but if you are called for interview with the provider, you will need to explain to them the circumstances whereby you would find it difficult to attend their office.
Nothing on this board constitutes legal advice - always consult a professional about specific problems