It shouldn't make any difference whatsoever.
If you choose not to divulge your telephone number to the DWP as is my wont, that is your prerogative. Indeed, I would strongly advise claimants not to deal with any benefit-related matters with the DWP over the telephone, as there is a distinct possibility that vital evidence/information will go missing or will be misconstrued.
Nothing on this board constitutes legal advice - always consult a professional about specific problems