I am strongly against any phone conversations with case managers for numerous reasons. The main reason is that it puts you at an unfair advantage by having to instantly respond to a question when you may not have all the details, or you may not be able to verbally communicate the answer you want (due to anxiety or other problems). Imagine yourself later on telling an appeals committee “That isn’t what I meant!” when they play a recording of your call. Like you said, you felt confused at the end of a call. This is the same reason why you would never allow yourself to be interogated by a professional police officer without having a lawyer present. The case managers are experts at this and know how to get the information they want.
So I think it’s great of you to be proactive in this and eliminate phone calls entirely. As for how to phrase it, simply make it about you and not him. Tell him that you have a lot of anxiety talking on the phone and request that he send you the questions via email or postal mail. If he replies that he cannot do that and it is mandatory for a phone conversation, kindly ask him to send you the policy that says that. If he continues to push, just stand your ground and refuse to take the phone call. There is nothing he can do. Keep a paper trail of it all. Be warned that a few (small percentage) of case managers may take offense to this and begin a campaign of harrassment. Sort of like “Okay, you don’t want to take phone calls? Fine. Then every month forever I will request a full medical documentation from your doctor and ALL your specialists.” If that happens you have recourse. They cannot harrass you by demanding excess medical information. If your condition is permanent with little hope for change, then you can get a doctor to write you a letter saying that only X month (e.g. 6 month follow ups) are required and same with specialists, and then the case manager really has no power.