Well that is a huge conflict. It would be traumatizing to have to be examined by this Doctor again. You have definite grounds that are fully understandable to not be examined by this Doctor.
I don’t see how your Insurer could justify a need that this Doctor be retained for an IME.
You should secretly record any IME if you are not legally represented so you can make accurate notes of what was said and what occurred for your records.
Before you go to any IME ask for copies of any and all documents that are being sent to the Doctor and copies of any intake forms that the Doctor would have you fill out prior. Request with proof of receipt. Include a request that a copy of the IME report be sent to you at the same time it is released to your Insurer and offer to pay for photo-copying.