I saw this thread earlier when it was posted, but been super slammed with orders. This is most definitely an aftermarket mirror based on this wire colors, sleeving, and overall finish of the wire. Can you explain the pink wire that is out? Was it like this or did you do this?
The middle row of pins (5 pins total) are your circuits for - Mirror Heat (Pin #8), Mirror Motor Up/Down Control (Pin #9), Mirror Motor Left/Right Control (Pin #10), Ground (Pin #11), Driver Mirror Motor Common (Pin # 12). The empty slot is position #7 which is not used (for mirror memory) in this mirror.
The top (can only see a red wire in your pic) and the bottom row (all empty), are usually the lighting circuits.
What I would do - is disconnect the passenger mirror and focus on driver only mirror first. With the driver mirror only plugged in, what happens? While you're at it, can you confirm the part # of the switch you are using?
Let's keep the troubleshooting here, so it will be visible to everyone.
I have seen aftermarket mirrors that have their motor wires flipped... so this will take some troubleshooting.