How to reset MongoDB replicate set settings after changing hostnames
Recently, I changed the hostnames of my MongoDB servers, all of which happened to be part of a replica set. This was a big mistake as I first should have unconfigured the replica set by starting each member without the --replset option, updated the replica set configuration manually, changed the hostnames, and reboot the servers. Since I didn't do this, I instead had to learn from my mistake and do it after the fact.
Essentially, what I did was drop the local database from each replica set member, then run through the process to recreate the replica set. This was easy to do since each mongoDB instance was starting in a standalone server mode since it was not able to determine the members of the previously defined replica set (which contained the old hostnames).
The best option would have been to just remove the local.system.replset collection to ensure the replica set configuration was wiped instead of dumping the entire local database, but I knew I could drop local without impacting my environment, so please consider this if you run into the same situation. You can delete the local database one of two ways:
1. stop mongodb on each node, then literally remove the local directory from the file system in the defined dbpath.
2. start mongo without the --replSet option and use db.dropdatabase on the local database (or use external link below to update system.replset)
Once the local.system.replset collection is empty, or you've dropped the local database across all of your members, restart MongoDB on each server, and go through the replica set configuration process again as documented on mongodb.org.
To avoid this altogether, use the well documented procedure on changing hostnames provided by MongoDB (linked below)