When uploading your contacts the secret isn't your or their phone number. It's the relationships. The phone book never listed all of your friends and family.