So I kind of understand the basics of key exchanging, vendors have their key on their profile and when they send you a message you would use that key to decrypt their encrypted message.
And if you want to send them a message like with your address you encrypt it with one of your keys and you would give them the key to decrypt it.
But how is this safe? If a site were to be raided wouldn't the keys also be raided with the messages meaning they would be able to be decoded instantly?
You got this the wrong way round. Public keys are for encryption only.
The public key allows anyone to encrypt a message that can only be decrypted by the private key. If a vendor sends you an encrypted message (they will have used your public key) then you use your private key to decrypt.
Since the websites only store the public key they cannot be used to decrypt a message. All decryption is done off-site.