For E2EE apps like Signal what stops the server from giving you a fake public key for a user?

Say I want to send a message to Alice. To encrypt my message to Alice doesn't Signal have to send me her public key? What stops them from sending me a fake public key? I believe that at some point in the handshake process I probably sign something that validates my public key and she does the same. But couldn't the server still just replace with its own keys during this- so trust is required for at least initial contact?

I'm asking this, because assuming that its true, would for example using a custom signal client that additionally encrypts with an HKDF-derived key from a passphrase or something that was privately communicated improve security? (Since you don't have to trust Signal servers alone on initial contact)