> Correct me if I'm wrong, but the signed server-identifying cert is swapped in TLS before the connection is encrypted, no?

Only on old TLS versions. TLS 1.3 changed it so the server certificate is also encrypted.