RSA authentication is based on public key cryptography. The idea is that there are two encryption keys, one for encryption and another for decryption. It is not possible (on human time scale) to derive the decryption key from the encryption key. The encryption key is called the public key, because it can be given to anyone and it is not secret. The decryption key, on the other hand, is secret, and is called the private key.
RSA authentication is based on the impossibility of deriving the private key from the public key. The public key is stored on the server machine in the user's $HOME/.ssh/authorized_keys file. The private key is only kept on the user's local machine, laptop, or other secure storage. Then the user tries to log in, the client tells the server the public key that the user wishes to use for authentication. The server then checks if this public key is admissible. If so, it generates a 256 bit random number, encrypts it with the public key, and sends the value to the client. The client then decrypts the number with its private key, computes a 128 bit MD5 checksum from the resulting data, and sends the checksum back to the server. (Only a checksum is sent to prevent chosen-plaintext attacks against RSA.) The server checks computes a checksum from the correct data, and compares the checksums. Authentication is accepted if the checksums match. (Theoretically this indicates that the client only probably knows the correct key, but for all practical purposes there is no doubt.)
The RSA private key can be protected with a passphrase. The passphrase can be any string; it is hashed with MD5 to produce an encryption key for IDEA, which is used to encrypt the private part of the key file. With passphrase, authorization requires access to the key file and the passphrase. Without passphrase, authorization only depends on possession of the key file.
RSA authentication is the most secure form of authentication supported by this software. It does not rely on the network, routers, domain name servers, or the client machine. The only thing that matters is access to the private key.
All this, of course, depends on the security of the RSA algorithm itself. RSA has been widely known since about 1978, and no effective methods for breaking it are known if it is used properly. Care has been taken to avoid the well-known pitfalls. Breaking RSA is widely believed to be equivalent to factoring, which is a very hard mathematical problem that has received considerable public research. So far, no effective methods are known for numbers bigger than about 512 bits. However, as computer speeds and factoring methods are increasing, 512 bits can no longer be considered secure. The factoring work is exponential, and 768 or 1024 bits are widely considered to be secure in the near future.