The public key cryptosystem is one of the protection systems used to secure communication between computer terminals. Since no technique exists to prove that an encryption scheme is secure, the only test available is to see whether anyone can think of a way to break it. This paper outlines a selection of attacks that have been used and explains some of the basic tools available to the cryptanalyst. We presented survey of a collection of protocols in which the level of security is not actually attained, not because of a failure of the encryption algorithm used, but rather because of shortcomings in the design of the protocol. Guidelines will be extracted from the analysis of these protocols. A czyptanalytic attack on the use of short RSA secret exponent is described. The attack makes use of an algorithm based on continued fractions that find the numerator and denominator of a fraction in a polynomial time when a closed enough estimate of the fraction is known. The public exponent e and the modulus n = pq can be used to create an estimate of a fraction.