This paper presents a simulation of two-element microstrip antenna array fed by a Butler matrix over electromagnetic bandgap (EBG) substrate. The proposed structure represents a simple smart antenna system working in the ISM band at 5.8 GHz. Small size antenna arrays over high dielectric substrates place the study of mutual coupling problem in an important priority, as it affects the performance of the system. To compensate for mutual coupling effect, mushroom-like EBG structure is used due its winning features. EBG cells are inserted between the two antennas to reduce the mutual coupling effect and to increase the array gain. The performance of the structure with conventional substrate of relative
permittivity εr = 10.2 is compared to that with EBG substrate. The simulation results show that changing the number of EBG columns and rows affects the value of mutual coupling reduction and gain enhancement. Results also indicate that the optimum dimensions of the EBG structure depend on the number of EBG columns and rows. A 26 dB reduction of
mutual coupling is achieved by using 3 columns and 5 rows of EBG structure. Also, a 1.5 dB improvement of array gain is obtained for the case of 3 columns and 7 rows.