This paper proposes a convenient way to do a complete Bayesian analysis of bivariate time series generated by auto-regressive moving average models. The identification, estimation, diagnostic checking, and forecasting phases of time series analysis is done by referring to the appropriate posterior or predictive distribution. Using either a matrix normal-Wishart prior density, or a Jeffreys' vague prior, which is combined with an approximate conditional likelihood function, the proposed identification technique is based on the matrix posterior t distribution of the coefficients of a bivariate ARMA model. The coefficients of the process are then tested to be zero by a series of matrix t tests to identify a tentative model. Once the tentative model is chosen, diagnostic checking tests are done by doing some overfitting tests to achieve an adequate model. The parameters of the adequate model are estimated by using the matrix t and Wishart distributions. Finally, forecasting future observations is done by using the multivariate t distribution.