Since the start of the Convolutional Neural Networks (CNN) paradigm, they were applied in a wide range of computer vision tasks such as image classification, object detection, localization, tracking and action recognition where they were able to show breakthrough performance and generate a new state of the art results. This paper surveys the progress of CNN from an architectural and optimization perspective. While many CNN reviews exist in the literature, most of them had focused on providing a survey either from a network architecture prospective or an application one, unlike this one which provides a brief general overview for the key features of CNN, followed by reviewing the progress of the state of the art architectures and finally considers the change in the merit of figure of how the CNN are evaluated to include the optimization methods to provide practical CNN that can be deployed on today's hardware infrastructure without significantly impacting the achieved accuracy.