Millimeter wave communications rely on narrow-beam transmissions to cope with the strong signal attenuation at these frequencies, thus demanding precise beam alignment between transmitter and receiver. The communication overhead incurred to achieve beam alignment may become a severe impairment in mobile networks. This paper addresses the problem of optimizing beam alignment acquisition, with the goal of maximizing throughput. Specifically, the algorithm jointly determines the portion of time devoted to beam alignment acquisition, as well as, within this portion of time, the optimal beam search parameters, using the framework of Markov decision processes. It is proved that a bisection search algorithm is optimal, and that it outperforms exhaustive and iterative search algorithms proposed in the literature. The duration of the beam alignment phase is optimized so as to maximize the overall throughput. The numerical results show that the throughput, optimized with respect to the duration of the beam alignment phase, achievable under the exhaustive algorithm is 88.3% lower than that achievable under the bisection algorithm. Similarly, the throughput achievable by the iterative search algorithm for a division factor of 4 and 8 is, respectively, 12.8% and 36.4% lower than that achievable by the bisection algorithm.