This paper considers a millimeter-wave communication system and proposes an efficient channel estimation scheme with a minimum number of pilots. We model the dynamics of the channel’s second-order statistics by a Markov process and develop a learning framework to obtain these dynamics from an unlabeled set of measured angles of arrival and departure. We then find the optimal precoding and combining vectors for pilot signals. Using these vectors, the transmitter and receiver will sequentially estimate the corresponding angles of departure and arrival, and then refine the pilot precoding and combining vectors to minimize the error of estimating the channel gains.