Remarkable progress on single image super-resolution (SISR) has been achieved with deep convolutional neural network (CNN) based approaches. These methods usually divide the images into high-frequency (HF) and low-frequency (LF) components and mainly recover the high-frequency component in a supervised manner. However, only simple interpolation manners, such as bilinear and bicubic, are utilized in the LF components recovery process, which limits the performance of these SISR approaches. We argue that the recovery of LF components also plays important roles in SISR, and to relieve the problem, we propose an iterative two-stream network (ITS-Net) which recovers the LF and HF components with convolution operations, respectively, thus better high-resolution images can be obtained. To achieve this, we utilize a sub-network with convolution and deconvolution operations to recover the LF components, and an iterative learning strategy is used to obtain well recovered LF and HF components. Extensive experiments on various benchmarking datasets demonstrate the effectiveness of our approach comparing with state-of-the-art CNN based approaches.