After their inception in the late 1980s, recurrent neural networks (RNNs) as a sequence computing model have seen mushrooming interests in communities of natural language processing, speech recognition, computer vision, etc. Recent variations of RNNs have made breakthroughs in fields such as machine translation where machines can achieve human level quality. RNNs assisted speech recognition technology is providing services on subtitles for live streaming videos. In this survey, we will offer a historical perspective by walking through the early years of RNNs all the way to their modern forms, detailing their most popular architectural designs and, perhaps more importantly, demystify the mathematical aspect behind their memory behaviors.