The number of people with mental illness is increasing because of stress or environmental influences. They require close monitoring because of their unpredictable behaviors; however, this is challenging given the lack of adequate numbers of medical staff. To overcome this problem, we propose a personalized abnormal behavior recognition alarm system for closed wards. The proposed system utilizes real-time video analysis to detect and track the locations of the patients, enabling recognition of their abnormal behaviors. In addition, new definitions are provided for specific abnormal behaviors that commonly occur in closed wards, with the adaptation of continual learning in the system. This architecture allows the creation of an abnormal behavior dataset while enhancing the recognition accuracy. The average abnormal behavior recognition accuracy with this system is over 92%. According to test results in real hospitals, about 84% of the medical staff were satisfied with the proposed system. Through the proposed alarm system, the staff could implement immediate actions without careful monitoring. By reducing the probability of occurrence of dangerous incidents, the system not only benefits the health of the patients but also enhances the working environment of the medical staff.