Automating Weekly Restarts
by Hisyam Darwis
We are seeing a Keycloak docker image unresponsive after a prolong period of action. We also assume that there is a memory leak from the older version of Keycloak. The resolution is always through restarting of the image. However, we cannot determine how long the active session will last until a reboot is required.
In lieu of the above, there is a need to implement auto restart of the Keycloak app, and various other modules within the SAFWA E-Tender eco-system.
Our aim is to implement an automation process to restart Keycloak Docker instances on prod-webapp 1a and prod-webapp 1b EC2 machines each Thursday evening. The selected day allows two days of preparation for troubleshooting and recovery in case of problems or difficulties occurring during the automation.
- Process Execution:
Assessment of Keycloak Docker Memory Leak: Due to a memory leak on the Keycloak Docker, it has been determined that implementing an automation process for regular restarts is necessary to increasing system stability, ensuring instances scalability, utilizing resource efficiency, and improving docker performance.
- Selection of Restart Schedule:
Thursday evening was selected as the preferred time to perform the reboot process. This decision is based on the rationale that it allows two days (Friday and Saturday) for the SysOps team to troubleshoot and recover from any potential issues or problems that may arise during or after the automation.
- Automated Execution:
On the first implementation date, which was today, the SysOps team performed the automation process at 11:00 p.m. The process was successful and produced the desired outcome.
- Testing and Verification:
To ensure the reliability of the automation, the Support team conducted tests in incognito mode to eliminate any potential cache-related disruptions. The test results confirmed that the automation was functioning smoothly and producing the expected output.
- Monitoring and Maintenance:
The SysOps and Support team will continue to monitor the automation process regularly to ensure its ongoing effectiveness. In the event of any deviations or unexpected behaviour, appropriate troubleshooting measures will be taken promptly to maintain the system’s stability and performance.
By following this proper procedure, Schinkels’ SysOps team has successfully implemented an automation process to restart the Keycloak Docker instances on prod-webapp 1a and prod-webapp 1b EC2 machines every Thursday night. This proactive approach helps mitigate the impact of any potential issues, providing the team with ample time to troubleshoot and recover, if necessary.
