On Thursday morning, 3/20/2025, iSalus experienced an issue that resulted in multiple servers being in an unhealthy state. Some of the processes running on the servers crashed due to an out of memory condition. The result of this was intermittent errors and slowness attempting to use the OfficeEMR application for a period of approximately 45 minutes. The impacted processes/servers had to be restarted to return to a healthy operating state. Normal system performance resumed once these actions were completed and user traffic began balancing out across the servers.
As a preventative action, we increased the amount of memory on the servers and also updated how it was being allocated across the various processes to provide sufficient overhead and ensure that this error state is not encountered again. This was completed the evening of 3/20/2025. Long term, we are also working to replace the process that crashed and ultimately caused the issues in order to improve overall system reliability.
While we do actively monitor overall memory usage on the servers, this event did not trigger those alerts based on the fact that the memory was quickly used and then released when the process ultimately crashed. That being said, we will be implementing other alerts which would have allowed us to reduce the duration of the incident.
Please contact Support with any further questions. Thank you.