When Encryption Goes Wrong and RDS is Inaccessible

Recently, a colleague had an issue where an encrypted RDS SQL database had lost access to the KMS service. To clarify, RDS (Amazon Relational Database Service) is AWS’s PaaS database service and KMS is their Key Management Service.  When an RDS database instance needs to be encrypted at rest, KMS keys would be used to encrypt the database.

AWS services don’t offer 100% availability. Everything fails. When RDS loses that connection to KMS, the error inaccessible-encryption-credentials would be displayed. My assumption was that once the services are able to reestablish connection everything would come back online. However, this is not the case.

When a KMS key is unavailable to an RDS instance, either by revoking key access, or if a configured key expiration date is reached, the RDS instance may report an “inaccessible-encryption-credentials” error resulting in it becoming inaccessible.  When the instance is in this state, there is no connection or ability to modify the instance by the end user (or AWS for that matter). It is then considered being in a terminal state.

That’s right…terminal. Meaning you need to turn to backups of the RDS instance (because you have those, right? Along with a tested process…) and perform a restore from backup/snapshot or performing point-in-time restore. Note, the original key must be available for the restore to be successful.

Oddly enough, other RDS instances using the same KMS key (which was not deleted or expired) had no issue. Additionally, this was a Development RDS instance, so the outage was not considered critical.

Finally, if you find yourself with any AWS or possible business continuity/disaster recovery needs, please feel free to reach out to us with any questions. We would love to help you out.