Recently, a colleague had an issue where an encrypted RDS SQL database had lost access to the KMS service. To clarify, RDS (Amazon Relational Database Service) is AWS’s PaaS database service and KMS is their Key Management Service. When an RDS database instance needs to be encrypted at rest, KMS keys would be used to encrypt the database.
AWS services don’t offer 100% availability. Everything fails. When RDS loses that connection to KMS, the error inaccessible-encryption-credentials would be displayed. My assumption was that once the services are able to reestablish connection everything would come back online. However, this is not the case.
When a KMS key is unavailable to an RDS instance, either by revoking key access, or if a configured key expiration date is reached, the RDS instance may report an “inaccessible-encryption-credentials” error resulting in it becoming inaccessible. When the instance is in this state, there is no connection or ability to modify the instance by the end user (or AWS for that matter). It is then considered being in a terminal state.
That’s right…terminal. Meaning you need to turn to backups of the RDS instance (because you have those, right? Along with a tested process…) and perform a restore from backup/snapshot or performing point-in-time restore. Note, the original key must be available for the restore to be successful.
Oddly enough, other RDS instances using the same KMS key (which was not deleted or expired) had no issue. Additionally, this was a Development RDS instance, so the outage was not considered critical.
Finally, if you find yourself with any AWS or possible business continuity/disaster recovery needs, please feel free to reach out to us with any questions. We would love to help you out.
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
|cookielawinfo-checbox-analytics||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".|
|cookielawinfo-checbox-functional||11 months||The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".|
|cookielawinfo-checbox-others||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.|
|cookielawinfo-checkbox-necessary||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".|
|cookielawinfo-checkbox-performance||11 months||This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".|
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.