On May 30, 2020, an issue with the certificate chain used by Ripple affected client connections to several XRP Ledger public servers that Ripple hosts for development and testing purposes.
The following development servers hosted by Ripple were affected:
- s1.ripple.com
- s2.ripple.com
- r.ripple.com
- repos.ripple.com
- s-west.ripple.com
- s-east.ripple.com
The operational impact of this issue was, primarily, limited to the WebSocket and RPC interfaces provided by those servers; the repos.ripple.com package repository was also affected. At no time during this incident was the XRP Ledger itself impacted, and the network continued to operate normally.
This issue was caused by the AddTrust External CA Root Expiration, which occurred on May 30, 2020, and affected countless other internet services across the world.
Once the certificate expiration issue was detected and confirmed, our technical operations team immediately updated the certificate chain which would, normally, resolve the issue. However, updating the certificate chain revealed an issue in older applications and devices which depend on the expired cross-signed root certificate. In particular, client software based on OpenSSL prior to version 1.1.1 appeared to have broken certificate path validation logic.
As a workaround, our technical operations team purchased a new certificate, from a different certificate authority, and subsequently updated and restarted all affected servers.
This remediation effort fully restored client connections to the following XRP Ledger public servers that Ripple hosts for development and testing purposes:
- s1.ripple.com
- s2.ripple.com
- r.ripple.com
- repos.ripple.com
The following legacy servers are still affected by the certificate issue, and will be deprecated soon:
- s-west.ripple.com
- s-east.ripple.com
The incident also revealed a deficiency in the internal monitoring and alerting capabilities currently in place for certificate expiration. Additional remediation efforts are underway to enhance the capabilities of our internal monitoring of certificates.
