The above error surfaced recently when logging in at an O365 tenant. While the message itself is a fairly generic one, and can be attributed to a number of possible causes, in this case the organization (using identity federation) had an expired token signing certificate in their AD FS 2.0 configuration. They had automatic certificate rollover disabled on their AD FS farm so that AD FS could not rollover the configuration with new certificates. For some organizations, with web applications such as Sharepoint 2010, this may be a desirable configuration, with the process of certificate rollover being manually administered, rather than AD FS silently and automatically handling it.
Hindsight is a wonderful thing in this case and paying heed to warning messages about expiring certificates in the event logs in the preceding weeks may well avert some nasty surprises. However, calamities happen and should this problem evade your radar in the future, looking at what recovery options are available may be useful.
When the token signing certificate is due to expire (2-3 weeks before), the AD FS 2.0 Admin Event Log will begin to blurt out warning messages (Event ID:385).
AD FS 2.0 detected that one or more certificates in AD FS configuration database need to be updated manually because they are expired, or will expire soon. See additional details for more information
Token-signing certificate with thumbprint ‘4E8F5194309E48EDB7EF8440B47082FB3D1594B6’ is set to expire on 12/30/2011 1:00:00 AM.
Should the warnings be ignored, and the certificate has expired, the Event ID 385 warning changes to:
Token-signing certificate with thumbprint ‘4E8F5194309E48EDB7EF8440B47082FB3D1594B6’ expired on 12/30/2011 1:00:00 AM.
At this point the AD FS wheels are wobbling and an Event ID 381 may start popping up:
An error occurred during an attempt to build the certificate chain for configuration certificate identified by thumbprint ‘4E8F5194309E48EDB7EF8440B47082FB3D1594B6’. Possible causes are that the certificate has been revoked or certificate is not within its validity period.
The following errors occurred while building the certificate chain:
MSIS2013: A required certificate is not within its validity period when verifying against the current system clock.
Ensure that the certificate is valid and has not been revoked or expired.
This may be followed closely behind by Event ID 102:
There was an error in enabling endpoints of Federation Service. Fix configuration errors using PowerShell cmdlets and restart the Federation Service.
System.ArgumentNullException: Value cannot be null.
Parameter name: certificate
at System.IdentityModel.Tokens.X509SecurityToken..ctor(X509Certificate2 certificate, String id, Boolean clone, Boolean disposable)
at System.IdentityModel.Tokens.X509SecurityToken..ctor(X509Certificate2 certificate)
at Microsoft.IdentityServer.Service.Configuration.MSISSecurityTokenServiceConfiguration.Create(Boolean forSaml)
at Microsoft.IdentityServer.Service.SecurityTokenService.STSService.StartProxyPolicyStoreService(ServiceHostManager serviceHostManager)
at Microsoft.IdentityServer.Service.SecurityTokenService.STSService.OnStartInternal(Boolean requestAdditionalTime)
As the original title graphic intimated, federated logon at this point is not working.
Depending on your AD FS configuration you may have automatic certificate rollover enabled. This can be checked via Get-ADFSProperties
In this particular case it was set to:
This meant that that once the signing token certificate expired, a replacement was not automatically assigned. In this case a new certificate needs to be provisioned manually. Of course, you could also elect to activate the autocertificaterollover function.
Preparing to make the change
Whether you elect to go the manual or the automatic route, in either case, you’ll still have some investigation to do. Before nuking your AD FS configuration, consider the impact of assigning / replacing the token signing/decryption certificates in AD FS.
- Who are the claims providers and relying parties involved?
- Can they process metadata?
- What application specific changes are required to support the certificate change?
Of course, if you’re facing a production federation service outage then there’s likely to be other pressures to consider as well , but try and determine which relying parties or claims providers will be able to process the details of the change (automatically) via updated federation metadata exchange or not. In the case of the latter, some sort of manual intervention may be required. Applications, for example, such as Office 365 and Sharepoint 2010 will need to be informed of this change.
Option 1: Manual Rollover
Via this method, you can generate your own certs using the Self-SSL feature built into IIS 7.x, or use Keytool, OpenSSL etc.. Alternatively, you may have your own internal PKI. I don’t see any benefit in using a third-party certificate for token signing encryption/decryption. No doubt someone out there may have a valid reason for doing so, but self-signed certs / internal PKI serve the purpose in this scenario. With IIS, under Service Certificates, I opted to create a self-signed certificate
Give the certificate a recognizable “friendly” name and IIS creates the certificate.
If you’ve elected to use an internal PKI, then there may be a certificate chain involved. Claims providers or relying parties will require a copy of the corresponding chain (.P7B) file or individual CA (.cer) files. The AD FS service account needs to be able to manage the private key of the certificate. This can be accomplished through the MMC Certificates|Computer snap-in. Right-click over the certificate:
Select Manage Private Keys and give the AD FS service account Full Control to the certificate. Now you need to prepare to make the change.
Option 2: Automatic Rollover
To configure automatic rollover:
Set-ADFSProperties -AutoCertificateRollover $true
To trigger an automatic replacement of the existing certificates the following needs to be entered.
BEWARE! By selecting the urgent option you are voluntarily hitting the self-destruct button and hosing any working configuration. Relying parties and claims providers will not have refreshed their federation metadata with you and at this stage will be working against stale metadata. It is best, therefore, to carry out these activities out out-of-hours and co-ordinate with the parties concerned refreshing of metadata. As I discuss later in this post, some parties need manual intervention.
With the automatic rollover option selected, any existing token signing / decrypting certificates will be deprecated and replaced with a fresh one years’ validity alternate.
Exporting certificates to file
Once you’ve added a new token-signing certificate (manual route) or run the Powershell script to set automatic rollover you’ll need to export the certificate to file. Double click on the AD FS Token Signing certificate, click on the Details tab and then select Copy to File. The export format varies according to the target system. Windows systems will generally accept DER and Base64. If it’s a Java web container / application server then use Base64. Ditto with Linux/Unix…
In the examples below, I’ve used Office 365 and Sharepoint 2010 as two examples of web applications that need manual intervention.
Office 365 – MSOL
To rectify the problem of a token signing certificate change in Office 365, we need to update Online Services with new information concerning our certificate. As covered in the post here: https://blog.auth360.net/2011/10/15/ad-fs-2-0-rollup-i-multiple-issuer-support/ AD FS 2.0 now supports multiple UPN suffixes, so you’ll need to run this per UPN domain to update the respective information in Powershell.
Connect-MSOLService -cred $cred
Update-MsolFederatedDomain -DomainName mydomain.com -SupportMultipleDomain
Update-MsolFederatedDomain -DomainName yourdomain.com -SupportMultipleDomain
Substitute your own UPN suffixes with the examples above and the effect should be instantaneous and logon to MSOL possible once the update has been carried out. Run this through your test environment first to make sure you get the desired effect.
Sharepoint throws an error when the token signing certificate is invalid when accessing the Sharepoint Security Token Service at https://sharepoint.mydomain.com/_trust/
An operation failed because the following certificate has validation errors:\n\nSubject Name: CN=ADFS Signing – sts.mydomain.com\nIssuer Name: CN=ADFS Signing – sts.mydomain.com\nThumbprint: 611CBD4AE55FE2F2C3F0648C16C57A231AA51491\n\nErrors:\n\n The root of the certificate chain is not a trusted root authority..
In the above example, I triggered automatic certificate rollover on the AD FS side without notifying Sharepoint of the change and this threw an error (screenshot). To rectify this we’ll need to export the token signing certificate from AD FS instance and import it into Sharepoint 2010. We’ll then assign the register and assign the replacement certificate. This occurs at two levels:
- Registering the certificate as a root authority. The certificate is self-signed and Sharepoint stores its own certificate trust hierarchy, outside of normal Windows conventions. This means that the self-signed certificate, effectively a root certificate, needs to be assigned as a Trusted Root Authority.
- Updating the existing trusted token issuer. This means replacing the existing trusted identity token issuer value for the signing certificate with the new one.
The following Powershell script shows an example configuration.
$certPath = "C:\temp\ADFS Token.cer"
$cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2("$certPath")
$root = New-SPTrustedRootAuthority "AD FS 2.0 STS" -Certificate $cert
$upd = Set-SPTrustedIdentityTokenIssuer -Identity "ENTER THE NAME OF YOUR EXISTING ADFS INSTANCE " -ImportTrustCertificate $cert
Your own configuration may vary and you’ll need to ensure that the identity to be updated is set correctly.
As ever TEST, TEST, TEST!!!!…..