SharePoint 2013: Cannot follow - (403) Forbidden

Symptoms

When trying to follow a site or a document a generic error occurs in the UI. In the ULS logs, the error is:
“The remote server returned an error: (403) Forbidden”

Cause

Customer had a publishing / consuming farm scenario where team sites were located in one farm (consuming) and the UPA and Mysite web application was in another farm (publishing). When you try to follow something, a client-side object model (CSOM) call is made to your mysite. In this scenario, this is a server to server (S2S) call and we need S2S authentication (Oauth) configured between the two farms for it to succeed.
Customer had not configured the S2S trust correctly.
In both farms, the NameIdentifier for SPSecurityTokenServiceConfig and the NameId and RegisteredIssuerName for SPTrustedSecurityTokenIssuer were still set to SharePointOAuthID@FarmID. Example:
00000003-0000-0ff1-ce00-000000000000@bf1cf54e-2890-4b45-a4c1-97005a0f757e
These should be set to SharePointOAuthID@RealmName
Example: 00000003-0000-0ff1-ce00-000000000000@SP2013S2S

Resolution

Run these commands in both farms:
Get-SPAuthenticationRealm
Get-SPTrustedSecurityTokenIssuer
Get-SPSecurityTokenServiceConfig
-- Get-SPAuthenticationRealm should return the realm name the customer has set.
-- Get-SPTrustedSecurityTokenIssuer should show a TrustedSecurityTokenIssuer that has NameId and RegisteredIssuerName set to 00000003-0000-0ff1-ce00-000000000000@<RealmName>
-- Get-SpSecurityTokenServiceConfig should show that NameIdentifier is set to 00000003-0000-0ff1-ce00-000000000000@<RealmName>
If the values are not correct for NameIdentifier, NameId, and RegisteredIssuerName, then you will need to remove the TrustedSecurityTokenIssuers from both farms, set SPSecurityTokenServiceConfig correctly, and then re-create the TrustedSecurityTokenIssuers.
1. Remove the TrustedSecurityTokenIssuers.a. Run Get-SPTrustedSecurityTokenIssuer. Note the ID of the one you want to remove
b. Run Remove-SPTrustedSecurityTokenIssuer -id <The ID of the one you want to remove>
c. Go into Central Admin | Security | Manage Trust and remove the S2S trust that was created when New-SPTrustedSecurityTokenIssuer was run. Leaving these extra trusts shouldn’t be a problem, but this is a good cleanup step.
2. Set up the S2S trust correctly by following these steps to recreate connection:

Change the farm authentication realm:
-- This needs to be done in Publishing and Consuming Farm(s)
The farms share a realm name, which must be set to the same value on all publishing and consuming farms.

Run the following on all farms, using the same realm name for both:
Set-SPAuthenticationRealm -realm <Realm Name>
Update the Security Token Service Config
This step updates the STS with the realm named shared between the publishing / consuming farms
-- This needs to be done in Publishing and Consuming Farm(s)
$sts=Get-SPSecurityTokenServiceConfig
$Realm=Get-SpAuthenticationRealm
$nameId = "00000003-0000-0ff1-ce00-000000000000@$Realm"
Write-Host "Setting STS NameId to $nameId"
$sts.NameIdentifier = $nameId
$sts.Update()
iisreset

Create new Trusted Security Token Issuer
This step creates a new Trusted Security Token Issuer on each farm
-- Run on Publishing and Consuming farms
Run this step last to create the final TrustedSecurityTokenIssuer’s, and note that the names specified for <SomeName> need not match between publishing and consuming farms.

On the consuming farm run:
New-SPTrustedSecurityTokenIssuer -name <SomeName> -MetadataEndPoint https://<PublishingFarmWebApp>/_layouts/15/metadata/json/1
On the publishing farm run:
New-SPTrustedSecurityTokenIssuer -name <SomeName> -MetadataEndPoint https://<ConsumingFarmWebApp>/_layouts/15/metadata/json/1
Note:
<PublishingFarmWebApp> and <ConsumingFarmWebApp> can be any SSL-enabled web application of the farm that will be sending server-to-server requests.

More Information

Here are the details of what you would find in the ULS logs when you run into this problem. Please note that you need to look at the logs on both the consuming and publishing sides:

-- From the Consuming farm (where you are trying to follow something):
w3wp.exe (0x156C) 0x17F4 SharePoint Portal Server User Profiles agb0e Medium SPS2SAppContext: Executing remote query to https://mysite.contoso.com/personal/userName/ with SourceID fda11a2b-5292-4181-8d51-cf0946acb5c1
w3wp.exe (0x156C) 0x17F4 SharePoint Foundation Application Authentication aiv3e High Self-issued token request for '00000003-0000-0ff1-ce00-000000000000/mysite.contoso.com@<RealmName>' succeeded.
w3wp.exe (0x156C) 0x17F4 SharePoint Portal Server Content Following afilq Unexpected FollowedContent.FollowItem:Exception:System.Net.WebException: The remote server returned an error: (403) Forbidden. at System.Net.HttpWebRequest.GetResponse() at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute() at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb) at Microsoft.Office.Server.UserProfiles.FollowedContentProxy.Execute(String methodName) at Microsoft.Office.Server.UserProfiles.FollowedContent.FollowItem(FollowedItem item, Boolean isInternal)

-- Meanwhile on the UPA / MySite (publishing) farm, I see this:
w3wp.exe (0x0868) 0x25EC SharePoint Foundation Application Authentication ahkpt Medium SPApplicationAuthenticationModule Authentication finished successfully for user:0#.w|contoso\userName and actor:0i.t|ms.sp.ext|00000003-0000-0ff1-ce00-000000000000@cbc091f7-bec4-44f2-a4e7-13d66c447d22
w3wp.exe (0x0868) 0x25EC SharePoint Foundation Authentication Authorization ag6al Medium OAuth app principal Name=i:0i.t|ms.sp.ext|00000003-0000-0ff1-ce00-000000000000@cbc091f7-bec4-44f2-a4e7-13d66c447d22, IsAppOnlyRequest=False, UserIdentityName=0#.w|contoso\userName, ClaimsCount=166
w3wp.exe (0x0868) 0x23A0 SharePoint Foundation Authentication Authorization adhp1 High The app principal I:0I.T|MS.SP.EXT|00000003-0000-0FF1-CE00-000000000000@CBC091F7-BEC4-44F2-A4E7-13D66C447D22 does not exists.
w3wp.exe (0x0868) 0x23A0 SharePoint Foundation Authentication Authorization adhqj High Unable to get tenant permission for app I:0I.T|MS.SP.EXT|00000003-0000-0FF1-CE00-000000000000@CBC091F7-BEC4-44F2-A4E7-13D66C447D22. hr=-2130575150
w3wp.exe (0x0868) 0x23A0 SharePoint Foundation Authentication Authorization adhqm High Failed to initilize the m_pSiteAppPrincipal -2130575150
w3wp.exe (0x0868) 0x23A0 SharePoint Foundation General 8kh7 High The app principal does not exist.
w3wp.exe (0x0868) 0x23A0 SharePoint Foundation General aix9j High SPRequest.OpenWebInternal: UserPrincipalName=i:0).w|s-1-5-21-527237240-1682526488-1417001333-670443, AppPrincipalName=I:0I.T|MS.SP.EXT|00000003-0000-0FF1-CE00-000000000000@CBC091F7-BEC4-44F2-A4E7-13D66C447D22 ,bstrUrl=https://mysite.contoso.com/personal/userName/_vti_bin/client.svc/ProcessQuery
w3wp.exe (0x0868) 0x23A0 SharePoint Foundation General ai1wu Medium System.Runtime.InteropServices.COMException: The app principal does not exist., StackTrace: at Microsoft.SharePoint.SPWeb.InitWebPublic() at Microsoft.SharePoint.SPContext.DefaultKey(HttpContext context, SPWeb web) at Microsoft.SharePoint.SPContext.get_Current() at Microsoft.SharePoint.Client.SPClientServiceHost.OnBeginRequest() at Microsoft.SharePoint.Client.ClientRequestServiceImpl.ProcessQuery(Stream inputStream, IList`1 pendingDisposableContainer) at Microsoft.SharePoint.Client.ClientRequestService.ProcessQuery(Stream inputStream) at SyncInvokeProcessQuery(Object , Object[] , Object[] ) at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
w3wp.exe (0x0868) 0x23A0 SharePoint Foundation CSOM ajeo1 High The app principal i:0i.t|ms.sp.ext|00000003-0000-0ff1-ce00-000000000000@cbc091f7-bec4-44f2-a4e7-13d66c447d22 does not exist, sending 403