When running the Hybrid Configuration Wizard (HCW), at the point where the Hybrid Agent installs, it installs and registers fine, but then fails the verification step.
The HCW logs (found in %AppData%\Microsoft\Exchange Hybrid Configuration) show the following error:2019.10.21 13:09:24.552 *ERROR* 10349 [Client=UX, Page=HybridConnectorInstall, Thread=20] The connection to the server '3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden.., The HTTP request was forbidden with client authentication scheme 'Negotiate'., The remote server returned an error: (403) Forbidden
And the Windows application log might show the following:Log Name: Application
Source: MsExchange BackEndRehydration
Date:
Event ID: 3002
Task Category: Requests
Level: Error
Keywords: Classic
User: N/A
Computer: <Computer Name>
Description:
Protocol /OAB failed to process request from identity NT AUTHORITY\SYSTEM. Exception: Microsoft.Exchange.Security.Authentication.BackendRehydrationException: Rehydration failed. Reason: Source server 'NT AUTHORITY\SYSTEM' does not have token serialization permission.
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. TryGetCommonAccessToken(HttpContext httpContext, Stopwatch stopwatch, CommonAccessToken& token)
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. ProcessRequest(HttpContext httpContext)
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. OnAuthenticateRequest(Object source, EventArgs args).
It points to an issue with EWS so check your EWS URLs are set correctly:
Get-WebServicesVirtualDirectory | fl InternalUrl, ExternalURL
It also seems serialization permissions are missing. Running the following commands in the Exchange Mgmt Shell to add the required permissions:
Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"
Get-ClientAccessServer | Add-ADPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"
Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"
Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"
Lastly run an iisreset in an administrative cmd box and rerun the wizard. The agent should now verify correctly.
With thanks to David Robinson of Kick ICT who provided me with this information
Thank you! Worked for me.
This worked, thank you
Thank you ! Worked for me too, just ran the commands and iisreset
really worked perfectly
What should be the EMS command – Should try this or is it something different. My Exchange Server is EX2019.
Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -Administrator “ramlan\ex2019”
Hi Ram,
your EMS command would look like this:
Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “ramlan\Exchange Servers”
Provided your Active Directory domain is ‘RAMLAN’
Regards,
Misja
I was able to get permission details for above against User Administrator and it is FALSE FALSE (Deny & Inherited). So do you think, I should be fine with Hybrid Configuration run?
Thanks for the info. I was able to run the command and the result is FALSE FALSE (Deny and Inherited) against -User Exchange Servers
Will do iis reset and try hybrid.
Ram
Hi Ram,
the user is “%domain%\Exchange Servers” (in your case “ramlan\Exchange servers”). Mind the quotes as there is a space in ‘Exchange Servers’
Regards,
Misja
+1 for the working solution. Thanks!
+1 work for me as well.. Thanks..!
Check IIS request filtering, if its installed
Allow svc extensions
Or else, youre in trouble
Hi Magnus,
thanks for your response. Can you elaborate on your comment please? I tried to research it myself, but I can’t find any relationship between my blog and your comment.
Cheers
worked for me as well
Exchange 2013 onprem to Exchange online 365
Thanks, I just setup 3 hybrids with this error!
Thanks so much, worked like a charme on Exchange Server 2019 CU10 Jan22SU
thanks, its works.
Thanks that worked perfectly!!
Still working in 2022. Thanks for taking the time to write this up.
Thanks, given solution resolved the issue.
Thanks for this! Sorted straight away
In My case user/ or group ‘domainname\exchangeservername’ was not found.
Hi, the user is “%domain%\Exchange Servers”, not “%domain%\exchangeservername”. “Exchange Servers” is not a placeholder but the actual name of the AD object.
Thank you very much! These steps worked for me as well.
I have been trying to figure out this error for 6 months. I even have a ticket in with Microsoft and they could not help. Thank you for this!
Hi We are on Exchange server 2019, running above command shows below, please assist.
[PS] C:\Windows\system32>Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “ourdomain\Exchange Servers”??
WARNING: The Get-ClientAccessServer cmdlet will be removed in a future version of Exchange. Use the Get-ClientAccessService cmdlet instead. If you have any scripts that use the Get-ClientAccessServer cmdlet, update them to use the
Get-ClientAccessService cmdlet. For more information, see http://go.microsoft.com/fwlink/p/?LinkId=254711.
The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input.
+ CategoryInfo : InvalidArgument: (exchange server:PSObject) [Add-ADPermission], ParameterBindingException
+ FullyQualifiedErrorId : InputObjectNotBound,Add-ADPermission
+ PSComputerName : exchange server.domain.qld.gov.au
Hi Raj,
based on information provided, one of the parameters used after the pipeline (|), might no match what is shown in the ‘Get-ClientAccessServer’. So run ‘Get-ClientAccessServer’ and check the output.
Furthermore, provided information suggests that the ‘domain’ might not be correct. Use the NETBIOS name for your AD domain. Open a cmd box and type ‘set l’ and note the ‘userdomain’
Lastly, make sure, if you copy the command from my website, you should substitute the “ with a “. Sometimes when you copy from a website, quotation marks are replaced with different quotation marks, which aren’t recognized in powershell
Regards,
Misja