O365 Hybrid Wizard – Hybrid Agent Fails to Validate

When running the Hybrid Configuration Wizard (HCW), at the point where the Hybrid Agent installs, it installs and registers fine, but then fails the verification step.

The HCW logs (found in %AppData%\Microsoft\Exchange Hybrid Configuration) show the following error:

2019.10.21 13:09:24.552 *ERROR* 10349 [Client=UX, Page=HybridConnectorInstall, Thread=20] The connection to the server '3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden.., The HTTP request was forbidden with client authentication scheme 'Negotiate'., The remote server returned an error: (403) Forbidden

And the Windows application log might show the following:

Log Name: Application
Source: MsExchange BackEndRehydration
Date:
Event ID: 3002
Task Category: Requests
Level: Error
Keywords: Classic
User: N/A
Computer: <Computer Name>
Description:
Protocol /OAB failed to process request from identity NT AUTHORITY\SYSTEM. Exception: Microsoft.Exchange.Security.Authentication.BackendRehydrationException: Rehydration failed. Reason: Source server 'NT AUTHORITY\SYSTEM' does not have token serialization permission.
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. TryGetCommonAccessToken(HttpContext httpContext, Stopwatch stopwatch, CommonAccessToken& token)
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. ProcessRequest(HttpContext httpContext)
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. OnAuthenticateRequest(Object source, EventArgs args).


It points to an issue with EWS so check your EWS URLs are set correctly:

Get-WebServicesVirtualDirectory | fl InternalUrl, ExternalURL

It also seems serialization permissions are missing. Running the following commands in the Exchange Mgmt Shell to add the required permissions:


Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"​​

Get-ClientAccessServer | Add-ADPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"​​

Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"​​

Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"

Lastly run an iisreset in an administrative cmd box and rerun the wizard. The agent should now verify correctly.

With thanks to David Robinson of Kick ICT who provided me with this information

25 comments

  1. David says:

    Thank you! Worked for me.

  2. Stephan says:

    This worked, thank you

  3. Lars Juhl says:

    Thank you ! Worked for me too, just ran the commands and iisreset

  4. Yusuf says:

    really worked perfectly

  5. Ram Lan says:

    What should be the EMS command – Should try this or is it something different. My Exchange Server is EX2019.

    Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -Administrator “ramlan\ex2019”

    • Hi Ram,

      your EMS command would look like this:

      Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “ramlan\Exchange Servers”​​

      Provided your Active Directory domain is ‘RAMLAN’

      Regards,
      Misja

  6. Ram Lan says:

    I was able to get permission details for above against User Administrator and it is FALSE FALSE (Deny & Inherited). So do you think, I should be fine with Hybrid Configuration run?

  7. Ram Lan says:

    Thanks for the info. I was able to run the command and the result is FALSE FALSE (Deny and Inherited) against -User Exchange Servers

    Will do iis reset and try hybrid.

    Ram

    • Hi Ram,

      the user is “%domain%\Exchange Servers” (in your case “ramlan\Exchange servers”). Mind the quotes as there is a space in ‘Exchange Servers’

      Regards,
      Misja

  8. Nicolas says:

    +1 for the working solution. Thanks!

  9. Erwin says:

    +1 work for me as well.. Thanks..!

  10. Check IIS request filtering, if its installed
    Allow svc extensions
    Or else, youre in trouble

    • Hi Magnus,

      thanks for your response. Can you elaborate on your comment please? I tried to research it myself, but I can’t find any relationship between my blog and your comment.

      Cheers

  11. Joe says:

    worked for me as well

    Exchange 2013 onprem to Exchange online 365

  12. Cameron Hancock says:

    Thanks, I just setup 3 hybrids with this error!

  13. Jonathan PREVOT says:

    Thanks so much, worked like a charme on Exchange Server 2019 CU10 Jan22SU

  14. qiT DS says:

    thanks, its works.

  15. Sheldon says:

    Thanks that worked perfectly!!

  16. Brett says:

    Still working in 2022. Thanks for taking the time to write this up.

  17. Raj says:

    Thanks, given solution resolved the issue.

  18. Andy says:

    Thanks for this! Sorted straight away

  19. Thiyagu Mani says:

    In My case user/ or group ‘domainname\exchangeservername’ was not found.

    • Hi, the user is “%domain%\Exchange Servers”, not “%domain%\exchangeservername”. “Exchange Servers” is not a placeholder but the actual name of the AD object.

  20. Tony says:

    Thank you very much! These steps worked for me as well.

  21. Mike says:

    I have been trying to figure out this error for 6 months. I even have a ticket in with Microsoft and they could not help. Thank you for this!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.