O365 Hybrid Wizard – Hybrid Agent Fails to Validate

When running the Hybrid Configuration Wizard (HCW), at the point where the Hybrid Agent installs, it installs and registers fine, but then fails the verification step.

The HCW logs (found in %AppData%\Microsoft\Exchange Hybrid Configuration) show the following error:

2019.10.21 13:09:24.552 *ERROR* 10349 [Client=UX, Page=HybridConnectorInstall, Thread=20] The connection to the server '3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden.., The HTTP request was forbidden with client authentication scheme 'Negotiate'., The remote server returned an error: (403) Forbidden

And the Windows application log might show the following:

Log Name: Application
Source: MsExchange BackEndRehydration
Date:
Event ID: 3002
Task Category: Requests
Level: Error
Keywords: Classic
User: N/A
Computer: <Computer Name>
Description:
Protocol /OAB failed to process request from identity NT AUTHORITY\SYSTEM. Exception: Microsoft.Exchange.Security.Authentication.BackendRehydrationException: Rehydration failed. Reason: Source server 'NT AUTHORITY\SYSTEM' does not have token serialization permission.
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. TryGetCommonAccessToken(HttpContext httpContext, Stopwatch stopwatch, CommonAccessToken& token)
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. ProcessRequest(HttpContext httpContext)
at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. OnAuthenticateRequest(Object source, EventArgs args).


It points to an issue with EWS so check your EWS URLs are set correctly:

Get-WebServicesVirtualDirectory | fl InternalUrl, ExternalURL

It also seems serialization permissions are missing. Running the following commands in the Exchange Mgmt Shell to add the required permissions:


Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"​​

Get-ClientAccessServer | Add-ADPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"​​

Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"​​

Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"

Lastly run an iisreset in an administrative cmd box and rerun the wizard. The agent should now verify correctly.

With thanks to David Robinson of Kick ICT who provided me with this information

27 comments

  1. David says:

    Thank you! Worked for me.

  2. Stephan says:

    This worked, thank you

  3. Lars Juhl says:

    Thank you ! Worked for me too, just ran the commands and iisreset

  4. Yusuf says:

    really worked perfectly

  5. Ram Lan says:

    What should be the EMS command – Should try this or is it something different. My Exchange Server is EX2019.

    Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -Administrator “ramlan\ex2019”

    • Hi Ram,

      your EMS command would look like this:

      Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “ramlan\Exchange Servers”​​

      Provided your Active Directory domain is ‘RAMLAN’

      Regards,
      Misja

  6. Ram Lan says:

    I was able to get permission details for above against User Administrator and it is FALSE FALSE (Deny & Inherited). So do you think, I should be fine with Hybrid Configuration run?

  7. Ram Lan says:

    Thanks for the info. I was able to run the command and the result is FALSE FALSE (Deny and Inherited) against -User Exchange Servers

    Will do iis reset and try hybrid.

    Ram

    • Hi Ram,

      the user is “%domain%\Exchange Servers” (in your case “ramlan\Exchange servers”). Mind the quotes as there is a space in ‘Exchange Servers’

      Regards,
      Misja

  8. Nicolas says:

    +1 for the working solution. Thanks!

  9. Erwin says:

    +1 work for me as well.. Thanks..!

  10. Check IIS request filtering, if its installed
    Allow svc extensions
    Or else, youre in trouble

    • Hi Magnus,

      thanks for your response. Can you elaborate on your comment please? I tried to research it myself, but I can’t find any relationship between my blog and your comment.

      Cheers

  11. Joe says:

    worked for me as well

    Exchange 2013 onprem to Exchange online 365

  12. Cameron Hancock says:

    Thanks, I just setup 3 hybrids with this error!

  13. Jonathan PREVOT says:

    Thanks so much, worked like a charme on Exchange Server 2019 CU10 Jan22SU

  14. qiT DS says:

    thanks, its works.

  15. Sheldon says:

    Thanks that worked perfectly!!

  16. Brett says:

    Still working in 2022. Thanks for taking the time to write this up.

  17. Raj says:

    Thanks, given solution resolved the issue.

  18. Andy says:

    Thanks for this! Sorted straight away

  19. Thiyagu Mani says:

    In My case user/ or group ‘domainname\exchangeservername’ was not found.

    • Hi, the user is “%domain%\Exchange Servers”, not “%domain%\exchangeservername”. “Exchange Servers” is not a placeholder but the actual name of the AD object.

  20. Tony says:

    Thank you very much! These steps worked for me as well.

  21. Mike says:

    I have been trying to figure out this error for 6 months. I even have a ticket in with Microsoft and they could not help. Thank you for this!

  22. Raj says:

    Hi We are on Exchange server 2019, running above command shows below, please assist.

    [PS] C:\Windows\system32>Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “ourdomain\Exchange Servers”??
    WARNING: The Get-ClientAccessServer cmdlet will be removed in a future version of Exchange. Use the Get-ClientAccessService cmdlet instead. If you have any scripts that use the Get-ClientAccessServer cmdlet, update them to use the
    Get-ClientAccessService cmdlet. For more information, see http://go.microsoft.com/fwlink/p/?LinkId=254711.
    The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input.
    + CategoryInfo : InvalidArgument: (exchange server:PSObject) [Add-ADPermission], ParameterBindingException
    + FullyQualifiedErrorId : InputObjectNotBound,Add-ADPermission
    + PSComputerName : exchange server.domain.qld.gov.au

    • Hi Raj,

      based on information provided, one of the parameters used after the pipeline (|), might no match what is shown in the ‘Get-ClientAccessServer’. So run ‘Get-ClientAccessServer’ and check the output.

      Furthermore, provided information suggests that the ‘domain’ might not be correct. Use the NETBIOS name for your AD domain. Open a cmd box and type ‘set l’ and note the ‘userdomain’

      Lastly, make sure, if you copy the command from my website, you should substitute the “ with a “. Sometimes when you copy from a website, quotation marks are replaced with different quotation marks, which aren’t recognized in powershell

      Regards,
      Misja

Leave a Reply to Raj Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.