O365 Hybrid Wizard – Hybrid Agent Fails to Validate

When running the Hybrid Configuration Wizard (HCW), at the point where the Hybrid Agent installs, it installs and registers fine, but then fails the verification step.

The HCW logs (found in %AppData%\Microsoft\Exchange Hybrid Configuration) show the following error:

2019.10.21 13:09:24.552 *ERROR* 10349 [Client=UX, Page=HybridConnectorInstall, Thread=20] The connection to the server '3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://3e6c87b2-dcc8-4e4d-a51e-63dc1ae42acf.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden.., The HTTP request was forbidden with client authentication scheme 'Negotiate'., The remote server returned an error: (403) Forbidden

And the Windows application log might show the following:

Log Name: Application Source: MsExchange BackEndRehydration Date: Event ID: 3002 Task Category: Requests Level: Error Keywords: Classic User: N/A Computer: <Computer Name> Description: Protocol /OAB failed to process request from identity NT AUTHORITY\SYSTEM. Exception: Microsoft.Exchange.Security.Authentication.BackendRehydrationException: Rehydration failed. Reason: Source server 'NT AUTHORITY\SYSTEM' does not have token serialization permission. at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. TryGetCommonAccessToken(HttpContext httpContext, Stopwatch stopwatch, CommonAccessToken& token) at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. ProcessRequest(HttpContext httpContext) at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule. OnAuthenticateRequest(Object source, EventArgs args).

It points to an issue with EWS so check your EWS URLs are set correctly:

Get-WebServicesVirtualDirectory | fl InternalUrl, ExternalURL

It also seems serialization permissions are missing. Running the following commands in the Exchange Mgmt Shell to add the required permissions:

Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"

Get-ClientAccessServer | Add-ADPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"

Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-exch-epi-token-serialization" -User "<domain>\Exchange Servers"

Get-MailboxServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights "ms-Exch-EPI-Impersonation" -User "<domain>\Exchange Servers"

Lastly run an iisreset in an administrative cmd box and rerun the wizard. The agent should now verify correctly.

With thanks to David Robinson of Kick ICT who provided me with this information

Posted

January 30, 2020

Exchange, Microsoft, Office 365

Misja Geuskens

Tags:

Hybrid Wizard, O365

Comments

32 responses to “O365 Hybrid Wizard – Hybrid Agent Fails to Validate”

David

April 21, 2020

Thank you! Worked for me.

Reply
Stephan

June 10, 2020

This worked, thank you

Reply
Lars Juhl

June 19, 2020

Thank you ! Worked for me too, just ran the commands and iisreset

Reply
Yusuf

July 3, 2020

really worked perfectly

Reply
Ram Lan

July 29, 2020

What should be the EMS command – Should try this or is it something different. My Exchange Server is EX2019.

Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -Administrator “ramlan\ex2019”

Reply
1. Misja Geuskens
  
  July 29, 2020
  
  Hi Ram,
  
  your EMS command would look like this:
  
  Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “ramlan\Exchange Servers”
  
  Provided your Active Directory domain is ‘RAMLAN’
  
  Regards,
  Misja
  
  Reply
Ram Lan

July 29, 2020

I was able to get permission details for above against User Administrator and it is FALSE FALSE (Deny & Inherited). So do you think, I should be fine with Hybrid Configuration run?

Reply
Ram Lan

July 29, 2020

Thanks for the info. I was able to run the command and the result is FALSE FALSE (Deny and Inherited) against -User Exchange Servers

Will do iis reset and try hybrid.

Ram

Reply
1. Misja Geuskens
  
  July 31, 2020
  
  Hi Ram,
  
  the user is “%domain%\Exchange Servers” (in your case “ramlan\Exchange servers”). Mind the quotes as there is a space in ‘Exchange Servers’
  
  Regards,
  Misja
  
  Reply
Nicolas

November 11, 2020

+1 for the working solution. Thanks!

Reply
Erwin

January 22, 2021

+1 work for me as well.. Thanks..!

Reply
Magnus Rönnberg

March 5, 2021

Check IIS request filtering, if its installed
Allow svc extensions
Or else, youre in trouble

Reply
1. Misja Geuskens
  
  March 6, 2021
  
  Hi Magnus,
  
  thanks for your response. Can you elaborate on your comment please? I tried to research it myself, but I can’t find any relationship between my blog and your comment.
  
  Cheers
  
  Reply
Joe

April 2, 2021

worked for me as well

Exchange 2013 onprem to Exchange online 365

Reply
Cameron Hancock

August 25, 2021

Thanks, I just setup 3 hybrids with this error!

Reply
Jonathan PREVOT

January 19, 2022

Thanks so much, worked like a charme on Exchange Server 2019 CU10 Jan22SU

Reply
qiT DS

February 15, 2022

thanks, its works.

Reply
Sheldon

April 11, 2022

Thanks that worked perfectly!!

Reply
Brett

October 6, 2022

Still working in 2022. Thanks for taking the time to write this up.

Reply
Raj

January 14, 2023

Thanks, given solution resolved the issue.

Reply
Andy

February 8, 2023

Thanks for this! Sorted straight away

Reply
Thiyagu Mani

March 18, 2023

In My case user/ or group ‘domainname\exchangeservername’ was not found.

Reply
1. Misja Geuskens
  
  March 18, 2023
  
  Hi, the user is “%domain%\Exchange Servers”, not “%domain%\exchangeservername”. “Exchange Servers” is not a placeholder but the actual name of the AD object.
  
  Reply
Tony

May 17, 2023

Thank you very much! These steps worked for me as well.

Reply
Mike

June 21, 2023

I have been trying to figure out this error for 6 months. I even have a ticket in with Microsoft and they could not help. Thank you for this!

Reply
Raj

March 4, 2024

Hi We are on Exchange server 2019, running above command shows below, please assist.

[PS] C:\Windows\system32>Get-ClientAccessServer | Add-AdPermission -AccessRights ExtendedRight -ExtendedRights “ms-exch-epi-token-serialization” -User “ourdomain\Exchange Servers”??
WARNING: The Get-ClientAccessServer cmdlet will be removed in a future version of Exchange. Use the Get-ClientAccessService cmdlet instead. If you have any scripts that use the Get-ClientAccessServer cmdlet, update them to use the
Get-ClientAccessService cmdlet. For more information, see http://go.microsoft.com/fwlink/p/?LinkId=254711.
The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input.
+ CategoryInfo : InvalidArgument: (exchange server:PSObject) [Add-ADPermission], ParameterBindingException
+ FullyQualifiedErrorId : InputObjectNotBound,Add-ADPermission
+ PSComputerName : exchange server.domain.qld.gov.au

Reply
1. Misja Geuskens
  
  March 4, 2024
  
  Hi Raj,
  
  based on information provided, one of the parameters used after the pipeline (|), might no match what is shown in the ‘Get-ClientAccessServer’. So run ‘Get-ClientAccessServer’ and check the output.
  
  Furthermore, provided information suggests that the ‘domain’ might not be correct. Use the NETBIOS name for your AD domain. Open a cmd box and type ‘set l’ and note the ‘userdomain’
  
  Lastly, make sure, if you copy the command from my website, you should substitute the “ with a “. Sometimes when you copy from a website, quotation marks are replaced with different quotation marks, which aren’t recognized in powershell
  
  Regards,
  Misja
  
  Reply
Dan Powell

April 25, 2024

Seemed to work perfectly for me.

Thank you for sharing!!!!

Reply
itzik Dahan

September 8, 2024

Hi,
I have Exchange 2016 and I’m getting the same error *ERROR* 10349 but without the error “The remote server returned an error: (403) Forbidden”
just this error “Error details: The open operation did not complete within the allotted timeout of 00:00:50. The time allotted to this operation may have been a portion of a longer timeout.. Result=Failed SupportsCutover=False}”
I’m afraid to run the commands you wrote that might damage the Exchange server. Is there a way to verify beforehand if these permissions exist?

Reply
1. Misja Geuskens
  
  September 9, 2024
  
  Hi Itzik,
  
  use the get-* command (without the ‘| add-ADPermission’) and filter on ADPermissions, to see what permissions are currently set
  
  Regards,
  Misja
  
  Reply
Desired2Learn

November 27, 2024

I am receiving Error:

“Couldn’t resolve the user or group “DOMAIN\Exchange Servers.” If the user or group is a foreign forest principal,
you must have either a two-way trust or an outgoing trust.
+ CategoryInfo : InvalidOperation: (:) [Add-ADPermission], LocalizedException
+ FullyQualifiedErrorId : [Server=COMPUTERNAME,RequestId=1d341b84-e28c-4bfd-8162-b19456eed2cd,TimeStamp=11/27/2
024 9:48:52 PM] [FailureCategory=Cmdlet-LocalizedException] 2BC299B3,Microsoft.Exchange.Management.RecipientTasks.
AddADPermission
+ PSComputerName : COMPUTERNAME.DOMAIN.local”

I removed my domain name and computer name that I am working with.

Reply
1. Misja Geuskens
  
  November 28, 2024
  
  Hi,
  
  “Couldn’t resolve the user or group “DOMAIN\Exchange Servers.” => Replace ‘Domain’ with your actual NETBIOS domain name and leave ‘Exchange Servers’ in place as is. ‘Exchange Servers’ is an actual domain group
  
  Example “Windtraders\Exchange Servers”
  
  Reply