Problem: Server 2022 with Exchange on it

[rjk]

Hello,

does anybody sucessfully uses BatchPatch to update Windows 2022 server with an Exchange server on it?
Windows 2022 Server without Exchange works fine. Also all other kind of Windows platforms, and we are patching lots of systems with BP…

We have three W2022 with exchange on it, that show this problem:
The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) occours on most of “Get information” and “Windows Updates” tasks in BatchPatch.

We are running the latest version of BP 2023.7.30.13.51
And also we RTFM about this error. 😉

Some more informations about that:
Get CPU Info does´t work (0x800706BA)
Get Comupter model works fine
Server 2022 without Exchange work fine! (eg file server, domain controller, sql server)
Older servers 2016 witch Exchange 2016 also work fine with BatchPatch.
This error only occurs on Server 2022 with Exchange 2019
When we stop AD topology service it works, but this is a bad solution…
All servers are VMs running under ESX

Has anybody here the same problem an eventually an solution.

Friendly regards
Ralf

[doug]

We don’t have 2022 with Exchange 2019 to be able to test this, but based on what you’ve described, I can at least note that the issue that you’re encountering is specific to remote WMI calls in BatchPatch. To be clear, ‘Get CPU info’ is a remote WMI call in BP, but ‘Get computer model’ is actually executed through PsExec without using a remote WMI call. The Windows Update actions in BatchPatch all have a remote WMI call in the beginning of them, hence why they are behaving the same as ‘Get CPU info’ in this case.

Normally “RPC server is unavailable” is caused by a firewall or other security-related software that is blocking/dropping the remote connection attempt because it means that BP isn’t even getting a response from the target computer, just like if that target computer didn’t even exist at all. You said that simply disabling the ‘Active Directory Topology’ service is enough to cause things to start working again. This is quite surprising to hear since one wouldn’t expect that the ‘Active Directory Topology’ service would be involved in behavior that is similar to a firewall. However, clearly whatever is occurring is only affecting remote WMI connections and not other remote connections (because PsExec operations are still working, for example), so perhaps it’s just some type of bug in that service. Or perhaps it’s even some type of security-hardening behavior in the service that’s there intentionally to block remote WMI connections from being made to the Exchange server.

I think if I were troubleshooting this issue, one thing I’d want to test is can WMI be used locally if not remotely. I’m not sure if this would lead to any kind of solution/resolution, but it might at least give a bit more detail about where the problem is occurring. A free utility like WMI Explorer could be used locally on the machine and remotely to see if locally works when remotely doesn’t work. (Remotely it most likely should behave the same as BP, working when BP works, and not working when BP does not work). This might simply help you gather more information about the nature of the problem in case you end up reporting a bug to Microsoft (since the bug report would be that remote WMI connections are broken by the Active Directory Topology service, not that BatchPatch is broken by the service).

I’d also want to test starting and stopping that Active Directory Topology service numerous times, with attempts to use BP each time, just to ensure that the behavior is really somehow tied to that service. Like maybe when that service is started, somehow it modifies the Windows firewall behavior on that machine? Or is there anything that might be occurring with IP addresses, network interfaces, and Exchange load balancing that somehow is causing a request to the target computer to get a response out of a different network interface instead of the same interface that received the request… but only when the service is running? Also test putting the IP address instead of the host name into the BatchPatch grid and see if that makes a difference (and if your Exchange server has more than one IP, try each IP separately from BP to see if any of them behave differently). I’d also then want to see if there is anything in the event log on the target computer that gives any more information about what’s occurring. Also enable logging on the Windows Firewall on the target computer and see if it shows anything helpful about connection attempts being dropped when the AD Topology service is running vs stopped.

[rjk]

Hello Doug,

thank you for your detailed answer!
I have some news about that.

Or perhaps it’s even some type of security-hardening behavior in the service that’s there intentionally to block remote WMI connections from being made to the Exchange server.

I think that’s the point. On your advice, I tried to reach the EXCH server using WMI Explorer:
1.) Within the customer LAN (DC with WMI Explorer -> EXCH server): works
2.) From the BaPa server outside via WAN: as you suspected, same error as in BaPa
This would indicate the firewall, but then why does it work remotely with DC and APP Server?

In BP we gernerally use the IP adress instead of hostname.
Also the windows firewall isn´t active. There is a hardware firewall infront.
The eventlog shows absolutly nothing that helps
The system we use for testing has only 1 network card and consequently 1 IP adress

Which of the high ports actually uses BatchPatch? According to the log files of the hardware firewall, this happens dynamically as we have seen. Each customer has different ports in the log. We then opened all ports over 1024 as a test and taadaaa: batchPatch works again with the Exchange servers.
But this gets around the real problem.
I suspect it has to do with the hardening of the Server2022/EXCH 2019 by an SU, that blocks remote WMI connections. However, I haven’t yet discovered which registry key or setting is the culprit.

I will keep you informed.

Friendly regards
Ralf

[doug]

I don’t know enough details about your WAN vs LAN setup to make a comment about your particular environment, but from your description everything sounds like it’s working properly and as expected. You modified your hardware firewall to get things working, which indicates that the problem has nothing to do with the server itself, and this is exactly what we would expect. If you’re saying that you’re using BP over the WAN to the DC and APP servers without encountering the same issue as with your Server 2022/Exchange 2019 machine, then it would indicate that you have previously carved out allowances in the firewall for the DC and APP machines but not for the Exchange machine.

Typically BatchPatch is used on a LAN and not over a WAN because WAN connections are typically locked down (as they often should be) to be more secure. We would not normally advise using BP over a WAN unless the WAN connection is established over a VPN. In most environments, unless the WAN connection is a site-to-site VPN, admins would typically run BP in the target LAN, and the WAN connection would only be used for establishing a remote desktop connection in order to operate the BP instance in the remote LAN. From a security perspective I can’t comment on your setup because I don’t know the details of how you have everything setup and the particular use-case for your WAN, but I mention this only to note that what you’ve described about your usage of BP across a WAN sounds unusual and potentially concerning from a security perspective. Obviously you and your team will decide what is and is not appropriate from a security perspective in your environment. I only wanted to mention it to bring it to your attention in case it wasn’t already.

WMI connections operate over dynamic, not static ports. In hardware firewalls this type of allowance is typically made as DCE/RPC rather than just allowing every port over 1024. See here for more ( https://batchpatch.com/batchpatch-ports )

[Author]

Posts