Microsoft has launched a repair for a harebrained Trade Server bug that shut down on-premises mail supply world wide simply as clocks have been chiming within the new 12 months.
The mass disruption stemmed from a date test failure in Trade Server 2016 and 2019 that made it unimaginable for servers to accommodate the 12 months 2022, prompting some to name it the Y2K22 bug. The mail applications saved dates and occasions as signed integers, which max out at 2147483647, or 231 – 1. Microsoft makes use of the primary two numbers of an replace model to indicate the 12 months it was launched. So long as the 12 months was 2021 or earlier, the whole lot labored positive.
“What within the absolute hell Microsoft?”
When Microsoft launched model 2201010001 on New 12 months’s Eve, nonetheless, on-premises servers crashed as a result of they have been unable to interpret the date. Consequently, messages bought caught in transport queues. Admins world wide have been left frantically attempting to troubleshoot as an alternative of ringing within the New 12 months with family and friends. All they needed to go on have been two cryptic log messages that seemed like this:
Log Title: Utility Supply: FIPFS Logged: 1/1/2022 1:03:42 AM Occasion ID: 5300 Stage: Error Pc: server1.contoso.com Description: The FIP-FS "Microsoft" Scan Engine didn't load. PID: 23092, Error Code: 0x80004005. Error Description: Cannot convert "2201010001" to lengthy.
Log Title: Utility Supply: FIPFS Logged: 1/1/2022 11:47:16 AM Occasion ID: 1106 Stage: Error Pc: server1.contoso.com Description: The FIP-FS Scan Course of failed initialization. Error: 0x80004005. Error Particulars: Unspecified error.
“What within the absolute hell Microsoft!?” one admin wrote on this Reddit thread, which was one of many first boards to report the mass failure. “On New 12 months’s Eve!? First place I test is Reddit and also you guys save my life earlier than we even get an engineer on the telephone.”
The following day, Microsoft launched a repair. It is available in two varieties: an automatic PowerShell script, or a handbook resolution within the occasion the script didn’t work correctly, as reported by some admins. In both case, the fixes should be carried out on each on-premises Trade 2016 and Trade 2019 server inside an affected group. The automated script can run on a number of servers in parallel. The software program maker stated the automated script “would possibly take a while to run” and urged admins to be affected person.
The date and time test was carried out when Trade checked the model of the FIP-FS, a scanning engine that’s a part of Trade antimalware protections. As soon as FIP-FS variations started with the numbers 22, the test was unable to finish, and mail supply was abruptly halted. The repair stops the Microsoft Filtering Administration and Microsoft Trade Transport providers, deletes present AV engine information, and installs and begins a patched AV engine.
By Monday, issues have been getting again to regular for a lot of affected organizations. It’s not clear how lengthy the buggy date storage had been in place, however judging from the 2 affected variations, it was presumably launched when Trade Server 2016 was below growth.