Category Archives: Exchange

Is your Exchange Hybrid Server internet-facing? You have likely already been hacked

– This is twice as big as the SolarWinds breach.

– Patching is not enough! If your Exchange Server was open to the internet via TCP 80 or 443 between February 26 and March 3rd (or later) assume it was compromised.

At least 30,000 organizations have had a backdoor installed on their Microsoft Exchange Server (on-premises).

We know it is at least this many because researchers have built an NMAP script to scan the internet for infected hosts.

There is nothing to indicate that Exchange Online has been impacted, but organizations in O365 could still have been hacked because most of those customers still have an internet-facing Exchange Hybrid server.

How to hunt for the existence of a backdoor known as a web shell.
(A web shell is an internet accessible web page that the hacker places on the Exchange Server that gives the attackers administrative access to the Exchange Server)

Indicators of Compromise 

<random_name>.aspx (often 8 characters)

In these directories:

o c:\inetpub\wwwroot\aspnet_client\

o c:\inetpub\wwwroot\aspnet_client\system_web\

o C:\Exchange\FrontEnd\HttpProxy\owa\auth\

o %PROGRAMFILES%\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\

– Signs of large amounts of SMB network traffic from the Exchange Server to internal network

– Scheduled Task on Exchange Server named Winnet

– Check for suspicious .zip, .rar, and .7z files in C:\ProgramData\, which may indicate possible data exfiltration.

– Monitor c:\root and c:\windows\temp for LSASS dumps (attackers used procdump64.exe) or rundll32 C:\windows\system32\comsvcs.dll MiniDump lsass.dmp

In some cases, additional dynamic link libraries (DLLs) and compiled aspx files are created shortly after the  webshells are first interacted with via POST requests in the following locations:

  • C:\Windows\Microsoft.NET\Framework64\<version>\Temporary ASP.NET Files\root\
  • C:\Windows\Microsoft.NET\Framework64\<version>\Temporary ASP.NET Files\owa\

Administrator is removed from the “Exchange Organization administrators” group (credit rapid7)

– Scan Exchange Logs for IOCs (manually here) or with the Microsoft script (here)

– You can also use this NMAP script to see if your servers are vulnerable after patching them.

In the attacks observed, the threat actor used these vulnerabilities to access on-premises Exchange servers which enabled access to email accounts and allowed installation of additional malware to facilitate long-term access to victim environments.

On March 6th, a security researcher created a honeypot with these vulnerabilities and found that it was exploited 5 times in less than 24 hours. This indicates several copy-cat threat actors are already targeting these vulnerabilities.

If you find evidence of compromise, activate your incident response procedures. You may have a legal requirement to notify within 72 hours of sensitive data was accessed in your email or network.

What versions are affected?

– Exchange 2013 Versions < 15.00.1497.012

– Exchange 2016 CU18 < 15.01.2106.013

– Exchange 2016 CU19 < 15.01.2176.009

– Exchange 2019 CU7 < 15.02.0721.013

– Exchange 2019 CU8 < 15.02.0792.010

– Microsoft issued CU 31 for Exchange Server 2010 – best to apply that, but it would be better to upgrade your hybrid server since Exchange 2010 normally does not receive security updates (this was a kind gesture on Microsoft’s part).

How do we prevent this from happening again?

– The only reason an Exchange Hybrid Server should still be internet-facing is if there are still on-premises mailboxes. Move those to the cloud and then shut off internet access to your hybrid server after moving the Autodiscover DNS record to point to

– If you have no on-premises mailboxes, you should close TCP 80/443 after moving Autodiscover to cloud

Learn more: Read the announcement, or view the Exchange blog. We also recommend the Volexity post (here) and Rapid7 post (here).

Use PowerShell to Connect to Exchange Online unattended in a Scheduled Task

If you have MFA enabled, how do you connect to Exchange Online in an unattended script, like a Scheduled Task? Some people may have embedded a password into their scripts, but that will stop working in mid 2021 when Microsoft retires basic authentication in Office 365.

Microsoft has a preview version of Exchange Online v2 PowerShell that allows you to use a Certificate to authenticate.

Why Certificates? Because you don’t want an MFA push notification on your iPhone every morning at 1:00 AM, right?

Recommendation: Review to see if you have any automated scripts connecting to Exchange Online (typically scheduled tasks).

How? Follow the steps below to use certificates to connect to Exchange Online PowerShell


Install-Module ExchangeOnlineManagement -RequiredVersion 2.0.3-Preview

Or if you already have a previous version of the module installed:

Update-Module ExchangeOnlineManagement -RequiredVersion 2.0.3-Preview

Note: If you get an error “A parameter cannot be found that matches parameter name ‘AllowPrerelease’” then run this command
Install-Module PowershellGet -Force (then close and re-open PowerShell)


  1. Register an app in Azure AD (here).
    (The app is the entry point to Exchange Online PowerShell because it creates a service account called a service principal to perform administrative actions)
  2. Click API Permissions on left navigation > Add a permission
  3. Scroll to the bottom of the Request API permissions pane and click on Exchange under the Supported legacy APIs section.
  4. Click on Application permissions

  5. Expand the Exchange entry, and select the Exchange.ManageAsApp permission.
    Click the
    Add permissions button below to complete the operation.
  6. Click “Grant Admin consent for your tenant”
  7. Create a Role to assign to the App (Thanks to Tony Redmond for this tip)

$ExoAppSp = (Get-AzureADServicePrincipal -Filter “DisplayName eq ‘Exchange Online Scripting'”).ObjectId

$ExoRoleId = (Get-AzureADDirectoryRole | ? {$_.DisplayName -eq “Exchange Service Administrator”}).ObjectId

Add-AzureADDirectoryRoleMember -ObjectId $ExoRoleId -RefObjectId $ExoAppSp

  1. Create a self-signed X.509 certificate that will be used for authentication
    New-SelfSignedCertificate -Subject “Exo Background Process” -CertStoreLocation “cert:\CurrentUser\My” -KeySpec KeyExchange -FriendlyName “For EXO V2 Background Jobs”
  2. Open MMC, add Certificates, Find the new Cert, and Export it *without* the private key, as a .CER file.
  3. Upload this file to the app you registered in the Azure Portal

After adding the certificate, we need three items before we can finally connect unattended with PowerShell

  • The AppId of the application you created.
    Get-AzureADApplication -Filter “DisplayName eq ‘Exchange Online Scripting'”
  • The thumbprint of the certificate loaded into the app
    Get-ChildItem -path ‘Cert:\*’ -Recurse |where {$_.Subject -like ‘*EXO*’}
  • The service domain for your tenant (like

With these values, you can connect to Exchange Online using certificate-based authentication with a command like:
Connect-ExchangeOnline -CertificateThumbprint ” 960BD967A9287CE83DF4138805B5CE2FCA4C9B8B” -AppId “b83c46c6-044e-40e5-929c-634f80045a11” -ShowBanner:$false -Organization


· Microsoft Documentation

· Vasil Michev

· Tony Redmond: Office 365 IT Pros

Deploying MailItemsAccessed Audit Event in Office 365

MailItemsAccessed is a new audit event in Office 365 that records when email data is accessed by mail protocols and clients.

Why is MailItemsAccessed so important?

During an investigation where a mailbox has been accessed by an unauthorized party, there are often legal requirements (State, Federal and Global Treaties such as GDPR) to notify individuals if their personally identifiable information was accessed. Without MailItemsAccessed we could only say that the attacker had the capability of accessing all mailbox contents, but we couldn’t say which exact emails were accessed. The sync event is still not as definitive as we would like, but it does show that the attacker now has possession of the mailbox contents. If the attacker accessed the mailbox via a web browser, then it’s helpful to know which individual items were accessed.

If a privileged account was compromised, it’s also a good idea to check whether the attacker enabled the Bypass audit log PowerShell command to cover their tracks.

For more details, see Access to crucial events for investigations.

How does MailItemsAccessed compare to MessageBind?

MailItemsAccessed replaces the audit event ‘MessageBind’ which was deprecated in Exchange Online on 1/23/2019. This audit event began rolling out in Q1 2020 after a 12 month pause from the first announcement in January 2019. Tony Redmond has documented the history of this rollout on his blog, with his latest post on March 6, 2020 (here).

MessageBind was only available for the AuditAdmin user logon type and did not record actions performed by the Mailbox Owner or Delegate. MessageBind only covered actions by a mail client and did not record sync activities. MessageBind also generated multiple audit records for the same email message. MailItemsAccessed fixes all these deficiencies.

MailItemsAccessed applies to all logon types (Owner, Admin and Delegate)

MailItemsAccessed applies to both an individual email being read in addition to a ‘sync’ event such as MAPI, POP or IMAP downloading all email in a client.

MailItemsAccessed aggregates multiple events into fewer audit records.


Office E5 or (M365 E3 + “E5 Compliance” add-on)

One question that is often asked is: “If I buy just one license, does this enable the capability for all users.” The answer is no. Only users with the appropriate license will have the MailItemsAccessed logged.


MailItemsAccessed is only enabled by default when the E5 feature “Microsoft 365 Advanced Auditing” license has been applied to the account.


(In PowerShell the auditing license will appear as “M365_ADVANCED_AUDITING”).

To find out how many of your current mailboxes are logging the MailItemsAccessed event run this Exchange Online PowerShell command:

get-mailbox -ResultSize unlimited | where {$_.AuditOwner -like ‘*Accessed*’}

Note: See troubleshooting section if you have the right license and MailItemsAccessed is still not appearing.

Prior to 2/1/2019, Mailbox Owner auditing only logged a single event by default: MailboxLogin. After 2/1/2019, additional events were added unless auditing had been customized. If you customized the mailbox actions to audit for any logon type before mailbox auditing on by default was enabled in your organization, the customized settings are preserved on the mailbox and aren’t overwritten by the default mailbox actions that were since added. The exception to this rule seems to be with MailItemsAccessed because it was appended to the E5 mailboxes that had been set to use customized audit events.

To reset auditing to defaults you can run use the DefaultAuditSet parameter, which is generally recommended because according to the documentation “Any new mailbox actions that are released by Microsoft are automatically added to the list of audited actions for the logon type.”

Set-Mailbox -Identity (mailbox identity) –DefaultAuditSet Admin,Delegate,Owner

If you want to find out the mailboxes that have the default auditing enabled, run this command and look for “Admin, Delegate, Owner.” The absence of one of these means that audit customizations was applied to that mailbox.

get-mailbox | select name,DefaultAuditSet


For example, in the screen shot above (1) means that all three roles were customized and (2) means only the Owner was customized because it is absent from the list.

Note: DefaultAuditSet does not audit all possible audit events. It will audit the following seven events (or eight when an E5 license is applied to the mailbox)

1. Update

2. MoveToDeletedItems

3. SoftDelete

4. HardDelete

5. UpdateFolderPermissions

6. UpdateInboxRules

7. UpdateCalendarDelegation

8. MailItemsAccessed (<- THIS IS ONLY ADDED IF AN E5 License is applied to the mailbox)

For a list of the defaults see the documentation (here).

The following three events can be added as additional audit events for the Owner logon type:

1. Create

2. MailboxLogin

3. Move

Therefore, to apply all 11 possible audit events, run this command:

get-mailbox -ResultSize unlimited | set-mailbox -AuditOwner @{Add= “Update”,”MoveToDeletedItems”,”SoftDelete”,”HardDelete”,”UpdateFolderPermissions”,”UpdateInboxRules”,”UpdateCalendarDelegation”,”Create”,”MailboxLogin”,”Move”,”MailItemsAccessed”}

Repeat above command and swap out AuditOwner for AuditDelegate and AuditAdmin but remember to check the table (here) because not all audit events are available for Admin

Note: A user with full mailbox access to another user’s mailbox is logged as AuditDelegate.

Searching Audit Events

You can search for MailItemsAccessed events in

image will eventually replace


You can also use Exchange Online PowerShell to search for audit events, which is required if you need to search for events older than 90 days. Search-UnifiedAuditLog -Operations MailItemsAccessed or Search-MailboxAuditLog -Operations MailItemsAccessed

Other useful Exchange Online PowerShell commands:

View which events are being logged on a single mailbox (“Joe”)

get-mailbox joe | select -ExpandProperty auditadmin

get-mailbox joe | select -ExpandProperty auditowner

get-mailbox joe | select -ExpandProperty auditdelegate

Report how many accounts have auditing enabled:

get-mailbox | group-object AuditEnabled

Enable auditing on all mailboxes and increase audit log retention from 90 days to 180 days:

get-mailbox -resultsize unlimited | set-mailbox -AuditEnabled $true -AuditLogAgeLimit 180

Audit Log retention is independent of whether or not a retention policy (aka Legal hold) is applied to the mailbox. For example, if a mailbox is under legal hold, the audit events are not retained longer than the duration set by the AuditLogAgeLimit parameter.

If you increase the age beyond 90 days, you can only find those items in PowerShell using Search-MailboxAuditLog.

The following capacity limitations apply to mailbox auditing:

– No more than 3 million audit records are allowed per mailbox

– No more than 30 GB of audit records are allowed per mailbox (100GB if legal hold or retention policy has been applied to the mailbox)

– Tony also states in his blog “Exchange Online applies throttling for MailItemsAccessed events. If a mailbox generates more than 1,000 bind events in a 24-hour period, Exchange Online stops recording MailItemsAccessed events for bind operations for another 24 hours before resuming capture of these events. Microsoft says that less than 1% of mailboxes are subject to throttling.”


Assume you have a mailbox where MailItemsAccessed is not applied, but the mailbox has an E5 license.


You then try to add the audit event but you get an error that its only available for users with the appropriate license.


Double-check to see that you have the “Microsoft 365 Advanced Auditing” license type assigned.


Note: In my case, even though the box was checked, it did not work because this license assignment was inherited from an Azure AD P1 feature called Group-based licensing. So to work-around this bug, I directly assigned the license via PowerShell (since I couldn’t via the GUI since the checkbox was already selected) and that allowed the MailItemsAccessed to be applied.



$myO365Sku1 = New-MsolLicenseOptions -AccountSkuId $MSOLSKU

Set-MsolUserLicense -UserPrincipalName (username) -LicenseOptions $myO365Sku1

Top 10 Fixes for troubleshooting free/busy between Exchange on-premises and Exchange Online in Office 365

Free/busy often fails to work out-of-the-box after configuring Hybrid Exchange with Office 365. Here are my top ten fixes:


  1. Set the sharing policy to match on-premises and cloud.

    First, Connect to Exchange Online Remote Powershell and run get-sharingpolicy

    Then connect to on-premises Exchange Management Shell and run get-sharing policy

    Then make the two match on both sides.


Set-SharingPolicy -Identity SharingPolicy01 -Domains ‘ CalendarSharingFreeBusySimple’, ‘ CalendarSharingFreeBusyReviewer’, ‘ CalendarSharingFreeBusyReviewer’


  1. Set the organization relationship domains to include all accepted domains on both on-premises and cloud (always requires an IISRESET for it to take effect)
    This script helps identify missing domains in an existing relationship:


    if ( (Get-OrganizationRelationship).DomainNames -contains (Get-Mailbox user).PrimarySmtpAddress.Domain) { write-host “The domain was found” -ForegroundColor Green } else { write-host (Get-Mailbox user).PrimarySmtpAddress.Domain “was not found” -ForegroundColor Yellow}


    $OrgRel = Get-OrganizationRelationship Contoso

    $OrgRel.DomainNames += “”

    Set-OrganizationRelationship $OrgRel.Name -DomainName $OrgRel.DomainNames



    1. If the autodiscover DNS name is not published in external DNS, and if the client doesn’t want to do that, then manually configure TargetSharingEpr to use the published EWS path

      Get-OrganizationRelationship -Identity “O365 to On-premises – (GUID)” | Set-OrganizationRelationship -TargetSharingEpr

    4) For ‘401 errors’ try disabling the IOC connector in Exchange 2013 to have oAuth fall back to dAuth

    5) Sometimes it’s necessary to set the on-premises EWS virtual directory “WSSecurityAuthentication” value back to defaults (some clients change this if they do load balanced CAS)
    (this is commonly a last resort)

    Need to change WSSecurityAuthentication to False for EWS Virtual directory.

        a.       Set-WebServicesVirtualDirectory “Exch CAS\ews*” –WSSecurityAuthentication $false

        b.      Need to Stop MSExchangeServicesAppPool.

        c.       Need to Start  MSExchangeServicesAppPool.


      Need to change WSSecurityAuthentication to True again for EWS Virtual Directory.

        a.       Set-WebServicesVirtualDirectory “Exch CAS\ews*” –WSSecurityAuthentication $True

        b.      Need to Stop MSExchangeServicesAppPool.

        c.       Need to Start  MSExchangeServicesAppPool.


      Need to change WSSecurityAuthentication to False for Autodiscover Virtual directory.

        a.       Set-AutodiscoverVirtualDirectory “Exch CAS\Auto*” –WSSecurityAuthentication $false

        b.      Stop MSExchangeAutodiscoverAppPool.

        c.       Start  MSExchangeAutodiscoverAppPool.


      Change WSSecurityAuthentication to True again for Autodiscover Virtual Directory.

        a.       Set-AutodiscoverVirtualDirectory “Exch CAS\Auto*” –WSSecurityAuthentication $true

        b.      Stop MSExchangeAutodiscoverAppPool.

        c.       Start  MSExchangeAutodiscoverAppPool.


    6) If the Exchange Server is behind a web proxy then it is usually necessary to configure InternetWebProxy Set-ExchangeServer <Server Name> -InternetWebProxy:http://<Proxy Address>:<Proxy Port>


    7)  Verify the availability address space and see required SMTP domain with access method.

        Get-AvailabilityAddressSpace (Run this on-prem)


    8) Try running diagnostic commands:
    You can also use the Test-FederationTrust (on prem only) and Test-OrganizationRelationship  (run this both on prem and in cloud too)

    And you can also use this website to run tests:

    9) Make sure that the cloud user you are searching for has a valid (tenant) alias on their target mailbox (make sure Azure AD Connect is properly replicating that attribute, and/or, that the Exchange Address Policy is not blocking inheritance on that particular user/object).


    10) Run these commands to gather diagnostic information:



    Get-FederationTrust | fl

    Get-FederatedOrganizationIdentifier | fl

    Get-OrganizationRelationship | fl

    Get-WebServicesVirtualDirectory | Export-Clixml C:\temp\WebVdir.xml

    Get-AutoDiscoverVirtualDirectory | Export-Clixml C:\temp\AutoDVdir.xml

    Get-RemoteMailbox bobc_sync | fl

    Get-Mailbox “on-premises John Doe User” | fl

    Test-FederationTrust -UserIdentity [email protected] | fl

    Test-FederationTrustCertificate | fl

    Get-IntraOrganizationConnector | fl





    Get-FederationTrust | fl

    Get-FederatedOrganizationIdentifier | fl

    Get-OrganizationRelationship | fl

    Get-MailUser “on-premises John Doe User” | fl

    Get-Mailbox “Cloud user” | fl

    Get-IntraOrganizationConnector | fl

    get-OrganizationRelationship | Test-OrganizationRelationship -UserIdentity “cloud user”





    And when all else fails I reference these two blog articles:


How to fix Exchange Online Hybrid Outbound Connector 454 4.7.5 Certificate Validation Failure

While recently helping a client setup an Exchange Hybrid, the cloud to on-premises mail flow was failing validation due to 454 4.7.5 Certificate Validation Failure.

The next step was to verify that the TlsCertificateName value was properly set on the send and receive connectors to match the certificate name, following these articles:

In this case, the TlsCertificateName was already set correctly to match the certificate name (the Hybrid Exchange Wizard does a good job at setting that correctly).

The next step was to enable Verbose logging on the on-premises receive connector so that we can get a better look at the error.

To save time, I restarted the “Microsoft Exchange Frontend Transport” service so that the logging would take effect sooner.

Then navigating to the log directory can be a bit tricky:

C:\Program Files\Microsoft\Exchange Server\v15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive

Opening up the file revealed a very helpful bit of information! The SSL Certificate that Microsoft Office 365 is presenting to the Exchange server for the TLS encrypted email is not a trusted root. How can this be?


To cut to the chase, the root cause was that the server had not had windows updates run in a LONG time and therefore was really far behind in its root certificates.

The least disruptive solution was to download the Office certificate chains from Microsoft (here) and install them on the on-premises Exchange Server. Then after restarting the “Microsoft Exchange Frontend Transport” service and waiting a few minutes, the validation was successful.

PowerShell script to automatically heal non-deliverable emails (NDRs) as X500

One of the possible causes for a non-delivery report (NDR) is when a mail object in Exchange is moved or removed (Contact, Mailbox, MailUser, etc). When an object is moved, any internal user who had previously emailed that object will have a cached entry in their autocomplete cache that no longer matches up to what now exists. This happens because the autocomplete cache stores the value of the LegacyExchangeDN attribute of the original object before it is moved. When an object is moved, a new LegacyExchangeDN is created.

When sending a message, Outlook will check the Global Address List (GAL), and if it can’t find a match in the GAL, IMCEA encapsulation is used (Internet Mail Connector Encapsulated Addressing). An IMCEA encapsulated address looks like:

IMCEAEX-_O=[email protected]

So when an email is sent to the original cached object, an NDR will be sent back to the user containing a construct of the LegacyExchangeDN that it failed to reach, in a slightly different format:

A trailing “EX” at the end of IMCEAEX indicates that a non-SMTP address was encapsulated.

While a quick fix is to have the sender clear their autocomplete cache and re-send the message, a more automated solution is desirable when there are hundreds of potential senders who cached the old object.

I discovered a nifty PowerShell script (here) written by Michael England on 2/11/2013 that searches the Message Tracking Logs for NDRs, and then reconstructs the LegacyExchangeDN from the IMCEA format and adds that as an X500 proxy alias on the target recipient object (if it can be found). This works great in a scenario where the contact object was replaced by a Mailbox or MailUser object.

One of the things I appreciate about how he wrote this script is that when you run the script it reports what would be modified, then when you are ready to modify you just add the -Autoheal parameter to the end of the script. This way you can get an idea of what would be modified before it happens.

I had to modify Michael’s script because Microsoft changed the behavior of the LegacyExchangeDN value in Exchange 2010 SP1 Rollup 6 (released 10/27/2011) to add 3 random hex characters for uniqueness at the end of the LegacyExchangeDN. So when I attempted to use Michael’s script, it was not finding the destination object to add the X500 to it because the 3 random characters threw the search off. So I have posted a very minor change to an otherwise awesome script to strip the 3 characters when searching for the object to place the proxy alias on.

            Write-Host “looking for: ” $user.Substring(0,$user.Length-3)

            $user = $user.Substring(0,$user.Length-3)


I also made three separate copies, depending on the recipient object type (Mailbox, MailContact or MailUser).

You can download my slightly modified version of Michael’s script from the Microsoft Technet Gallery here.

I left all original credit to Michael in the script, since I only slightly changed the code to strip the 3 random hex characters out, and made separate copies of the script to search for MailUser and MailContact since his script only searched for Mailboxes. I was stoked to discover this awesome script by Michael.

Also shout out to a different Michael, Michael de Rooij for an excellent blog article on this topic here: